Jump to content

Jud

Members
  • Content Count

    75
  • Joined

  • Last visited

Community Reputation

1 Neutral

Technical Information

  • Delphi-Version
    Delphi 10.4 Sydney

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. Jud

    Many TTasks

    But I know that the number of tasks is going to be considerably larger than the number of logical CPUs.
  2. Jud

    Many TTasks

    I was just using that as an example. But I have a routine that tells me the number of logical cores.
  3. Jud

    Many TTasks

    Thanks, that is what I was wondering. I know how to wait for ONE task to finish and to wait for ALL tasks to finish. So for 9 tasks with 8 logical CPUs, I can wait for one to finish before starting the 9th one. But what if there are, say, 20 tasks, and I don't want to run all so in parallel. Is there an easy way to keep only 8 running in parallel at the time?
  4. If you have started more TTasks than you have logical CPUs, does it do them in parallel or does it finish one before starting the next one in the queue? Example, you have a CPU with 8 logical cores and start 9 TTasks. Does it finish one of the first 8 before starting #9?
  5. Jud

    Maximum static memory

    I don't know how. Is there a blog for Delphi?
  6. Jud

    Maximum static memory

    Maybe. It was a few years ago. I was getting the memory available inside my program too, I think. But last night I had a good insight - rather than set up the 100,000,000+ potential buckets when only a few thousand of them will have something in them, do a pass through the data to see what buckets will have something in them and set up only that many buckets! I haven't had a chance to work on that yet, but it should be faster than quicksort and make the final processing step faster too.
  7. Jud

    Maximum static memory

    I could extract the relevant code and give a real data set, but it would take some work and the data would be very large. But several days ago I abandoned the bucket sort method and now the quicksort method is working and producing results. I think the problem was because there are millions of buckets and almost all of them are empty, so there is a lot of work with memory. (Actually, on two small test cases, the bucket sort system was 30% and 3X faster than the quicksort version, but on big data sets, there are way too many buckets, almost all empty.) Also, I can multithread the version that uses quicksort. I wouldn't be able to do that effectively with the bucket sort system because of the large memory. (The workstations here have 128GB of RAM, and one instance was taking up nearly half of it.) --------------- A few years ago when I had a large dynamic array of short tLists, it seemed that it was reserving 4KB for each list, even if the list was much smaller. Is that how the memory manager works?
  8. Jud

    Maximum static memory

    I thought the minimum was 4KB. So if I have a dynamic array with the SetLength to 1000 bytes, it is actually going to allocate 64KB? I could rig up a small demo, with real data in a couple of hours, but I think that the problem is with bucket sort, it turns out that almost all of the buckets are empty (I couldn't tell this until I got it working and collected some statistics). A secondary cause could be that I have a static array of counters to tell how many items are in each bucket, and I use FillChar to zero out in each of the 143,000,000 calls to the procedure. That array has about 60,000,000 bytes, but I only zero out as much as I need to, which varies with each call to the subroutine.
  9. Jud

    Maximum static memory

    I don't know if anyone is still reading this, but I got the bucket sort method working. On small cases, it is up to 3 times faster than the quicksort method. But on data sets that are of interest, it is much slower. On a test case with real data, the quicksort method takes 61 minutes but the bucket sort method takes 27 hours. The procedure that does the work doesn't know how many buckets there will be or how large they will need to be until it gets the data. For that reason, I used a dynamic array of dynamic arrays, outside the procedure. I did some statistics on the test case (that has real data). The procedure is called about 143,000,000 times. There are up to about 26,000,000 buckets. However, there are at most about 5,400 items spread out in those buckets. So about 99.98% of the buckets are empty. That is the reason it is so much slower than quicksort - going through all of those empty buckets. The bucket sort was potentially a good idea, but it didn't turn out well. I wasted a lot of time on it, but I learned some things.
  10. Jud

    Maximum static memory

    No, it depends on what the rest of the program is doing. An O(n log n) quicksort could take 99%+ of the time, depending. I've been using quicksort for over 40 years and I tested it on this data.
  11. Jud

    Maximum static memory

    It is late and I'm about to be away for a couple of days. I haven't finished all of the changes I talked about (with fewer but bigger buckets) but it definitely seems that the problem is in swapping out the very large dynamic array to the drive millions and millions of times. Details by the middle of next week.
  12. Jud

    Maximum static memory

    Well, I think I have figured it out. I think the memory manager is allocating memory for the dynamic arrays in 4KB chunks, no matter how much is needed. That is causing it to swap to the SATA SSD like crazy. I develop on a computer that has 32GB. I have it show me how much memory is available after I SetLength, and it says about 19GB left. I took the very same EXE and ran it on a computer with only 16GB. It should use up all of the memory, but it was showing about 9GB available. So it has to be swapping out. In the old days, I would have hard the hard drive accesses. In Task Manager, it does NOT show the drive activity, but the system must be doing it, I think (otherwise there is no way it would run at all on the 16GB machine). If it is allocating minimum chunks of 4KB, as I think, the total memory use is approaching 200GB. Which brings be back to the original Delphi philosophy of "you'll never need more than 2GB, even in 64-bit programs". But I've thought of some work-arounds. Primarily, reduce the number of buckets by a factor of, say, 256. Then anything that would have gone into buckets 1-256 go into the first bucket, 257-512 go into the second bucket, etc. That will make efficient use of the 4K blocks. Then a little processing on each of these bigger buckets will get it ready for the final processing stage.
  13. Jud

    Maximum static memory

    I think the problem is probably calling SetLength on a fairly large dynamic array millions of times inside a procedure is overwhelming the memory manager. I need to keep a count showing how many times it has been called in the last second, and see if that starts to drop. Maybe I can get some sample code. I'm trying different things. Today I finished a run after making the 2-dimensional non-rectangular array global, so SetLength isn't called for it millions of time inside the procedure that prepares the data for the final phase. This was faster, but the run took 39.6 hours, as opposed to about 2.2 for the quicksort approach. But I have a dynamic array inside that procedure that tells me how many elements across the big array is. I gathered some statistics during the run and it shows that this array could be less than 200 megabytes, so I'm going to try making it a global static array. Inside the procedure that is called millions of times, I have to set part of the array to zero, but I know how much of that array needs to use FillChar.
  14. Jud

    Maximum static memory

    The final phase of the processing needs the items in buckets, so bucket sort seems logical. It should beat quicksort. I think the problem making it slow was that the procedure that prepares the data for final processing is called millions of times and the large dynamic array of arrays was inside that procedure, calling SetLength each time. I've just made the large dynamic array global and SetLength to once to reasonable estimates, and then increase it if necessary. That is working but I haven't compared it to quicksort yet.
  15. Jud

    Maximum static memory

    I'm hot shuffling the records, I'm grouping them according to how they are to be processed. The records are 16-bytes long (there are a lot of them). Yesterday (after watching a Webinar and installing Delphi 11.0, yes it is available) I did more profiling. I said that the quicksort was taking 80%. There is a SwapRecords procedure nested in the quicksort procedure that is taking an additional 7%. I went back to my earlier version (with compiler directives) - the one that uses a non-rectulangular dynamic array of dynamic arrays. The profile shows that SetLength is taking most of the time. The profiler shows "system" using 77% of the time. @FillChar is using the most, which must be zeroing out the dynamic array (which ( don't need). SysGetMem is second in the profiler. But yesterday I had the idea of using a large dynamic array of small static arrays. I think that is going to work (in 64-bit mode). Also, what Administrators said above just made me think of another possible approach. Keep the dynamic array of records and go through and pick out the ones with a certain key, and process them. Repeat with the next key. The problem with this is that there are so many keys (and each one will have to make a pass through the array) that I think it will be back to quadratic time. Probably tomorrow I will implement the dynamic array of small static array method.
×