Mike Torrettinni

That should be "raise e[a-z]*\(", right?

Yes, I agree.

OK, that makes sense. I use 10.2.3 so new LSP can't help here. You never used any RegEx on your code?

Thanks, but not all tools catch all issues. RegEx is pretty useful.

I started using QueryPerformanceCounter for measuring timings of execution, so I can measure just 1 method execution instead of calling x times for TStopWatch to measure in milliseconds. But, I'm getting strange timings that I'm not sure how to interpret: here I have example of timing Sleep(1) and the difference in 100 calls could be 100% between Min vs Max execution time: This was the biggest difference I got, while some timings have less difference, but most of the time the difference is just too big. Anybody has any idea why do the timings of simple Sleep(1) have such a vast range? Is it just a fact that this is normal when measuring such a small timings, like single Sleep(1) call? Code: uses System.SysUtils, Winapi.Windows, System.Math; procedure TestQPerf; const cLoop = 100; var i, vStart, vStop, vMin, vMax: Int64; vIntArr: TArray<integer>; begin SetLength(vIntArr, cLoop); for i := 0 to cLoop - 1 do begin QueryPerformanceCounter(vStart); Sleep(1); QueryPerformanceCounter(vStop); vIntArr[i] := vStop - vStart; end; for i := Low(vIntArr) to High(vIntArr) do Writeln(vIntArr[i].ToString); vMin := System.Math.MinIntValue(vIntArr); vMax := System.Math.MaxIntValue(vIntArr); Writeln('Min: ' + vMin.ToString); Writeln('Max: ' + vMax.ToString); Writeln('Diff: ' + (vMax - vMin).ToString); end; begin TestQPerf; Readln; end.

At 17mins he says (paraphrasing): "This whole library has a pile of internal logic to run amazing numbers of iterations until we get something reasonably stable... and here we have example of 22Mio iterations to conclude this operation takes 32 nano seconds... and if I run it again you will some fluctuations..." Wow.

What Delphi version is this? Does it say 200? Like 20.0?

Eh, I'm just a Delphi rookie asking questions. I can only dream about being so knowledgeable or experienced to suggest anything to M. or E. But it is interesting that so many factors affect simple 'precise' measurement. Not sure if anybody mentioned, but power usage plan on laptops also affects the QPC a lot! Just tested on my 9 year old laptop and numbers are off the charts away from expected 😉

OK, thanks.

@David Heffernan I read all your comments, Is just I finally figured it out that my 'what do you mean? can you give me an example, or details?' just annoy you. So, unless I have anything smart to respond, I try to to not bother you. Why would anybody block your comments?

Thanks, I guess I should've included this in first post. Thanks, will do.

Thanks, I need a little more precise. If testing StringReplace there could be 1000s of executions within 1ms, if benchmarking some long sort that takes seconds, than it could be enough. So, I need single approach that covers most cases, so more precise. QPC is enough.

I'm OK with what I have now, QPC, I don't see how this could be more useful for my purpose.

Oh, I hope it is clear that calculating QPC per second was supposed to find the 'real' QP Frequency. I assumed that on a computer where QPF = 10,000,000 is a bug, then the QPC is not aligned with it. But as we tested, QPF and QPC per second are the same (or as close as possible), 10,000,000 on my comp, 3,331,195 on yours.

Thanks! I would love say I understand everything, but I think I got the main point. I think I'm getting to realize this, yes. I already noticed using trimmed mean (20%) removes most of the edge cases, warmup and other odd values. Perhaps more than 100/1000/100000 reps and trimmed mean (50%) could be even better. Yes, well, accurate enough. Is just finding the right combination of benchmarking details and time and interpretation of results - reps, mean, min, max...

This updated function seems to be even closer to 10,000,000: than previous function: @Kas Ob. It takes into account odd/even issue - assuming if it hits 1 or 2 once, it will hit it again next 1s cycle. function GetQPCPerSecond: Int64; var ms1, ms2: integer; // vars for milliseconds t1, t2: Int64; // vars for QPC begin // assuming loop will run a few times within each 1ms, // so we just need to wait for first occurence of the same ms as ms1, after 1 cycle of 1 second // get starting point (start with next 1s interval, at ms1 = 1, or ms = 2, in case OS skips 1 or 2) repeat QueryPerformanceCounter(t1); ms1 := MilliSecondOf(System.SysUtils.Time); until (ms1 = 1) or (ms1 = 2); // wait until next ms repeat until MilliSecondOf(System.SysUtils.Time) > ms1; // get QPC at the next 1s cycle repeat QueryPerformanceCounter(t2); // acquire QPC ms2 := MilliSecondOf(System.SysUtils.Time); // get milliseconds of current time until (ms2 = ms1); // end when 1s has cyled Result := t2 - t1; end;

Good one! 🙂

Not sure if you do the same, but unless I need debugging, I always run performance timings in Release and Run without debugging Seems to give more constant results. Could this be also giving better results on your PC? Thanks, didn't think of that! Oh, it was just a simple idea. In my test that specific while loop was iterating around 6Mio times within 1 sec, so I assume it should hit 1ms quite confidently. But you are right, if OS skips odd/even milliseconds, then this could loop endlessly. I hope that higher up the scale we go, for example > 1s, > 1min... we could get to decently constant comparable results. For example: comparing performance of single call of StringReplace in Delphi 10.2.3 and 10.5 (when released) does not make sense. But preparing a benchmarking process that takes into account all (most/some) of the details pointed out in this thread, should give pretty good results to make conclusions of better/worse/same performance.

If we can trust that Date/Time functions consistently return same accurate timestamp, than I think we can just calculate QPC number within 1s. This function gets QPC within 999ms and if this is acceptable accurate, than we can calculate QPC per ms/µs/ns,... on a current PC. This could probably be optimized to be even more accurate, but my PC shows: 9,989,541 (within accepted accuracy of 1% difference, if it should be 10,000,000) uses System.SysUtils, Winapi.Windows, System.DateUtils; function GetQPCPerSecond: Int64; var ms1, ms2: integer; // vars for milliseconds t1, t2: Int64; // vars for QPC begin // assuming loop will run a few times within each 1ms, // so we just need to wait for first occurence of the same ms as ms1, after 1 cycle of 1 second QueryPerformanceCounter(t1); ms1 := MilliSecondOf(System.SysUtils.Time) - 1; // reduce by 1 to skip current millisecond while True do begin QueryPerformanceCounter(t2); // acquire QPC ms2 := MilliSecondOf(System.SysUtils.Time); // get milliseconds of current time if ms2 = ms1 then // check if milliseconds have cycled 1 second Break; end; Result := t2 - t1; end; var i: integer; begin for i := 1 to 10 do // run a few times in case computer is busy in bg Writeln('QPC per second: ' + GetQPCPerSecond.ToString); end; @Kas Ob. Can you test this function how it relates to your frequency of 3331195?

Do you know of Delphi version?

Sleep(1) was used to test accuracy after s := FormatDateTime('YYYYMMDD', Date); was giving odd timing results for single execution. I thought Sleep(1) will always execute in exactly ms, now I know it doesn't, as @Kas Ob. explained above,

Yes, that makes sense. But, funny enough!, I did a very simple timing log and 1s or 1000ms (14284 - 13283 in screenshot) is almost 10,000,000 ticks, like the 'defined feature/bug' frequency: 🙂 (the milliseconds are based on System.SysUtils.Time)

So, not only we have all these factors affecting precise measurements now we have 'bugs/features' that can change based on each OS update. Of course this is when precise timings are wanted, I assume timings based on System.SysUtils.GetTime timestamp function (or similar date time functions) would correctly express timings in seconds, minutes, hours.

Seems that a 10-100 repetitions is pretty much enough to even out the results - tried 10-10mio and results are within a few percentage points difference. It also helps to warmup with 1+ call before and trimmed mean (0.2 or 20%) also gives better value than standard average. And the more you separate full test run across multiple methods (organizing into smaller refactored pieces), the results get affected - the warmup seems to get lost 😉 @emailx45 I tried within a separate thread and affinity mask set, but the results were so odd, much bigger differences than in main thread, so I assume there are other factors of the thread handling affecting result, so i gave up this approach. But, I'm not expert on threads, anyway. Thank you @Kas Ob. , the QueryPerformanceFrequency is the key to turn into milliseconds or seconds, but my frequency is always: 10000000.

And then this comment: "Definitely add repeat. At chunks of <10s system handing CPU time to other tasks may affect test performance seriously. A test of ~10 minutes is more reliable. Also, kill all other CPU-intensive and disk-intensive processes." https://stackoverflow.com/questions/2072361/what-is-the-best-way-to-measure-how-long-code-takes-to-execute#comment2002250_2072367