Jump to content

Mike Torrettinni

Members
  • Content Count

    1048
  • Joined

  • Last visited

  • Days Won

    1

Everything posted by Mike Torrettinni

  1. I'm interested how you guys identify bottlenecks in your source code. When I profile my code, I usually try to optimize anything that could reduce execution time by least 10%+. It also depends what I see that can be optimized or is very clearly badly designed. Sometimes I think I can improve it and it ends up even worse πŸ˜‰ When do you decide you will try to optimize certain parts of the code? How you define bottlenecks, is it the expected percentage of improvement (5,10, 50%...)? Do you ever optimize something that is not a bottleneck, but a customer/user is requesting performance improvements?
  2. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    I just realized I can do a better comparison of new vs old methods, when optimizing code: in the past I would profile old version, and save profiling results, then run new and compare screenshots of timings. But is much better if you run both methods at the same time and compare results: This way I can keep both methods in the same code, compare small improvements and I can switch on/off old vs new when profiling and see the progress of eliminating or reducing bottlenecks.
  3. I use various different ways to split strings by delimiter: using TStringList, string helper Split or System.StrUtils.SplitString, most of the time TStringList. To refactor and choose fastest one I did some test and also have custom split function. I'm just looking for fastest, so the type of result is less important than the performance: Interesting is that with short strings, pre-counting delimiters to correctly initialize array is faster, but when using longer strings, pre-counting slows down the performance. UsingStringList Short str: 509 Medium str: 1107 Long str: 3473 UsingSplitHelper Short str: 293 Medium str: 692 Long str: 2116 UsingSplitString Short str: 476 Medium str: 1413 Long str: 5909 winners: * CustomSplitWithPrecount - count delimiters to initialize array Short str: 178 Medium str: 474 Long str: 1659 * CustomSplitNoPrecount - no counting of delimiters, resize array at the end Short str: 184 Medium str: 457 Long str: 1477 program Project1; {$APPTYPE CONSOLE} {$R *.res} uses System.SysUtils, System.StrUtils, System.Classes, System.Diagnostics, System.Generics.Collections, System.Generics.Defaults, System.Types; const cMaxLoop = 1000000; cDelimiter : char = ','; cShortStr : string = 'word,string,character'; cMediumStr : string = 'cat,dog,mouse,horse,pigeon,tiger,worm,lion,turtle,fish'; cLongStr : string = 'black,maroon,green,dark green,light green,olive,navy,purple,teal,silver,grey,red,ligh blue,dark blue,navy blue,cyan,grey,white,aqua,teal,silver,orange,violet,blue violet,dark red,deep pink,steel blue,sea blue,aquamarine,medium turquoise,violet,last colorX'; var xStrList: TStringList; xArray: TArray<string>; xStrDynArray: TStringDynArray; xSW: TStopWatch; i: integer; function UsingStringList(const aString: string; const aDelimiter: char): TStringList; begin Result := TStringList.Create; Result.StrictDelimiter := True; Result.Delimiter := aDelimiter; Result.DelimitedText := aString; end; function UsingSplitHelper(const aString: string; const aDelimiter: char): TArray<string>; begin Result := aString.Split([aDelimiter]); end; function UsingSplitString(const aString: string; const aDelimiter: char): TStringDynArray; begin Result := System.StrUtils.SplitString(aString, aDelimiter); end; function CustomSplitWithPrecount(const aString: string; const aDelimiter: Char): TArray<string>; var i, c: Integer; vCurrPos, vCurrTokenStart: PChar; begin vCurrPos := PChar(aString); // count delimiters to set array size c := 0; for i := 1 to aString.Length do begin if vCurrPos^ = aDelimiter then Inc(c); inc(vCurrPos); end; if c = 0 then Exit // exit if no delimiters found else SetLength(Result, c + 1); // tokens = no of delimiters + 1 // parse c := 0; vCurrPos := PChar(aString); vCurrTokenStart := vCurrPos; for i := 1 to length(aString) do begin if vCurrPos^ = aDelimiter then begin // save Token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); inc(c); inc(vCurrPos); vCurrTokenStart := vCurrPos; // stop looping string at last delimiter if c = Length(Result) - 1 then begin Inc(vCurrPos, aString.Length - i); Break; end; end else inc(vCurrPos); end; // get last token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); end; function CustomSplitNoPrecount(const aString: string; const aDelimiter: Char): TArray<string>; var i, c: Integer; vCurrPos, vCurrTokenStart: PChar; begin // Preset array size to max SetLength(Result, aString.Length); if aString.Length = 0 then Exit; // parse c := 0; vCurrPos := PChar(aString); vCurrTokenStart := vCurrPos; for i := 1 to length(aString) do begin if vCurrPos^ = aDelimiter then begin // save Token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); inc(c); inc(vCurrPos); vCurrTokenStart := vCurrPos; // stop looping string at last delimiter if c = Length(Result) - 1 then begin Inc(vCurrPos, aString.Length - i); Break; end; end else inc(vCurrPos); end; // get last token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); // re-set filan array size SetLength(Result, c + 1); end; begin Writeln('UsingStringList'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cShortStr, cDelimiter); xStrList.Free; end; Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cMediumStr, cDelimiter); xStrList.Free; end; Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cLongStr, cDelimiter); xStrList.Free; end; Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('UsingSplitHelper'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('UsingSplitString'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('CustomSplitWithPrecount'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('CustomSplitNoPrecount'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); readln; end. Anybody have example of faster function that splits strings by delimiter?
  4. Mike Torrettinni

    Delphi 10.4.2 first impressions

    Very good to hear this! I was planning to wait for 10.5, but I might reconsider and try 10.4.2!
  5. Mike Torrettinni

    Micro optimization: Split strings

    Very good! Why did you choose that LastIdx should be -1 and not 0, to parse all text? If you pass 0 it will not delimit string. So I guess MaxInt is best, if you want all delimited strings? If I call Split(str, delim, false, 0) it doesn't delimit, it only delimits full string with Split(str, delim, false, MaxInt), because of condition : if (LastIdx <> -1) and (CurrIdx = LastIdx) then // last index reached - write all up to end beginο»Ώ NextDelim := Length(Str)+1;ο»Ώ end and CurrIdx = 0 at the beginning. I would expect: LastIdx = 0 => parse All, LastIdx >0 parse to that LastIdx and end. No?
  6. Mike Torrettinni

    Micro optimization: Split strings

    Interesting how you used lastDelimiterPos. Pretty cool! You are right, if no delimiters are found, then input string is the 1 string that needs to be returned. Thanks!
  7. Mike Torrettinni

    Micro optimization: Split strings

    I thought PChar is faster... not sure why. It is marginally little faster function by index, except for longer strings PChar solution is marginally faster: CustomSplitWithPrecount2 Short str: 150 Medium str: 395 Long str: 1221 CustomSplitWithPrecountByIndex Short str: 145 Medium str: 386 Long str: 1228 But this is negligent difference. And index based solution is a little easier to handle, for me. So, here is the winner so far - with pre-calculated string length: function CustomSplitWithPrecountByIndex(const aString: string; const aDelimiter: Char): TArray<string>; var i, c2, c, s, vLen: Integer; begin vLen := aString.Length; // count delimiters to set array size c2 := 0; for i := 1 to vLen do if aString[i] = aDelimiter then Inc(c2); if c2 = 0 then Exit(nil) // exit if no delimiters found else SetLength(Result, c2 + 1); // tokens = no of delimiters + 1 // parse c := 0; s := 1; for i := 1 to vLen do begin if aString[i] = aDelimiter then begin // save Token SetString(Result[c], PChar(@aString[s]), i - s); inc(c); s := i + 1; // stop looping string at last delimiter if c = c2 then Break; end; end; // get last token SetString(Result[c], PChar(@aString[s]), vLen - s + 1); end;
  8. Mike Torrettinni

    Micro optimization: Split strings

    This is my try with Pos, but is slower: CustomSplitWithPrecount Short str: 148 Medium str: 389 Long str: 1227 CustomSplitWithPrecountPos Short str: 207 Medium str: 542 Long str: 1773 function CustomSplitWithPrecountPos(const aString: string; const aDelimiter: Char): TArray<string>; var i, c, p,s: Integer; vCurrPos, vCurrTokenStart: PChar; begin vCurrPos := PChar(aString); // count delimiters to set array size c := 0; for i := 1 to aString.Length do begin if vCurrPos^ = aDelimiter then Inc(c); inc(vCurrPos); end; if c = 0 then Exit(nil) // exit if no delimiters found else SetLength(Result, c + 1); // tokens = no of delimiters + 1 // parse c := 0; s := 1; p := 0; repeat p := Pos(aDelimiter, aString, p + 1); if p <> 0 then begin // save Token SetString(Result[c], PChar(@aString[s]), p - s); inc(c); s := p+1; end; until p = 0; // get last token SetString(Result[c], PChar(@aString[s]), aString.Length - s + 1); end;
  9. Mike Torrettinni

    Micro optimization: Split strings

    Actually, seems like pre-counting and initializing array is faster in Release mode - for all 3 lengths of strings: CustomSplitWithPrecount Short str: 150 Medium str: 394 Long str: 1265 CustomSplitNoPrecount Short str: 175 Medium str: 427 Long str: 1303
  10. Mike Torrettinni

    Micro optimization: Split strings

    In case any other Delphi rookies are using TStringList to split strings and are thinking why I'm looking for better replacement: in older part of the code, before I knew about StrictDelimiter property, I used StringReplace to replace spaces, otherwise it would split by spaces also! πŸ˜‰ So, I would StringReplace spaces, then parse with TStringList and then reverse back to spaces, with StringReplace again, I hope nobody else does this πŸ˜‰
  11. Mike Torrettinni

    Micro optimization: Split strings

    Thanks, good eyes! Well, yes, it might seems the opposite, but I do try to make it right πŸ˜‰ I need more practice,
  12. Delphi is 26 years old! And this: "...over 26 years has been used to build applications used by billions of people..." is pretty impressive achievement! πŸ˜‰ https://blogs.embarcadero.com/26-years-of-delphi/
  13. Mike Torrettinni

    Delphi is 26 years old - Marco's blog

    And next step is of course AI, I guess :
  14. Mike Torrettinni

    Delphi is 26 years old - Marco's blog

    Oh, the fun times of drawing stars for school assignment in Turbo Pascal πŸ˜‰
  15. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    Thanks, I agree! Sometimes is also fun πŸ˜‰
  16. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    I actually have similar situations quite often, and in topic using Pos before StringReplace I describe the benefit because of how many developers are affected. So, as you describe, not all optimization affect all users. But if it does some, why shouldn't I improve it? I have a feature that exports/import data and every feature in project ads some execution time, and every year I optimize parts of it so that it stays around 20s. If I didn't optimize it would probably be 60s by now. The thing is that every time I profile it, I see something I can improve, that I didn't see the last time - because I have more experience now. So, is not just a problem of deciding what to optimize, is also what I know how to optimize.
  17. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    I searched for it but can't find it. Can you point me to a comment, topic?
  18. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    Thanks. Oh, don't get me wrong, I had my share of mistakenly going down the road of wrong path, and I sure will do again πŸ˜‰ Unfortunately in most of my cases I can't just replace parts of the code with already made example of faster. Either it doesn't exist, or I don't know about it or it would take a big sum of money to hire an expert. Not sure if you noticed some of my topics about optimization, micro-optimization... as comments show some are really triggered that I post such basic things as big 'revelations', for me is a day by day discovery of new things. A lot of them because I try to optimize wrong things, some of them proved to be real improvements.
  19. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    Thank you, I read the same link a while ago and the best answer has very clearly written: " What this means is that, in the absence of measured performance issues you shouldn't optimize because you think you will get a performance gain. There are obvious optimizations (like not doing string concatenation inside a tight loop) but anything that isn't a trivially clear optimization should be avoided until it can be measured. " I understand this as: as long as it's a measured bottleneck, it is worth looking into. If you want to get away from the theory, I would love to hear about your last bottleneck, how you defined it.
  20. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    I do understand that not all applications need to be really fast and optimized. My project is kind of a parser project, so will never be fast enough - or, best outcome would be for the parsing process to be done within a timeframe of user pressing mouse button down and releasing it. So, I'm trying to be better at assessing bottlenecks. If 10% reduced process runtime is not worth improving, what is your criteria? How did you decide your last bottleneck needs fixing?
  21. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    Isn't that how you learn, get experience? You tackle a problem and try to fix, improve it. Sometimes it works, sometimes the experience is not there, yet, and try another approach or come back at another time.
  22. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    I've been reading a lot about optimization and it's interesting to see that phrase 'premature optimization' is usually just a BS, except for extremely basic cases. I agree with this and by attempting some minor optimizations I learned how to optimize other bigger parts of the code.
  23. I noticed @DesadaptadoDimensional said he is learning Delphi, I guess in school. Do we still have schools that teach Delphi, what countries? I remember schools in Turkey are supposed to start offering Delphi classes, but I don't know the status of this:
  24. Mike Torrettinni

    How do you identify bottleneck in Delphi code?

    Yes, I use profiler to see what methods gets called most and what methods take what percentage of total runtime. Then I decide which ones to optimize sometimes based on percentage of total runtime, sometimes based on number of calls. You don't measure the runtime of methods? Sometimes is very obvious what is bottleneck, sometimes is not... if you have 5 methods takes 15% of total runtime each, how do you decide which one to work on? Some methods could use some simple optimization tricks, some need different data structure to work with, different sorting algorithms... it can take hours to implement new approach, but then you measure it and you didn't gain anything.
  25. I use Copy and MidStr a lot, and I wanted to test which one is best to be used to check if String starts with substring, so I can refactor code and use just one function. There are also String.StartsWith helper and StartsStr function. So, I ran some simple tests and seems to be that Copy is fastest option: I noticed this question on SO: Difference between System.copy and StrUtils.MidStr https://stackoverflow.com/q/13411139 which explains that MidStrs uses Copy (I checked and in Delphi 10.2.3 it does), so Copy should be faster then. Substring exists at the start: Short string: Copy: 166 // WINNER MidStr: 184 StartsWith: 511 StartsStr: 792 Long string: Copy: 179 // WINNER MidStr: 190 StartsWith: 524 StartsStr: 840 Substring NOT exists at the start: Short string: Copy: 169 // WINNER MidStr: 180 StartsWith: 514 StartsStr: 886 Long string: Copy: 170 // WINNER MidStr: 182 StartsWith: 517 StartsStr: 897 Does anybody know of any faster option to check if string starts with substring? program Project1; {$APPTYPE CONSOLE} {$R *.res} uses System.SysUtils, System.Diagnostics, System.StrUtils; const cMaxLoop = 10000000; cText = 'Error: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.'; cTextNoHits = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.'; cStartsWithShort = 'E'; cStartsWithLong = 'Error:'; var vSW: TStopWatch; i, vLen: integer; vResult: boolean; begin Writeln('Substring exists at the start:'); // Short string Writeln('Short string:'); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := Copy(cText, 1, 1) = cStartsWithShort; Writeln(' Copy: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := MidStr(cText, 1, 1) = cStartsWithShort; Writeln(' MidStr: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := cText.StartsWith(cStartsWithShort); Writeln(' StartsWith: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := StartsStr(cStartsWithShort, cText); Writeln(' StartsStr: ' + vSW.ElapsedMilliseconds.ToString); // Long string Writeln('Long string:'); vLen := Length(cStartsWithLong); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := Copy(cText, 1, vLen) = cStartsWithLong; Writeln(' Copy: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := MidStr(cText, 1, vLen) = cStartsWithLong; Writeln(' MidStr: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := cText.StartsWith(cStartsWithLong); Writeln(' StartsWith: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := StartsStr(cStartsWithLong, cText); Writeln(' StartsStr: ' + vSW.ElapsedMilliseconds.ToString); Writeln; // Text DEOS NOT start with selected string Writeln('Substring NOT exists at the start:'); // Short string Writeln('Short string:'); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := Copy(cTextNoHits, 1, 1) = cStartsWithShort; Writeln(' Copy: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := MidStr(cTextNoHits, 1, 1) = cStartsWithShort; Writeln(' MidStr: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := cTextNoHits.StartsWith(cStartsWithShort); Writeln(' StartsWith: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := StartsStr(cStartsWithShort, cTextNoHits); Writeln(' StartsStr: ' + vSW.ElapsedMilliseconds.ToString); // Long string Writeln('Long string:'); vLen := Length(cStartsWithLong); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := Copy(cTextNoHits, 1, vLen) = cStartsWithLong; Writeln(' Copy: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := MidStr(cTextNoHits, 1, vLen) = cStartsWithLong; Writeln(' MidStr: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := cTextNoHits.StartsWith(cStartsWithLong); Writeln(' StartsWith: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do vResult := StartsStr(cStartsWithLong, cTextNoHits); Writeln(' StartsStr: ' + vSW.ElapsedMilliseconds.ToString); readln; end.
Γ—