Mike Torrettinni

I had similar situation with using Copy, Left, MidStr... It started years ago with one search, then same or similar search in another feature and so on... and each time I was asking myself do I really need more robust solution, if I just need it 'this time' (forgetting about the other 'this times'), and so it went unchecked for years until it was time to refactor and finding fast and robust solution. I have same thing now with refactoring all different look-up tables (arrays). Many many different implementations and now going to refactor into fast and robust solution(s). And just refactoring and improving memory consumption and using faster algorithms instead of many slow different implementation, I'm improving performance of the whole project, even though there was no single bottle-neck related to this. So, I guess I could say the bottle-neck were all these inefficient implementations all over the project, so everything 😉

Yes, this is because I'm implementing common fast class for lookup-up data. And in this case, the searched values are always unique. So, I can exit asap. Well, I didn't really check what would happen with non-unique data, but I guess I lucked out with this.

Aha, I assumed it's like OnClick assignment to event... the flow would go just to the assigned method.

Thanks, it makes sense. This jumping of debugger to the same line again and again, even though the flow should be past it, is something new to me. Cool stuff! 😉

When I debug step by step, I see that the Comparison assignment to CompareIDs method is made on every binary search call, again and again!? I assumed if I move it before the actual call of BinarySearch method, it will assign and construct Comparer once and use it. But the debugger keeps jumping to the line to xComparison := CompareIDs; while doing binary search. I assume this is the cause for slower execution?

True, eventually I want to upgrade, is just a matter of a version. I don't want to stay behind too many versions.

I know, but this will be a few cycles after 10.2.3, so I will at least try 10.5 out! 😉 Last one I tried was 10.3 - no go.

10.4.2. looked so promising at the beginning, seems like I will wait for 10.5. I can't event imagine how LSP would react to my projects (old, a lot of unrefactored code with beginners design).

I've been randomizing 6 characters string and after 205,329,862,927 tries in last 12h, still no luck for 'Delphi' 😉 I would assume after 205 billion tries it could generate it's own name!? It's pretty fast in randomizing 'D', 'De', 'Del', 'Delp', 'Delph': D found in 0 ms and 121 random strings. De found in 0 ms and 4512 random strings. Del found in 10 ms and 277252 random strings. Delp found in 115 ms and 2362835 random strings. ..................Delph found in 8654 ms and 180720440 random strings. but 'Delphi' is still going... program Project1; {$APPTYPE CONSOLE} {$R *.res} uses System.SysUtils, System.Math, System.Diagnostics; const cChars = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'; function GetRandomString(aNoOfChars: integer): string; inline; var i: Integer; begin SetLength(Result, aNoOfChars); for i := 1 to aNoOfChars do Result[i] := cChars[Random(Length(cChars)) + 1]; end; const cPhrases: array[1..6] of string = ('D', 'De', 'Del', 'Delp', 'Delph', 'Delphi'); var vSW: TStopWatch; c: Int64; vLen: integer; vRndString, vPhrase: string; begin Randomize; for vPhrase in cPhrases do begin vSW := TStopWatch.StartNew; c := 0; vLen := Length(vPhrase); vRndString := GetRandomString(vLen); while vPhrase <> vRndString do begin Inc(c); vRndString := GetRandomString(vLen); if c mod 10000000 = 0 then // progress, so we know it's still working write('.'); end; writeln(Format('%s found in %d ms and %d random strings.', [vPhrase, vSW.ElapsedMilliseconds, c])); end; readln; end.

Well, 'delphi' is not really worth even trying 😉

How did you eliminate constructing the comparer each call? I tried to 'pre-construct' comparer like this: // new comparer definitions var xComparison: TComparison<TDataLine>; xComparer: IComparer<TDataLine>; // Function that does the value comparison function CompareIDs(const Left, Right: TDataLine): Integer; begin Result := CompareValue(Left.SeqNo, Right.SeqNo); end; // and I 'pre-construct' Comparer before first GetName_TArrayBinarySearch call ... xComparison := CompareIDs; xComparer := TComparer<TDataLine>.Construct(xComparison); ... // New search function function GetName_TArrayBinarySearch(const aData: TArray<TDataLine>; const aItem: TDataLine): string; var vIdx: integer; begin if TArray.BinarySearch<TDataLine>(aData, aItem, vIdx, xComparer) // using pre-constructed xComparer then Result := aData[vIdx].CustomName; end; But this only speeds up a little bit, still: 776 vs 129 ms. So, this is not the correct solution, right?

24h and almost 400 billion tries, still no 'Delphi'. Experiment done ☹️

OK, good to know.

Could I assume if I use password 'Delphi' for a profile, nobody will be able to brute-force guess it, if they run similar random generator? 😉

If I compile for 64bit, would that be using 64bit random number generator or I need specialized library for this?

Oh, you think this could be the issue? I thought index of arrays is not important if you don't use index to loop it. I don't use index in this case, but for in.

I just realized I can do a better comparison of new vs old methods, when optimizing code: in the past I would profile old version, and save profiling results, then run new and compare screenshots of timings. But is much better if you run both methods at the same time and compare results: This way I can keep both methods in the same code, compare small improvements and I can switch on/off old vs new when profiling and see the progress of eliminating or reducing bottlenecks.

I use various different ways to split strings by delimiter: using TStringList, string helper Split or System.StrUtils.SplitString, most of the time TStringList. To refactor and choose fastest one I did some test and also have custom split function. I'm just looking for fastest, so the type of result is less important than the performance: Interesting is that with short strings, pre-counting delimiters to correctly initialize array is faster, but when using longer strings, pre-counting slows down the performance. UsingStringList Short str: 509 Medium str: 1107 Long str: 3473 UsingSplitHelper Short str: 293 Medium str: 692 Long str: 2116 UsingSplitString Short str: 476 Medium str: 1413 Long str: 5909 winners: * CustomSplitWithPrecount - count delimiters to initialize array Short str: 178 Medium str: 474 Long str: 1659 * CustomSplitNoPrecount - no counting of delimiters, resize array at the end Short str: 184 Medium str: 457 Long str: 1477 program Project1; {$APPTYPE CONSOLE} {$R *.res} uses System.SysUtils, System.StrUtils, System.Classes, System.Diagnostics, System.Generics.Collections, System.Generics.Defaults, System.Types; const cMaxLoop = 1000000; cDelimiter : char = ','; cShortStr : string = 'word,string,character'; cMediumStr : string = 'cat,dog,mouse,horse,pigeon,tiger,worm,lion,turtle,fish'; cLongStr : string = 'black,maroon,green,dark green,light green,olive,navy,purple,teal,silver,grey,red,ligh blue,dark blue,navy blue,cyan,grey,white,aqua,teal,silver,orange,violet,blue violet,dark red,deep pink,steel blue,sea blue,aquamarine,medium turquoise,violet,last colorX'; var xStrList: TStringList; xArray: TArray<string>; xStrDynArray: TStringDynArray; xSW: TStopWatch; i: integer; function UsingStringList(const aString: string; const aDelimiter: char): TStringList; begin Result := TStringList.Create; Result.StrictDelimiter := True; Result.Delimiter := aDelimiter; Result.DelimitedText := aString; end; function UsingSplitHelper(const aString: string; const aDelimiter: char): TArray<string>; begin Result := aString.Split([aDelimiter]); end; function UsingSplitString(const aString: string; const aDelimiter: char): TStringDynArray; begin Result := System.StrUtils.SplitString(aString, aDelimiter); end; function CustomSplitWithPrecount(const aString: string; const aDelimiter: Char): TArray<string>; var i, c: Integer; vCurrPos, vCurrTokenStart: PChar; begin vCurrPos := PChar(aString); // count delimiters to set array size c := 0; for i := 1 to aString.Length do begin if vCurrPos^ = aDelimiter then Inc(c); inc(vCurrPos); end; if c = 0 then Exit // exit if no delimiters found else SetLength(Result, c + 1); // tokens = no of delimiters + 1 // parse c := 0; vCurrPos := PChar(aString); vCurrTokenStart := vCurrPos; for i := 1 to length(aString) do begin if vCurrPos^ = aDelimiter then begin // save Token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); inc(c); inc(vCurrPos); vCurrTokenStart := vCurrPos; // stop looping string at last delimiter if c = Length(Result) - 1 then begin Inc(vCurrPos, aString.Length - i); Break; end; end else inc(vCurrPos); end; // get last token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); end; function CustomSplitNoPrecount(const aString: string; const aDelimiter: Char): TArray<string>; var i, c: Integer; vCurrPos, vCurrTokenStart: PChar; begin // Preset array size to max SetLength(Result, aString.Length); if aString.Length = 0 then Exit; // parse c := 0; vCurrPos := PChar(aString); vCurrTokenStart := vCurrPos; for i := 1 to length(aString) do begin if vCurrPos^ = aDelimiter then begin // save Token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); inc(c); inc(vCurrPos); vCurrTokenStart := vCurrPos; // stop looping string at last delimiter if c = Length(Result) - 1 then begin Inc(vCurrPos, aString.Length - i); Break; end; end else inc(vCurrPos); end; // get last token SetString(Result[c], vCurrTokenStart, vCurrPos - vCurrTokenStart); // re-set filan array size SetLength(Result, c + 1); end; begin Writeln('UsingStringList'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cShortStr, cDelimiter); xStrList.Free; end; Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cMediumStr, cDelimiter); xStrList.Free; end; Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do begin xStrList := UsingStringList(cLongStr, cDelimiter); xStrList.Free; end; Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('UsingSplitHelper'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := UsingSplitHelper(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('UsingSplitString'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xStrDynArray := UsingSplitString(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('CustomSplitWithPrecount'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitWithPrecount(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); writeln; Writeln('CustomSplitNoPrecount'); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cShortStr, cDelimiter); Writeln('Short str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cMediumStr, cDelimiter); Writeln('Medium str: ' + xSW.ElapsedMilliseconds.ToString); xSW := TStopWatch.StartNew; for i := 1 to cMaxLoop do xArray := CustomSplitNoPrecount(cLongStr, cDelimiter); Writeln('Long str: ' + xSW.ElapsedMilliseconds.ToString); readln; end. Anybody have example of faster function that splits strings by delimiter?

Very good to hear this! I was planning to wait for 10.5, but I might reconsider and try 10.4.2!

Very good! Why did you choose that LastIdx should be -1 and not 0, to parse all text? If you pass 0 it will not delimit string. So I guess MaxInt is best, if you want all delimited strings? If I call Split(str, delim, false, 0) it doesn't delimit, it only delimits full string with Split(str, delim, false, MaxInt), because of condition : if (LastIdx <> -1) and (CurrIdx = LastIdx) then // last index reached - write all up to end begin NextDelim := Length(Str)+1; end and CurrIdx = 0 at the beginning. I would expect: LastIdx = 0 => parse All, LastIdx >0 parse to that LastIdx and end. No?

Interesting how you used lastDelimiterPos. Pretty cool! You are right, if no delimiters are found, then input string is the 1 string that needs to be returned. Thanks!

I thought PChar is faster... not sure why. It is marginally little faster function by index, except for longer strings PChar solution is marginally faster: CustomSplitWithPrecount2 Short str: 150 Medium str: 395 Long str: 1221 CustomSplitWithPrecountByIndex Short str: 145 Medium str: 386 Long str: 1228 But this is negligent difference. And index based solution is a little easier to handle, for me. So, here is the winner so far - with pre-calculated string length: function CustomSplitWithPrecountByIndex(const aString: string; const aDelimiter: Char): TArray<string>; var i, c2, c, s, vLen: Integer; begin vLen := aString.Length; // count delimiters to set array size c2 := 0; for i := 1 to vLen do if aString[i] = aDelimiter then Inc(c2); if c2 = 0 then Exit(nil) // exit if no delimiters found else SetLength(Result, c2 + 1); // tokens = no of delimiters + 1 // parse c := 0; s := 1; for i := 1 to vLen do begin if aString[i] = aDelimiter then begin // save Token SetString(Result[c], PChar(@aString[s]), i - s); inc(c); s := i + 1; // stop looping string at last delimiter if c = c2 then Break; end; end; // get last token SetString(Result[c], PChar(@aString[s]), vLen - s + 1); end;

This is my try with Pos, but is slower: CustomSplitWithPrecount Short str: 148 Medium str: 389 Long str: 1227 CustomSplitWithPrecountPos Short str: 207 Medium str: 542 Long str: 1773 function CustomSplitWithPrecountPos(const aString: string; const aDelimiter: Char): TArray<string>; var i, c, p,s: Integer; vCurrPos, vCurrTokenStart: PChar; begin vCurrPos := PChar(aString); // count delimiters to set array size c := 0; for i := 1 to aString.Length do begin if vCurrPos^ = aDelimiter then Inc(c); inc(vCurrPos); end; if c = 0 then Exit(nil) // exit if no delimiters found else SetLength(Result, c + 1); // tokens = no of delimiters + 1 // parse c := 0; s := 1; p := 0; repeat p := Pos(aDelimiter, aString, p + 1); if p <> 0 then begin // save Token SetString(Result[c], PChar(@aString[s]), p - s); inc(c); s := p+1; end; until p = 0; // get last token SetString(Result[c], PChar(@aString[s]), aString.Length - s + 1); end;

Actually, seems like pre-counting and initializing array is faster in Release mode - for all 3 lengths of strings: CustomSplitWithPrecount Short str: 150 Medium str: 394 Long str: 1265 CustomSplitNoPrecount Short str: 175 Medium str: 427 Long str: 1303

In case any other Delphi rookies are using TStringList to split strings and are thinking why I'm looking for better replacement: in older part of the code, before I knew about StrictDelimiter property, I used StringReplace to replace spaces, otherwise it would split by spaces also! 😉 So, I would StringReplace spaces, then parse with TStringList and then reverse back to spaces, with StringReplace again, I hope nobody else does this 😉