Stefan Glienke

But we have the slow RTL routines that can easily be improved - we don't have the new ctors and dtors and actually I would not hold my breath for them to be implemented as you imagine them to - they will be driven by additional typeinfo (look into 10.3 System.pas where you can find all the relevant code for that feature because they just disabled it inside the compiler but did not revert the necessary RTL code). And still if you have an array of those record it would still have to loop through the array and call the dtor for every single item regardless. An optimized version of FinalizeArray/Record can just shift pointers over the array and do the cleanup - even if using the records managed field table - that is just a simple record structure, nothing fancy. Putting everything into nested loops and calls regardless the fact if the fields are even filled with something is what makes the current version slow - that is as I mentioned before what makes Rapid.Generics faster on its Clear.

Write a better _FinalizeArray routine for tkRecord then please as the current implementation is pretty terrible as it keeps calling _FinalizeRecord in a loop which again calls _FinalizeArray with ElemCount 1. That contributes to the slowness of Generics.Collections if you have records in a list. I did some patching today and added an ElemCount parameter to _FinalizeRecord to only call that once per _FinalizeArray - but did that in pure pascal.

Storing records in collections is a delicate topic as usually the code is a bit more complex than just storing integers or pointers. Even more so if the record has managed types like string. In this case the simple fact of having a local variable of T inside your generic code and doing one assignment more than necessary (because for example passing some olditem to a notification) might cause a severe slow-down. If you put such code into a single method it does the stack cleanup and finalization for that variable in all cases even if there is no notification to be called. The code in Rapid.Generics is using some shortcuts and produces even more convoluted code than System.Generics.Collections has since its refactoring in XE7. It does for example not use TArray<T> as backing storage for its list but pure pointer math. It also does not zero memory for this array which buys some speed by not doing the round trip to those all the code happening in System.DynArraySetLength - especially for managed types. That buys a bit of performance when adding items - especially if you don't set the capacity before. I know that the RTL collections had a ton of bugs caused by that refactoring as certain typekinds suddenly were not handled properly - I don't see any unit tests for Rapid.Generics though so I would not say that they are working for all kinds of types that you might store in those lists. As for the specific case of Clear taking longer in the RTL collections that is another optimization being done in Rapid.Generics where it simply cleans up the memory and is done whereas the RTL runs through some extra code which is not necessary if there is no OnChange attached to the list. Edit: I looked into the Rapid.Generics code for records and it maintains its own mechanism to cleanup any managed fields with a small performance improvement if the field is actually empty. This causes a major speedup if you are doing some benchmarks with empty records but I guess with real data this won't give much. I tested with a small record with 2 string fields and an integer and when they were empty the Clear call was very fast compared to the RTL list but not so much anymore when the fields contaied some strings that it had to cleanup.

Well this still does not tell anything about how the list is being used, What their element type is or if they have a change notification attached (because all that influences the performance). I am asking because I have been spending quite some time looking into the rapid.generics code and the 10.3 system.generics.collections improvements and did some own for spring4d. And what makes testing with isolated (micro)benchmarks kinda difficult is the fact that often enough the hardware effect kicks in and shows you big differences that are just there in this benchmark but in real code completely irrelevant or even contradictory.

Show the benchmark code please - I have seen so much flawed benchmarks (including some of my own) in the past months that I don't believe posted results anymore.

The bad thing about the IDE theming is that it's not even using the default VCL theming but a different one hacked into the IDE code because they did not find a better way than that to avoid the form designer being themed. That and doing obvious bad custom drawing such as the search bar in the title bar which keeps jumping around when you move the window. Also not a fan of the dark theme in RAD Studio - in Visual Studio and Visual Studio Code I use it all the time - simply because its the default and it works just nice - the default colors in the code editor fit nicely and the entire IDE is still very reactive. And even though there are also places where some UI controls are not themed in dark (like the project properties) it still works well together without cut off controls and useless scroll bars because some control is 2 pixels to wide.

Google for C++ template code bloat and you see that they also suffer from the very same problem including ridiculously long compile times and memory consumption. The suggested approach is very similar to what has been done in System.Generics.Collections. Delphi however adds some more problems into the mix like having RTTI turned on by default which is extremely problematic if you have extensive generic class hierarchies. For generic fluent APIs that return generic type which have many different type parameters because of the way the API is being used this can turn a few hundred lines unit into several megabytes dcu monsters (the size itself is not the problem but the compiler churning on them for a long time). If you multiply that with other factors it turns compiling a 330K LOC application into a minute or more while consuming close to 2 GB RAM and producing 250MB of dcus and 70MB exe. These are real numbers from our code and an ongoing refactoring of both sides - library code that contains generics (Spring4D) and calling side reduces this significantly.

That it what I looked into some years ago and started refactoring out some for me unnecessary code and cruft. You have to know that a coverage profiler is basically a debugger that puts a breakpoint into every single executable line (or at least those where you specified you are interested in). Now finding out what lines that are is the interesting part - the other is presenting the data nicely inside the IDE when executed under TestInsight (at least for me).

And not your job but Embarcadero's :)

Looks to me very much like a timing problem on shutdown. As soon as I debug or cause some slight delay before the end the AV disappears.

Is it worth the time?

This clearly seems to be a glitch in the compiler when doing the operator/overload resolution where it does find a proper match for Equal/NotEqual (did not test all possible operator combinations but those two).

System.Generics.Collections does not cause that much of a code bloat since the refactorings in XE7 - however it still causes more than it should but that is the limitation of the compiler. I did some tests with Rapid.Generics and while they are optimized for some scenarios it was not a stellar improvement over System.Generics.Collections in 10.3. And while I was doing benchmarks of those and Spring4D collections I saw that isolated benchmarks are often very much affected by certain CPU specifics - on different CPUs depending on their (non documented) behavior of the branch predictor and of course in a microbenchmark chances are high that all code fits into at least L2 cache.

Sorry I meant startIndex + Length(searchText) - 1 because the method simply works as follows: it starts to search at startIndex but does not consider any character in the string after that. That causes it not finding any searched text that extends beyond that index. So as Christian already stated it basically works as if you cut the string after startIndex. So what the documentation is simply missing is the fact that it would not find any occurence that extends past startIndex when you use any of those overloads, that's all.

The documentation is not precise enough - but fwiw the implementation as in Delphi is the same as it is in .NET or Java. The startindex actually also limits the length of the searched string. So any search text longer than 1 will only be found at startIndex - (Length(searchText) - 1) or left to that. And when Cristian said "first occurence" he actually meant "first occurence from the right" of course.

When used in a set, the ordinal value of the enum is basically the index of the bit used within the set and as they can only be 256 bit in size any ordinal value above 255 prevents an enum being used as a set. Personally I think you should rather avoid giving enums any ordinal values and only ever use them when using some interop with another system where they are used for.

If we just had some easy to use and nicely integrated into the IDE way to measure code coverage ...

Among a few minor things it allows for a more declarative approach of writing tests by using attributes to provide the test data. In a classic DUnit test if you have to test an algorithm with 20 different value combinations you need to write 20 tests (either as different methods or by putting them into one test). However DUnit can be pimped to allow the same so you don't need to migrate (*) - since DUnit is kinda abandoned development wise Vincent decided to roll his own library instead of modifying and possibly cleaning up the code from DUnit which dates way back. (*) I did that way back even before DUnitX existed: https://stackoverflow.com/a/9006662/587106 and later also put that into a unit of Spring4D. As you can see in the screenshot you have multiple tests shown although there is only one method declared the extension takes care of producing them so you can run them individually (if for example certain data produces a failure) while you are fixing it. Another thing (which personally never bothered me much) is that with DUnitX you can write testcase classes without inheriting from a base class (this can also be done with an extension for DUnit - did it but never put it anywhere because it was not useful for me). The last thing that I remember is the fact that DUnitX can do is have fixture setup/teardown - they only run once even if there are multiple tests in the class - classic DUnit runs Setup/TearDown methods before and after every single test - guess what? You can also plug that onto DUnit. My personal opinion: do automated unit tests but it does not matter what framework you are using. I use DUnit with the before mentioned extensions and we do so at work because its powerful and compact. And regardless which one you are using - use them with TestInsight and make them part of your CI. 😉

Wasted effort imo - the fundamental design and architecture of the IDE and how (designtime)packages work is flawed to get something like this working properly. Unloading of packages often does not work properly because they are not written for that or some things that happen during loading cannot be easily undone (sloppy developer or simply because the API might be missing). Adding component/library to the IDE is not just loading a module but adding paths in numerous places, like library path for each supported platform, browsing path for stepping into code or even special debug DCU path. And last but not least because there is no strict rule how packages and files should be structured and if the library path should only point to DCUs or to pas files which often causes vendors to place their stuff all over the place and even require modifying the windows path variable (OMG how I hate that!)

You should take a break - because it in fact is related to the ordinal value. The order is wtf, plough, foo, bar - but your array was wtf, plough, bar, foo

Confusing sets with arrays because they look similar is the pitfall It does not, bar and foo are still reversed 😉

That is why I wrote "strictly speaking" because there is always a difference between a formal language or syntax specification and its implementation based on its actual type. Currently probably every language that has some kind of for-in or foreach loop of course operates an array in order but that does not mean that you can assume this for every type operable with such loops. And when we are talking about "just for arrays" this is what is called "implementation detail".

AFAIK you cannot inline declare an array in a for in loop so having some elements in between square brackets will always be a set. You can see it in your very own code on your blog article where you had to put them into an array variable first to then process them in the loop. The point I agree with though is that the fact that sets and arrays share its syntax might be confusing sometimes.

I could answer that by quoting your previous comment 😉

Strictly speaking assuming a specific order in a for-in loop is wrong to begin with.