Leaderboard

Have you tried in-memory datasets? There's one included with Firedac. If you are manageing multiple lists of 10,000 records and indexed dataset might be worth adding to your speed test.

I'm refactoring usage of various different look-up tables and I'm trying to come with 1 fast look-up table design. Most of them have 1 thing in common: search by integer (ID) and search by string (Name). All data is sorted. So, I bench-marked TArray.BinarySearch, TDictionary and custom binary search. Even though a flaw in design was pointed out for TArray.BinarySearch, (see https://en.delphipraxis.net/topic/4575-why-is-tarraybinarysearch-slower-than-normal-binary-search-function/), I still wanted to compare results: For search by integer, custom binary search seems to be a little faster than TDictionary, until large tables: For search by string, custom binary search wins even more, still for not large tables: Interesting is that for smaller searches, 10 records, a sequential search is faster than TDictionary. I also tried TGpStringHash (by Primoz Gabrielcic), which is very fast, but unfortunately no implementation for integers. TSynDictionary (from mORMot) is also very fast, but I don't use mORMotand license is not friendly for my commercial project. Is there any other fast search implementations that is not too specialized and ready to use out of the box, that I can try?

It's an improvement on a Trie. The Dr Dobbs article (still on line at https://www.drdobbs.com/database/ternary-search-trees/184410528) claims in their Conclusion, "Ternary search trees are efficient and easy to implement. They offer substantial advantages over both binary search trees and digital search tries. We feel that they are superior to hashing in many applications.". See also:

As David wrote, mORMot has a 3 license - if you use MPL it is very commercial-project-friendly. TL&LR: Nothing to pay, just mention somewhere in your software that you used it, and publish any modification you may do to the source code. Another article worth looking at: https://www.delphitools.info/2015/03/17/long-strings-hash-vs-sorted-vs-unsorted/ It depends what you expect. Also note that for long strings, hashing may have a cost - this is why we implemented https://blog.synopse.info/?post/2021/02/12/New-AesNiHash-for-mORMot-2

You still have not said what your use case is. My own need for fast string lookup (and finding either an integer or a pointer), is principally to load once, lookup many times, and I am not too concerned about memory usage or load time. The task usually is to handle up to a couple of hundred strings of about 25 to 200 characters. For this I have long used a ternary search tree, which I first learnt about from Dr Dobbs Journal, April 1998. I adapted the C version for Delphi 1, I think, and by now the lookup supports Unicode strings I cannot post useful timings, because you cannot compare my environment to yours, but I just did a test load of quarter of a million random 64-character strings into the tree in 390ms. The speed of lookup is what matters to me and this is too fast to bother to benchmark it. But those 250k strings (far more than I need in practice) took a tree size of 116MB. Whether this search method would be useful to you depends on how you value load speed and memory against lookup time. Lookup time may not the only thing you should be considering.

You can license mormot under MPL which is fine for use in a commercial product.

No surprise there - linear search is O(n), binary search is O(log n) and hash table is O(1), what you see with the differences for certain number of records are the different constant factors - building a hash has a certain cost which is why TDict is slower up to a certain point - this depends on the hash algorithm being used - indirections being caused by extra function calls like IEqualityComparer.GetHashCode and the memory layout of the hash table. You can even see that it's not strictly O(1) as it gets a little slower the more records you have which I would guess is due to memory layout within the hashtable where at a certain point memory access time kicks in because the data does not fit in the fast cache anymore (access pattern of the benchmarking code matters there as well). Without specifying the exact use cases it's hard to give any suggestions - is the data one time fill and then only lookup, is data being changed after - adding/removing records, is data within the records changing. Whats the typical lookup pattern - is data being looked up randomly or in certain patterns (such as ID or Name sequences in certain order). If you aim for maximum performance a custom data structure and algorithm will always win over some standard algorithm by a few percent - the question is if you want to do all that work including risking bugs for that negligible few percent or use some standard data structures and algorithms that are proven to work and easy to be applicable.

Um... no. To be "nicer", all the replacement code would have to be better than the existing Delphi code. At least much of the current Delphi code is stable. Rewrite your biggest project in a hurry and see if it's better.

@David MillingtonMaybe you could enlighten us as to the nature of the changes that made the Delphi IDE behave better with RD. Are there any lessons to be learnt by Delphi developers?

So what should the standard reaction be? I code my custom controls using my own double buffering and ignore the vcl's double buffering, which tends to be rather buggy and variable (different implementations of it throughout the vcl) - especially when vcl themes are enabled.

Just two notices. If you will ever use Oracle and RetIDList will contain more than 1000 elements, the code will fail. I don't know if any other RDBMS has this limitation though. How trusted is the file? Taking a string value from somewhere and putting it in a SQL command exposes your application to injection attacks.

var jsonString = '{"x": {"y": [2,3]}, "t": [3,5]}'; var obj = JSON.parse(jsonString); var arr = []; arr.push(obj.x.y) arr.push(obj.t) console.log(arr); I was wondering the same thing when I was working on https://www.slotozilla.com/free-slots/silver-lion .Don't create your json yourself, most languages have functions to turn your object/array into a valid json, so just create your object as you normally do and then use that function as in Javascript JSON.stringify().

*facepalm* Winsock is included into RTL and you can create client-server apps with it. What a stupid policy

@emailx45 It is great that you are enthusiastic and helpful - but - the wrong kind of help is not really helpful, is it? I guess you missed the "/rant" in my original post, as well as the comment about "logical, but annoying", and the explicit reference to string literals. I also clearly stated WHY I wanted a longer string literal; and @Angus Robertson provided another example of when this limit is a challenge. Yet you post a completely irrelevant suggestion that AnsiString is the answer to the problem. My advice: - Study posts until you are sure you understand the context - Avoid applying your own opinion as one of the Embarcadero engineers - Avoid what-about-ism - Avoid characterizing the desires of others as insane - Avoid posting bullshit comments not relevant to the discussion Fun fact: The line length limit in the Delphi IDE is 4096 chars.

True. Normally you would for readability, but splitting involves a manual operation that carries a risk of mangling the content. Yes, it is a compiler limit, but when you can split and add up a huge string "manually", it is annoying that the compiler can't hide this for you. /rant ended

Although it is true that the compiler ought to catch such things, it is also true that - Delphi declares type names like TIntArray - we should declare like TwmIntArray The more the Delphi types increase, the greater the likelihood of collisions with built-in types. Naming is hard, yes, because too often names are assigned thoughtlessly.

We've all been saying this for years and years. My old boss first said this to me 20+ years ago. We are so so so far behind.

I am temporarily disabled due to a fractured (bone fully severed at 75° acute angle) upper left arm after slipping on black ice. A nice clean fracture according to the doctors. No surgery or fancy stabilization planned, and the doctors want me to let the body handle it on its own, varying between letting it hang free and using a sling. Typing one handed on Android is somewhat frustrating, so I'll be less active until the worst pain phase is over.