Arnaud Bouchez

And don't forget the command line parser available in mORMot 2. - Works on all OS; - Works on Delphi and FPC; - Can auto-generate the help content with proper formatting; - Minimal code writing. In practice, resulting code is short and efficient: https://github.com/synopse/1brc-ObjectPascal/blob/main/entries/abouchez/src/brcmormot.lpr#L446

This is a "classical" problem with Delphi, overloaded functions, and floating points values. IMHO using currency is an awful workaround here, and should not be kept in any serious codebase. It works "by chance". Another workaround, still using the Math unit, is to make the computation in two steps: function RoundExDouble(x: double): Double; begin x := x * 10; Result := Ceil(x) / 10; end; function RoundExInteger(x: double): Integer; begin x := x * 10; Result := Ceil(x); end; So the expected Ceil(double) overload is clearly picked up by the compiler. Or define our own unique Ceil() function, which does not depend on Math.pas and ensure we use the right value type: function ceil(x: double): integer; begin result := trunc(x) + ord(frac(x) > 0); end; This is this solution I used in my entry of the challenge: stay away from Math.pas. 😉

I missed the info, sorry. For a SQL solution, it is very good. Out of curiosity, how much memory does it need for the 16.5 GB file? Does it use SQLite3 and its virtual tables internally for its SQL dialect (something like https://www.sqlite.org/csv.html)?

Easy, and wrong. You are reading the station weathers reference data with one row per station. And making a min/max/average of a single data per station. The challenge is to read a 1 billion (1,000,000,000) rows of CSV data for all those 41343 stations, and compute it. There is a generator of a 16GB CSV file to inject and process. So 0.33s for 41343 rows would make around 8000 seconds, i.e. 5.5 days.

Since years, our Open Source mORMot framework offers several ways to work with any kind of runtime arrays/objects documents defined, e.g. via JSON, with a lot of features, and very high performance. Our TDocVariant custom variant type is a powerful way of working with such schema-less data, but it was found confusing by some users. So we developed a new set of interface definitions around it, to ease its usage, without sacrificing its power. We modelized them around Python Lists and Dictionaries, which is proven ground - with some extensions of course. Two one-liners may show how our mORMot library is quite unique in the forest/jungle of JSON libraries for Delphi (and FPC): +] assert(DocList('[{ab:1,cd:{ef:"two"}}]')[0].cd.ef = 'two'); assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two'); Yes, this code compiles and run on Delphi - even the antique Delphi 7. 😆 If you compare e.g. to how the standard JSON library works in the RTL, with all its per-node classes, you may find quite a difference! And there is no secret in regard to performance, that our JSON processing is fast. More info: https://blog.synopse.info/?post/2024/02/01/Easy-JSON-with-Delphi-and-FPC

OpenSource unit https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.zip.pas supports deletion - just use TZipWrite.CreateFrom() with a list of files to ignore when reading the existing zip.

You may try to implement it properly, with all the appropriate testing coverage, and I guess you will find out. There are name/value parsing, execution timing, and multiple parameters involved. Existing implementations in Delphi are not widely used nor maintained. Not what we could call a "standard".

@FaFaFooey Argon2 is nice for sure - but really complex, and difficult to implement. So I am not sure it is "the standard of storing a password hash" on Delphi, in which I was not able to easily find a full-features library. I like very much the https://en.wikipedia.org/wiki/Proof_of_work concept: Proof-of-Work Puzzle: the nonce is used as the puzzle for a Proof-of-Work challenge which the client has to solve (as part of anti-brute force protection). A simple scheme can be to send a nonce and ask the client to find a string that, when appended to your nonce, will have a SHA-256 hash beginning or ending with a certain number of zeroes (adjusting the number of zeroes allows adjusting the puzzle difficulty). When getting a puzzle response, check first the nonce validity, then the hash. See https://www.delphitools.info/2016/01/07/nonces-and-tokens for the diverse means of authenticating an user.

With this new year, it was time to make a mORMot 2 release. We went a bit behind the SQlite3 engine, and we added some nice features. 😎 Main added features: - OpenSSL 3.0/3.1 direct support in addition to 1.1 (with auto-switch at runtime) - Native X.509, RSA and HSM support - mget utility (wget + peer cache) Main changes: - Upgraded SQLite3 to 3.44.2 - Lots of bug fixes and enhancements (especially about cryptography and networking) - A lot of optimizations, so that we eventually reached in top 12 ranking of all frameworks tested by TFB https://github.com/synopse/mORMot2/releases/tag/2.2.stable 🙂

Last year 2023 was perhaps not the best ever, and, just after Christmas, we think about all people we know still in war or distress. But in the small mORMot world, 2023 was a fine millesima. A lot of exciting features, a pretty good rank in benchmarks, and a proof of being ready for the next decade. For this new year, we would like to introduce you to a new mORMot baby: the mget command line tool, a HTTP/HTTPS web client with peer-to-peer caching. It is just a wrapper around a set of the new PeerCache feature, built-in the framework web client class - so you can use it in your own projects if you need to. More information about this Open Souce tool and its source code: https://blog.synopse.info/?post/2024/01/01/Happy-New-Year-2024-and-Welcome-MGET Happy new year, Delphi coders!

@Edwin Yip Yes, we will try to make a TLS 1.3 layer beginning of this new 2024 year. 😉

This is nothing about salting. @Kas Ob. wrote it fvery nicely: this is because the AES encoding algorithm is a block algorithm by itself: it works on blocks of 16 bytes as input and output. It works only on full blocks, i.e. encoded output is always a multiple of 16 bytes. So you need a way of "cutting" the result without leaking any information. This is known as "padding" the output - PKCS#7 defined the most common way: you output up to 16 more bytes, in which you fill the remaining bytes of your data with the length of your data modulo 16. So once decrypted, you can "cut" back the result into the original size. For some chaining modes (like AES-CTR) there are some other ways of padding with no additional output bytes, but it is less common. My point was: do not reinvent the wheel, or you are likely to obtain a weak safety.

Some libraries are not part of GetIt for obscure reasons within Embarcadero decision makers, e.g. our https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.core.pas which is probably the fastest AES library able to be statically linked, OpenSource with full source code, and highly maintained, on Delphi (and FPC). And before using AES in your project, ensure you understand its basics: - it is very easy to make something weak, so you need to know a little about today's best practices. - don't reinvent the wheel: if you want to transmit data, use TLS; if you want to store passwords in a DB, don't use encryption but salted hashes which can't be reversed; see if using OS level disk encryption (like BitLocker) is not a better approach; etc... - don't use direct/naive AES in your project, named ECB because it is weak https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_(ECB) - consider AES-CTR (or AES-GCM which also makes authentication) for instance. - there is also a need for 16-bytes padding of the encoded output - here Pkcs7 seems the de-facto standard. - something tricky is to create the 128-bit or 256-bit key: you need to have a strong derivation function, like PBKDF2 from human input or at least HMAC-SHA256 if you already have a secret key. - consider also asymmetric encryption: if you can use ECDHE derivation using public keys in your project, this is the way to go - it is much safer to share two public keys between peers and derivate a secret, than sharing a symmetric key, which can be leaked. - a proper IV filling is also essential: a common practice it to generate it from random, and put it at the beginning of the encrypted stream, or even better get this IV from some bits of the ECDHE derivation function. - in practice, AES with 128-bit is enough security for an application - it has more security that RSA-2048 certificates which are still used almost everywhere in the Internet. Our units allows all this, and much more.

This is funny, a few weeks ago I added advanced cross-platform support of network adapters in the mORMot 2 network layer. It can retrieve all mac addresses of the network interfaces, and also much more information like each kind of adapter, its MTU, speed or gateway/broadcast IPs. And it is cross-platform, so it works not only on Windows but also on POSIX (tested on MacOS and Linux). https://github.com/synopse/mORMot2/blob/fe3fdf53e54dc770bda8d9858c8d6ff5ebf4ac4d/src/net/mormot.net.sock.pas#L517

@Edwin Yip Not yet, the TLS layer is not yet available. But I would not use Indy anyway, but the mORMot direct client classes instead, which already allows both OpenSSL and SSPI so you could use the latest TLS standard on the latest Windows revision. 😉