Arnaud Bouchez

This was time for a new "stable" release of the Open Source mORMot 2 framework! You can find it on our GitHub repository: https://github.com/synopse/mORMot2/releases/tag/2.3.stable 🙂 This release will be a Long Term Support Security Branch, as we will maintain it for the next years for main bugs and security fixes. Added - Swagger/OpenAPI Client Generator - IDocList/IDocDict Containers - SID/DACL/SACL/SDDL/ACE Security Objects - async web server: IOCP support on Windows, metrics gathering and standard logging - TSynMustache can work on plain data via RTTI, in addition to TDocVariant - introducing TRttiMap for DTO process. Changed - Upgraded SQLite3 to 3.46.1 - Enhancements to the LDAP client, HTTP/HTTPS client, Kerberos auth, Peer Cache, ORM. - Lots other bug fixes, optimisations and enhancements. More information on our blog: https://blog.synopse.info/?post/2024/10/16/Release-of-mORMot-2.3-Stable

Personal biais: the mORMot 2 Open Source framework has a very efficient JSON library, and several ways to use it: - from RTTI, using classes, records, collections, dynamic arrays, mORMot generics... - from variants, and a custom "document" variant type to store JSON objects or arrays... - from high-level IDocList / IDocDict holders. See https://blog.synopse.info/?post/2024/02/01/Easy-JSON-with-Delphi-and-FPC It is perhaps the fastest library available, working on Delphi and FPC, with unique features, like: list := DocList('[{"a":10,"b":20},{"a":1,"b":21},{"a":11,"b":20}]'); // sort a list/array by the nested objects field(s) list.SortByKeyValue(['b', 'a']); assert(list.Json = '[{"a":10,"b":20},{"a":11,"b":20},{"a":1,"b":21}]'); // create a dictionary from key:value pairs supplied from code dict := DocDict(['one', 1, 'two', 2, 'three', _Arr([5, 6, 7, 'huit'])]); assert(dict.Len = 3); // one dictionary with 3 elements assert(dict.Json = '{"one":1,"two":2,"three":[5,6,7,"huit"]}'); // convert to JSON with nice formatting (line feeds and spaces) Memo1.Caption := dic.ToString(jsonHumanReadable); // integrated search / filter assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two');

@David Schwartz Command line tool is now available: https://github.com/synopse/mORMot2/tree/master/src/tools/mopenapi (this is a good showcase how cross-platform and cross-compiler command line switches parsing could be done with mORMot) I will release a Win32 binary somewhere in the next days. Thanks for the feedback!

This is not yet clean OOP for sure, since it would break the Liskov principle. IMHO there is no way to make it "elegant". My guess is that the "elegant" way is to use composition. That is, compose a "shared object", available to your function, which would have a circle and a rectangle properties, then additional properties. OOP should be as natural as possible. If you are fighting against your types, some refactoring may be needed. Multiple inheritance is IMHO never mandatory, for a clean OOP model. And always refer the the SOLID principles!

Since it seems to be a trending subject, a new OpenAPI client generator is now available for Delphi: https://blog.synopse.info/?post/2024/09/06/Swagger/OpenAPI-Client-Generator-for-Delphi-and-FPC ;D Here are some features of our OpenAPI code generator for Delphi and FPC: - Fully OpenSource, both generator and generated source code - Use high-level pascal records and dynamic arrays for "object" DTOs - Use high-level pascal enumerations and sets for "enum" values - Translate HTTP status error codes into high-level pascal Exceptions - Recognize similar "properties" or "enum" to reuse the same pascal type - Support of nested "$ref" for objects, parameters or types - Support "allOf" attribute, with proper properties inheritance/overloading - Support "oneOf" attribute, for strings or alternate record types - Support of "in":"header" and "in":"cookie" parameter attributes - Fallback to variant pascal type for "oneOf" or "anyOf" JSON values - Generated source code units are very small and easy to use, read and debug - Can generate very detailed comment documentation in the unit source code - Tunable engine, with plenty of generation options (e.g. about verbosity) - Tested with several Swagger 2 and OpenAPI 3 reference content But still not fully compliant to all existing files: feedback is welcome! Here is a snippet of a generated method signature, with high-level Data Transfer Objects as records (TUserShort/TDBAccount), and complete documentation, with proper Exceptions generation for the HTTP results error codes: // post_account_res_add_grant_auth [post] /accounts/{uuid}/add-grant-auth/ // // Summary: Gives a user permissions to grant authorization on the account // Description: // Roles: vm_admin for vm object, templates otherwise // // Params: // - [path] Uuid (required): Hypervisor uuid // - [body] Payload (required) // // Responses: // - 200 (main): Success // - 400 [EValidationErrorResponse]: Parameters have invalid format or type // - 401 [EUnauthorizedResponse]: User is not authenticated // - 403 [EForbiddenResponse]: User has insufficient permissions // - 404 [EResourceNotFoundError]: Hypervisor not found // - 422 [EIntegrityErrorResponse]: Parameters have valid format but are not compatible // with the server state function PostAccountResAddGrantAuth(const Uuid: string; const Payload: TUserShort): TDbAccount; The generated code is compatible even with oldest Delphi (7-2009) - nice to add native REST/HTTP client abilities to an existing monolithic application. It is even fully compatible with FPC, because we should not be tied to a single compiler. The source code of the generator is a single unit which leverages the mORMot RTTI and JSON kernel for its internal plumbing. And you will find several generated sample code units in the blog article, to make yourself your idea about the level of this little unit. Feedback is welcome!

I have edited the blog post to make it clear:

Indeed. You need some basic units of mORMot for the client code, because it uses its HTTP, RTTI and JSON cross-platform and cross-compiler abilities. But are not tied to use the other mORMot ORM or SOA features, which are uncoupled from this. To use the client methods, you just use the generated class methods, as with regular Delphi code. No fancy mORMot usage for the end user: the rodent stays hidden in his hole. You have plenty of output sample in the blog article. For instance: https://gist.github.com/synopse/e0dacfcc870db67013de55e43276f07b unit auth.api; {$I mormot.defines.inc} interface { -------------------------------------------------------------------- AUTHENTIQ API client as TAuthClient class Generated 6 Sep 2024 by ab via mormot2tests - DO NOT MODIFY BY HAND! -------------------------------------------------------------------- } uses classes, sysutils, mormot.core.base, mormot.core.unicode, mormot.core.text, mormot.core.buffers, mormot.core.datetime, mormot.core.rtti, mormot.core.json, mormot.core.variants, mormot.net.client;

Just to inform anyone looking for an OpenAPI client code generator, that I just made a new one, for both Delphi and FPC. It seems to be more advanced than Wagner's generator (e.g. it converts errors, enums and allOf/oneOf attributes), and is fully Open Source. https://blog.synopse.info/?post/2024/09/06/Swagger/OpenAPI-Client-Generator-for-Delphi-and-FPC Here are the top features of this OpenAPI client code generator for Delphi and FPC: Use high-level pascal records and dynamic arrays for "object" DTOs Use high-level pascal enumerations and sets for "enum" values Translate HTTP status error codes into high-level pascal Exceptions Recognize similar "properties" or "enum" to reuse the same pascal type Support of nested "$ref" for objects, parameters or types Support "allOf" attribute, with proper properties inheritance/overloading Support "oneOf" attribute, for strings or alternate record types Support of "in":"header" and "in":"cookie" parameter attributes Fallback to variant pascal type for "oneOf" or "anyOf" JSON values Each method execution is thread-safe and blocking, for safety Generated source code units are very small and easy to use, read and debug Can generate very detailed comment documentation in the unit source code Tunable engine, with plenty of generation options (e.g. about verbosity) Leverage the mORMot RTTI and JSON kernel for its internal plumbing Compatible with FPC and oldest Delphi (7-2009) Tested with several Swagger 2 and OpenAPI 3 reference content, but your input is welcome, because it is not fully compliant! Hoping you may find it interesting. The blog article has several example of the actual output of the generator, from several sources. It is the kind of information I would have wanted to see with other libraries. Use the source, Luke!

Two last weeks of the challenge... Still time to post your own entry!

And don't forget the command line parser available in mORMot 2. - Works on all OS; - Works on Delphi and FPC; - Can auto-generate the help content with proper formatting; - Minimal code writing. In practice, resulting code is short and efficient: https://github.com/synopse/1brc-ObjectPascal/blob/main/entries/abouchez/src/brcmormot.lpr#L446

This is a "classical" problem with Delphi, overloaded functions, and floating points values. IMHO using currency is an awful workaround here, and should not be kept in any serious codebase. It works "by chance". Another workaround, still using the Math unit, is to make the computation in two steps: function RoundExDouble(x: double): Double; begin x := x * 10; Result := Ceil(x) / 10; end; function RoundExInteger(x: double): Integer; begin x := x * 10; Result := Ceil(x); end; So the expected Ceil(double) overload is clearly picked up by the compiler. Or define our own unique Ceil() function, which does not depend on Math.pas and ensure we use the right value type: function ceil(x: double): integer; begin result := trunc(x) + ord(frac(x) > 0); end; This is this solution I used in my entry of the challenge: stay away from Math.pas. 😉

I missed the info, sorry. For a SQL solution, it is very good. Out of curiosity, how much memory does it need for the 16.5 GB file? Does it use SQLite3 and its virtual tables internally for its SQL dialect (something like https://www.sqlite.org/csv.html)?

Easy, and wrong. You are reading the station weathers reference data with one row per station. And making a min/max/average of a single data per station. The challenge is to read a 1 billion (1,000,000,000) rows of CSV data for all those 41343 stations, and compute it. There is a generator of a 16GB CSV file to inject and process. So 0.33s for 41343 rows would make around 8000 seconds, i.e. 5.5 days.

Since years, our Open Source mORMot framework offers several ways to work with any kind of runtime arrays/objects documents defined, e.g. via JSON, with a lot of features, and very high performance. Our TDocVariant custom variant type is a powerful way of working with such schema-less data, but it was found confusing by some users. So we developed a new set of interface definitions around it, to ease its usage, without sacrificing its power. We modelized them around Python Lists and Dictionaries, which is proven ground - with some extensions of course. Two one-liners may show how our mORMot library is quite unique in the forest/jungle of JSON libraries for Delphi (and FPC): +] assert(DocList('[{ab:1,cd:{ef:"two"}}]')[0].cd.ef = 'two'); assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two'); Yes, this code compiles and run on Delphi - even the antique Delphi 7. 😆 If you compare e.g. to how the standard JSON library works in the RTL, with all its per-node classes, you may find quite a difference! And there is no secret in regard to performance, that our JSON processing is fast. More info: https://blog.synopse.info/?post/2024/02/01/Easy-JSON-with-Delphi-and-FPC

OpenSource unit https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.zip.pas supports deletion - just use TZipWrite.CreateFrom() with a list of files to ignore when reading the existing zip.

You may try to implement it properly, with all the appropriate testing coverage, and I guess you will find out. There are name/value parsing, execution timing, and multiple parameters involved. Existing implementations in Delphi are not widely used nor maintained. Not what we could call a "standard".

@FaFaFooey Argon2 is nice for sure - but really complex, and difficult to implement. So I am not sure it is "the standard of storing a password hash" on Delphi, in which I was not able to easily find a full-features library. I like very much the https://en.wikipedia.org/wiki/Proof_of_work concept: Proof-of-Work Puzzle: the nonce is used as the puzzle for a Proof-of-Work challenge which the client has to solve (as part of anti-brute force protection). A simple scheme can be to send a nonce and ask the client to find a string that, when appended to your nonce, will have a SHA-256 hash beginning or ending with a certain number of zeroes (adjusting the number of zeroes allows adjusting the puzzle difficulty). When getting a puzzle response, check first the nonce validity, then the hash. See https://www.delphitools.info/2016/01/07/nonces-and-tokens for the diverse means of authenticating an user.

With this new year, it was time to make a mORMot 2 release. We went a bit behind the SQlite3 engine, and we added some nice features. 😎 Main added features: - OpenSSL 3.0/3.1 direct support in addition to 1.1 (with auto-switch at runtime) - Native X.509, RSA and HSM support - mget utility (wget + peer cache) Main changes: - Upgraded SQLite3 to 3.44.2 - Lots of bug fixes and enhancements (especially about cryptography and networking) - A lot of optimizations, so that we eventually reached in top 12 ranking of all frameworks tested by TFB https://github.com/synopse/mORMot2/releases/tag/2.2.stable 🙂

Today, almost all computer security relies on asymmetric cryptography and X.509 certificates as file or hardware modules. And the RSA algorithm is still used to sign the vast majority of those certificates. Even if there are better options (like ECC-256), RSA-2048 seems the actual standard, at least still allowed for a few years. So we added pure pascal RSA cryptography and X.509 certificates support in mORMot 2. Last but not least, we also added Hardware Security Modules support via the PKCS#11 standard. Until now, we were mostly relying on OpenSSL, but a native embedded solution would be smaller in code size, better for reducing dependencies, and easier to work with (especially for HSM). The main idea is to offer only safe algorithms and methods, so that you can write reliable software, even if you are no cryptographic expert. 😉 More information in our blog article about this almost unique features set in Delphi (and FPC): https://blog.synopse.info/?post/2023/12/09/Native-X.509-and-RSA-Support

Last year 2023 was perhaps not the best ever, and, just after Christmas, we think about all people we know still in war or distress. But in the small mORMot world, 2023 was a fine millesima. A lot of exciting features, a pretty good rank in benchmarks, and a proof of being ready for the next decade. For this new year, we would like to introduce you to a new mORMot baby: the mget command line tool, a HTTP/HTTPS web client with peer-to-peer caching. It is just a wrapper around a set of the new PeerCache feature, built-in the framework web client class - so you can use it in your own projects if you need to. More information about this Open Souce tool and its source code: https://blog.synopse.info/?post/2024/01/01/Happy-New-Year-2024-and-Welcome-MGET Happy new year, Delphi coders!

@Edwin Yip Yes, we will try to make a TLS 1.3 layer beginning of this new 2024 year. 😉

This is nothing about salting. @Kas Ob. wrote it fvery nicely: this is because the AES encoding algorithm is a block algorithm by itself: it works on blocks of 16 bytes as input and output. It works only on full blocks, i.e. encoded output is always a multiple of 16 bytes. So you need a way of "cutting" the result without leaking any information. This is known as "padding" the output - PKCS#7 defined the most common way: you output up to 16 more bytes, in which you fill the remaining bytes of your data with the length of your data modulo 16. So once decrypted, you can "cut" back the result into the original size. For some chaining modes (like AES-CTR) there are some other ways of padding with no additional output bytes, but it is less common. My point was: do not reinvent the wheel, or you are likely to obtain a weak safety.

Some libraries are not part of GetIt for obscure reasons within Embarcadero decision makers, e.g. our https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.core.pas which is probably the fastest AES library able to be statically linked, OpenSource with full source code, and highly maintained, on Delphi (and FPC). And before using AES in your project, ensure you understand its basics: - it is very easy to make something weak, so you need to know a little about today's best practices. - don't reinvent the wheel: if you want to transmit data, use TLS; if you want to store passwords in a DB, don't use encryption but salted hashes which can't be reversed; see if using OS level disk encryption (like BitLocker) is not a better approach; etc... - don't use direct/naive AES in your project, named ECB because it is weak https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_(ECB) - consider AES-CTR (or AES-GCM which also makes authentication) for instance. - there is also a need for 16-bytes padding of the encoded output - here Pkcs7 seems the de-facto standard. - something tricky is to create the 128-bit or 256-bit key: you need to have a strong derivation function, like PBKDF2 from human input or at least HMAC-SHA256 if you already have a secret key. - consider also asymmetric encryption: if you can use ECDHE derivation using public keys in your project, this is the way to go - it is much safer to share two public keys between peers and derivate a secret, than sharing a symmetric key, which can be leaked. - a proper IV filling is also essential: a common practice it to generate it from random, and put it at the beginning of the encrypted stream, or even better get this IV from some bits of the ECDHE derivation function. - in practice, AES with 128-bit is enough security for an application - it has more security that RSA-2048 certificates which are still used almost everywhere in the Internet. Our units allows all this, and much more.

This is funny, a few weeks ago I added advanced cross-platform support of network adapters in the mORMot 2 network layer. It can retrieve all mac addresses of the network interfaces, and also much more information like each kind of adapter, its MTU, speed or gateway/broadcast IPs. And it is cross-platform, so it works not only on Windows but also on POSIX (tested on MacOS and Linux). https://github.com/synopse/mORMot2/blob/fe3fdf53e54dc770bda8d9858c8d6ff5ebf4ac4d/src/net/mormot.net.sock.pas#L517

@Edwin Yip Not yet, the TLS layer is not yet available. But I would not use Indy anyway, but the mORMot direct client classes instead, which already allows both OpenSSL and SSPI so you could use the latest TLS standard on the latest Windows revision. 😉