Arnaud Bouchez

Personal biais: the mORMot 2 Open Source framework has a very efficient JSON library, and several ways to use it:

- from RTTI, using classes, records, collections, dynamic arrays, mORMot generics...
- from variants, and a custom "document" variant type to store JSON objects or arrays...
- from high-level IDocList / IDocDict holders.

See https://blog.synopse.info/?post/2024/02/01/Easy-JSON-with-Delphi-and-FPC

It is perhaps the fastest library available, working on Delphi and FPC, with unique features, like:

  list := DocList('[{"a":10,"b":20},{"a":1,"b":21},{"a":11,"b":20}]');
  // sort a list/array by the nested objects field(s)
  list.SortByKeyValue(['b', 'a']);
  assert(list.Json = '[{"a":10,"b":20},{"a":11,"b":20},{"a":1,"b":21}]');

  // create a dictionary from key:value pairs supplied from code
  dict := DocDict(['one', 1, 'two', 2, 'three', _Arr([5, 6, 7, 'huit'])]);
  assert(dict.Len = 3); // one dictionary with 3 elements
  assert(dict.Json = '{"one":1,"two":2,"three":[5,6,7,"huit"]}');
  // convert to JSON with nice formatting (line feeds and spaces)
  Memo1.Caption := dic.ToString(jsonHumanReadable);

  // integrated search / filter
  assert(DocList('[{ab:1,cd:{ef:"two"}}]').First('ab<>0').cd.ef = 'two');

@David Schwartz

Command line tool is now available:
https://github.com/synopse/mORMot2/tree/master/src/tools/mopenapi

(this is a good showcase how cross-platform and cross-compiler command line switches parsing could be done with mORMot)

I will release a Win32 binary somewhere in the next days.

Thanks for the feedback!

This is not yet clean OOP for sure, since it would break the Liskov principle.

IMHO there is no way to make it "elegant".

My guess is that the "elegant" way is to use composition.

That is, compose a "shared object", available to your function, which would have a circle and a rectangle properties, then additional properties.

OOP should be as natural as possible.

If you are fighting against your types, some refactoring may be needed.

Multiple inheritance is IMHO never mandatory, for a clean OOP model.

And always refer the the SOLID principles!

I have edited the blog post to make it clear:

Quote

Of course, you would need some basic mORMot units in your client code. The tool does not generate a "pure Delphi RTL" client. But to be fair, there was no JSON support in early Delphi, and maintaining the differences between versions of compilers and RTL, especially about JSON, RTTI, HTTP would end up in something as reinventing the mORMot. We just use what works.
Note that the generated client code does not depend at all of the other mORMot features, like ORM or SOA. It is perfectly uncoupled from those very powerful, but sometimes confusing features. With the client code, you will use the mORMot, without noticing it. The rodent will hide in its hole. But if you need it, e.g. for adding logs or services, it would be glad to help you. 🙂

Indeed.

You need some basic units of  mORMot for the client code, because it uses its HTTP, RTTI and JSON cross-platform and cross-compiler abilities.
But are not tied to use the other mORMot ORM or SOA features, which are uncoupled from this.
To use the client methods, you just use the generated class methods, as with regular Delphi code. No fancy mORMot usage for the end user: the rodent stays hidden in his hole.
 

You have plenty of output sample in the blog article.

For instance: https://gist.github.com/synopse/e0dacfcc870db67013de55e43276f07b

unit auth.api;

{$I mormot.defines.inc}

interface

{
  --------------------------------------------------------------------
  AUTHENTIQ API client as TAuthClient class
  Generated 6 Sep 2024 by ab via mormot2tests - DO NOT MODIFY BY HAND!
  --------------------------------------------------------------------
}

uses
  classes,
  sysutils,
  mormot.core.base,
  mormot.core.unicode,
  mormot.core.text,
  mormot.core.buffers,
  mormot.core.datetime,
  mormot.core.rtti,
  mormot.core.json,
  mormot.core.variants,
  mormot.net.client;

Since it seems to be a trending subject, a new OpenAPI client generator is now available for Delphi:
https://blog.synopse.info/?post/2024/09/06/Swagger/OpenAPI-Client-Generator-for-Delphi-and-FPC
 ;D

Here are some features of our OpenAPI code generator for Delphi and FPC:
- Fully OpenSource, both generator and generated source code
- Use high-level pascal records and dynamic arrays for "object" DTOs
- Use high-level pascal enumerations and sets for "enum" values
- Translate HTTP status error codes into high-level pascal Exceptions
- Recognize similar "properties" or "enum" to reuse the same pascal type
- Support of nested "$ref" for objects, parameters or types
- Support "allOf" attribute, with proper properties inheritance/overloading
- Support "oneOf" attribute, for strings or alternate record types
- Support of "in":"header" and "in":"cookie" parameter attributes
- Fallback to variant pascal type for "oneOf" or "anyOf" JSON values
- Generated source code units are very small and easy to use, read and debug
- Can generate very detailed comment documentation in the unit source code
- Tunable engine, with plenty of generation options (e.g. about verbosity)
- Tested with several Swagger 2 and OpenAPI 3 reference content
But still not fully compliant to all existing files: feedback is welcome!

Here is a snippet of a generated method signature, with high-level Data Transfer Objects as records (TUserShort/TDBAccount), and complete documentation, with proper Exceptions generation for the HTTP results error codes:

    // post_account_res_add_grant_auth [post] /accounts/{uuid}/add-grant-auth/
    //
    // Summary: Gives a user permissions to grant authorization on the account
    // Description:
    //   Roles: vm_admin for vm object, templates otherwise
    //
    // Params:
    // - [path] Uuid (required): Hypervisor uuid
    // - [body] Payload  (required)
    //
    // Responses:
    // - 200 (main): Success
    // - 400 [EValidationErrorResponse]: Parameters have invalid format or type
    // - 401 [EUnauthorizedResponse]: User is not authenticated
    // - 403 [EForbiddenResponse]: User has insufficient permissions
    // - 404 [EResourceNotFoundError]: Hypervisor not found
    // - 422 [EIntegrityErrorResponse]: Parameters have valid format but are not compatible
    // with the server state
    function PostAccountResAddGrantAuth(const Uuid: string; const Payload: TUserShort): TDbAccount;

The generated code is compatible even with oldest Delphi (7-2009) - nice to add native REST/HTTP client abilities to an existing monolithic application.
It is even fully compatible with FPC, because we should not be tied to a single compiler.

The source code of the generator is a single unit which leverages the mORMot RTTI and JSON kernel for its internal plumbing.

And you will find several generated sample code units in the blog article, to make yourself your idea about the level of this little unit.

Feedback is welcome!

Just to inform anyone looking for an OpenAPI client code generator, that I just made a new one, for both Delphi and FPC.

It seems to be more advanced than Wagner's generator (e.g. it converts errors, enums and allOf/oneOf attributes), and is fully Open Source.

https://blog.synopse.info/?post/2024/09/06/Swagger/OpenAPI-Client-Generator-for-Delphi-and-FPC

Here are the top features of this OpenAPI client code generator for Delphi and FPC:

• Use high-level pascal records and dynamic arrays for "object" DTOs
• Use high-level pascal enumerations and sets for "enum" values
• Translate HTTP status error codes into high-level pascal Exceptions
• Recognize similar "properties" or "enum" to reuse the same pascal type
• Support of nested "$ref" for objects, parameters or types
• Support "allOf" attribute, with proper properties inheritance/overloading
• Support "oneOf" attribute, for strings or alternate record types
• Support of "in":"header" and "in":"cookie" parameter attributes
• Fallback to variant pascal type for "oneOf" or "anyOf" JSON values
• Each method execution is thread-safe and blocking, for safety
• Generated source code units are very small and easy to use, read and debug
• Can generate very detailed comment documentation in the unit source code
• Tunable engine, with plenty of generation options (e.g. about verbosity)
• Leverage the mORMot RTTI and JSON kernel for its internal plumbing
• Compatible with FPC and oldest Delphi (7-2009)
• Tested with several Swagger 2 and OpenAPI 3 reference content, but your input is welcome, because it is not fully compliant!

Hoping you may find it interesting.

The blog article has several example of the actual output of the generator, from several sources.
It is the kind of information I would have wanted to see with other libraries. Use the source, Luke!

Two last weeks of the challenge...

Still time to post your own entry!

On 3/29/2024 at 11:37 AM, Rollo62 said:

Nice list, thanks 👍

 

And don't forget the command line parser available in mORMot 2.

- Works on all OS;

- Works on Delphi and FPC;

- Can auto-generate the help content with proper formatting;

- Minimal code writing.

In practice, resulting code is short and efficient:

https://github.com/synopse/1brc-ObjectPascal/blob/main/entries/abouchez/src/brcmormot.lpr#L446

This is a "classical" problem with Delphi, overloaded functions, and floating points values.

IMHO using currency is an awful workaround here, and should not be kept in any serious codebase.
It works "by chance".

Another workaround, still using the Math unit, is to make the computation in two steps:

function RoundExDouble(x: double): Double;
begin
  x := x * 10;
  Result := Ceil(x) / 10;
end;

function RoundExInteger(x: double): Integer;
begin
  x := x * 10;
  Result := Ceil(x);
end;

So the expected Ceil(double) overload is clearly picked up by the compiler.

Or define our own unique Ceil() function, which does not depend on Math.pas and ensure we use the right value type:

function ceil(x: double): integer;
begin
  result := trunc(x) + ord(frac(x) > 0);
end;

This is this solution I used in my entry of the challenge: stay away from Math.pas. 😉

I missed the info, sorry.

For a SQL solution, it is very good.

Out of curiosity, how much memory does it need for the 16.5 GB file?
Does it use SQLite3 and its virtual tables internally for its SQL dialect (something like https://www.sqlite.org/csv.html)?

On 3/10/2024 at 9:06 AM, Alexander Sviridenkov said:

HTML Library / SQL framework: 0.33s. Zero lines of code)

 

 

 

Easy, and wrong.

You are reading the station weathers reference data with one row per station.
And making a min/max/average of a single data per station.

The challenge is to read a 1 billion (1,000,000,000) rows of CSV data for all those 41343 stations, and compute it.

There is a generator of a 16GB CSV file to inject and process.

So 0.33s for 41343 rows would make around 8000 seconds, i.e. 5.5 days.

Since years, our Open Source mORMot framework offers several ways to work with any kind of runtime arrays/objects documents defined, e.g. via JSON, with a lot of features, and very high performance.

Our TDocVariant custom variant type is a powerful way of working with such schema-less data, but it was found confusing by some users.
So we developed a new set of interface definitions around it, to ease its usage, without sacrificing its power. We modelized them around Python Lists and Dictionaries, which is proven ground - with some extensions of course.

Two one-liners may show how our mORMot library is quite unique in the forest/jungle of JSON libraries for Delphi (and FPC):
+]

Yes, this code compiles and run on Delphi - even the antique Delphi 7. 😆

If you compare e.g. to how the standard JSON library works in the RTL, with all its per-node classes, you may find quite a difference!
And there is no secret in regard to performance, that our JSON processing is fast.

More info:

https://blog.synopse.info/?post/2024/02/01/Easy-JSON-with-Delphi-and-FPC

On 10/3/2023 at 9:22 PM, aehimself said:

I wish this was explained further. Maybe we will finally get TZipFile.Delete / TZipFile.Remove? 😄

OpenSource unit
https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.zip.pas

supports deletion - just use TZipWrite.CreateFrom() with a list of files to ignore when reading the existing zip.

10 hours ago, PeaShooter_OMO said:

Why is it difficult to implement? Or shall I rather say; what about it makes it difficult to implement?

You may try to implement it properly, with all the appropriate testing coverage, and I guess you will find out.

There are name/value parsing, execution timing, and multiple parameters involved.

Existing implementations in Delphi are not widely used nor maintained. Not what we could call a "standard".

@FaFaFooey
Argon2 is nice for sure - but really complex, and difficult to implement.

So I am not sure it is "the standard of storing a password hash" on Delphi, in which I was not able to easily find a full-features library.

I like very much the https://en.wikipedia.org/wiki/Proof_of_work concept:

  Proof-of-Work Puzzle: the nonce is used as the puzzle for a Proof-of-Work challenge which the client has to solve (as part of anti-brute force protection). A simple scheme can be to send a nonce and ask the client to find a string that, when appended to your nonce, will have a SHA-256 hash beginning or ending with a certain number of zeroes (adjusting the number of zeroes allows adjusting the puzzle difficulty). When getting a puzzle response, check first the nonce validity, then the hash.

See https://www.delphitools.info/2016/01/07/nonces-and-tokens for the diverse means of authenticating an user.

With this new year, it was time to make a mORMot 2 release.
We went a bit behind the SQlite3 engine, and we added some nice features.
😎

Main added features:
- OpenSSL 3.0/3.1 direct support in addition to 1.1 (with auto-switch at runtime)
- Native X.509, RSA and HSM support
- mget utility (wget + peer cache)

Main changes:
- Upgraded SQLite3 to 3.44.2
- Lots of bug fixes and enhancements (especially about cryptography and networking)
- A lot of optimizations, so that we eventually reached in top 12 ranking of all frameworks tested by TFB

https://github.com/synopse/mORMot2/releases/tag/2.2.stable

🙂

@Edwin Yip Yes, we will try to make a TLS 1.3 layer beginning of this new 2024 year. 😉

On 12/18/2023 at 9:11 AM, Tommi Prami said:

What do you mean here actually. Add "salt" to the data to be crypted or to the crypted it result data?

 

and why is 16 bytes important, or it is 16 bytes or more, or any multiple of n bytes?

This is nothing about salting.

@Kas Ob. wrote it fvery nicely: this is because the AES encoding algorithm is a block algorithm by itself: it works on blocks of 16 bytes as input and output. It works only on full blocks, i.e. encoded output is always a multiple of 16 bytes.

So you need a way of "cutting" the result without leaking any information.

This is known as "padding" the output - PKCS#7 defined the most common way: you output up to 16 more bytes, in which you fill the remaining bytes of your data with the length of your data modulo 16. So once decrypted, you can "cut" back the result into the original size. For some chaining modes (like AES-CTR) there are some other ways of padding with no additional output bytes, but it is less common.

My point was: do not reinvent the wheel, or you are likely to obtain a weak safety.

Some libraries are not part of GetIt for obscure reasons within Embarcadero decision makers, e.g. our

https://github.com/synopse/mORMot2/blob/master/src/crypt/mormot.crypt.core.pas

which is probably the fastest AES library able to be statically linked, OpenSource with full source code, and highly maintained, on Delphi (and FPC).

And before using AES in your project, ensure you understand its basics:
- it is very easy to make something weak, so you need to know a little about today's best practices.
- don't reinvent the wheel: if you want to transmit data, use TLS; if you want to store passwords in a DB, don't use encryption but salted hashes which can't be reversed; see if using OS level disk encryption (like BitLocker) is not a better approach; etc...

- don't use direct/naive AES in your project, named ECB because it is weak https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_(ECB) - consider AES-CTR (or AES-GCM which also makes authentication) for instance.
- there is also a need for 16-bytes padding of the encoded output - here Pkcs7 seems the de-facto standard.

- something tricky is to create the 128-bit or 256-bit key: you need to have a strong derivation function, like PBKDF2 from human input or at least HMAC-SHA256 if you already have a secret key.
- consider also asymmetric encryption: if you can use ECDHE derivation using public keys in your project, this is the way to go - it is much safer to share two public keys between peers and derivate a secret, than sharing a symmetric key, which can be leaked.

- a proper IV filling is also essential: a common practice it to generate it from random, and put it at the beginning of the encrypted stream, or even better get this IV from some bits of the ECDHE derivation function.
- in practice, AES with 128-bit is enough security for an application - it has more security that RSA-2048 certificates which are still used almost everywhere in the Internet.

Our units allows all this, and much more.

This is funny, a few weeks ago I added advanced cross-platform support of network adapters in the mORMot 2 network layer.

It can retrieve all mac addresses of the network interfaces, and also much more information like each kind of adapter, its MTU, speed or gateway/broadcast IPs.

And it is cross-platform, so it works not only on Windows but also on POSIX (tested on MacOS and Linux).
https://github.com/synopse/mORMot2/blob/fe3fdf53e54dc770bda8d9858c8d6ff5ebf4ac4d/src/net/mormot.net.sock.pas#L517

@Edwin Yip
Not yet, the TLS layer is not yet available.
But I would not use Indy anyway, but the mORMot direct client classes instead, which already allows both OpenSSL and SSPI so you could use the latest TLS standard on the latest Windows revision. 😉

@Kas Ob.
Thanks a lot for the very interesting feedback.

Some remarks with no order:

1. As written in the blog article, we tried to follow FIPS specs, used Mbed TLS source as reference, and also the HAC document. Perhaps not at 100%, but at least for the most known cornercases.
2. RSA keys generation follows all those patterns, so should have very proven pattern to generate primes, and we fixed 65537 as exponent. Even the random source is the OS (which is likely to be trusted), then XORed with RdRand() on Intel/AMD, then XORed with our own CSPRNG (which is AES based, from an extensive set of entropy sources). XORing a random source with other sources is a common and safe practice to ensure there is no weakness or predictability. We therefore tried to avoid weaknesses like https://ieeexplore.ieee.org/document/9014350 - see the TBigInt.FillPrime method.
3. The "certificate cache" is a cache from the raw DER binary: the same ICryptCert instance will be reused for the very same DER/PEM input. Sounds safe.
4. In the code you screenshot, there is a x.Compare() call which actually ensure that the supplied certificate DER/PEM binary match the one we have in the trusted list as SKI.
5. If the application is using our high-level ICryptCert abstract interface, only safe levels will be available (e.g. a minimal RSA-2048 with SHA-256, or ECC-256 with SHA-256, and proven AES modes): it was meant to be as less error-prone as possible for the end user. You just can't sign a certificate from a MD5 or SHA1 hash, or encrypt with RC4, DES or AES-ECB.
6. Note that our plan is to implement TLS 1.3 (and only its version 1.3) in the close future, to mitigate even more MIM attacks during the TLS handshake (because all traffic is signed and verified).

To summarize, if we use some cache or search within the internal lists, we always ensure that the whole DER/PEM binary do match at 100%, not only some fields. We even don't use fingerprints, but every byte of the content.
So attacks from forged certificates with only partial fields should be avoided.

Of course, in real live if some company need its application to fulfill some high level of requirements, you may just use OpenSSL or any other library which fits what is needed.

With some other potential problems, like loading a wrong or forged external library, or running on a weak POSIX OS... but it is the fun of regulation. 😉 You follow the rules - even if they also are weak.
 

Perhaps we missed something, so your feedback is very welcome.
We would like to have our mormot.crypt.*.pas units code audited in the future, by some third party or government agency, in our EEC context, and especially the French regulations.
The mORMot 1 SynCrypto and SynEcc were already audited some years ago by a 1B$ company security experts - but it was an internal audit. But the security is even better with mORMot 2.

Please continue to look at the source, and if you see anything wrong and dubious, or see any incorrect comment, do not hesitate to come back!