Unicode normalization

[Anders Melander 1785]

I'm writing a shaper for complex text layout and for that, I need to do Unicode decomposition and composition (NFD and NFC normalization).
 

Does anyone know of a Pascal library that can do this?

 

I have the following basic requirements:

• Open source with a workable license (i.e. not GPL).
• Cross platform (i.e. not tied to Windows or whatever).
• Operate on UFC4/UTF-32 strings.
• Based on the standard Unicode data tables.
• Must have the ability to update tables when new Unicode tables are published.
• Must support both NFD decomposition and NFC composition.

 

So far I have found the following candidates:

Delphi Unicode libraries

• PUCU Pascal UniCode Utils Libary
  🔵 Origin: Benjamin Rosseaux.
✅ Maintained: Yes.
✅ Maintained by author: Yes.
✅ License: Zlib.
⛔ Readability: Poor. Very bad formatting.
✅ Performance: The native string format is UCS4 (32-bit).
✅ Features: Supports decomposition and composition.
✅ Dependencies: None.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
⛔ Table format: Generated inline arrays and code.
✅ Completeness: All Unicode tables are available.
✅ Hangul decomposition: Yes.
✅ Customizable: Required data and structures are exposed.
✅ Unicode level: Currently at Unicode v15.
⛔ Unicode normalization test suite: Fail/Crash
   
• FreePascal RTL
  🔵 Origin: Based on code by Inoussa Ouedrago.
✅ Maintained: Yes.
🔵 Maintained by author: No.
🔵 License: GPL with linking exception.
✅ Readability: Good. Code is clean.
✅ Performance: Code appears efficient.
⛔ Features: Only supports decomposition. Not composition.
✅ Dependencies: None.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
✅ Table format: Generated arrays in include files.
⛔ Completeness: Only some Unicode tables are available.
⛔ Hangul decomposition: No.
⛔ Customizable: Required data and structures are private.
✅ Unicode level: Currently at Unicode v14.
⛔ Unicode normalization test suite: N/A; Composition not supported.
   

• JEDI jcl
  🔵 Origin: Based on Mike Lischke's Unicode library.
🔵 Maintained: Sporadically.
🔵 Maintained by author: No.
✅ License: MPL.
✅ Readability: Good. Code is clean.
⛔ Performance: Very inefficient. String reallocations. The native string format is UCS4 (32-bit).
✅ Features: Supports decomposition and composition.
⛔ Dependencies: Has dependencies on a plethora of other JEDI units.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
✅ Table format: Generated resource files.
🔵 Unicode level: Currently at Unicode v13.
✅ Completeness: All Unicode tables are available.
✅ Hangul decomposition: Yes.
⛔ Customizable: Required data and structures are private.
⛔ Other: Requires installation (to generate the JEDI.inc file).
🔵 Unicode normalization test suite: Unknown

 

The FPC implementation has had the composition part removed so that immediately disqualifies it and the JEDI implementation, while based on an originally nice and clean implementation, has gotten the usual JEDI treatment so it pulls in the rest of the JEDI jcl as dependencies. I could clean that up but it would amount to a fork of the code and I would prefer not to have to also maintain that piece of code.

That leaves the PUCU library and I currently have that integrated and working - or so I thought... Unfortunately, I have now found a number of severe defects in it and that has prompted me to search for alternatives again.

 

Here's the project I need it for: https://gitlab.com/anders.bo.melander/pascaltype2

[Lars Fosdal 1792]

Is there Unicode support in @Alexander Sviridenkov's https://delphihtmlcomponents.com/index.html ?

[Anders Melander 1785]

51 minutes ago, Lars Fosdal said:

Is there Unicode support in @Alexander Sviridenkov's https://delphihtmlcomponents.com/index.html ?

Doesn't really matter:

2 hours ago, Anders Melander said:

I have the following basic requirements:

• Open source with a workable license (i.e. not GPL). 

 

[Fr0sT.Brutal 900]

20 hours ago, Anders Melander said:

I could clean that up but it would amount to a fork of the code and I would prefer not to have to also maintain that piece of code.

Why not extract only necessary external requirements to custom low-fat units and leave the unit of interest as is? Or there's too much deps?

[Anders Melander 1785]

1 minute ago, Fr0sT.Brutal said:

Why not extract only necessary external requirements to custom low-fat units and leave the unit of interest as is? Or there's too much deps?

I guess that's one way to remove the dependencies. I hadn't thought of that.

 

But I would still have to clean up the remaining source and rewrite parts of it, so again: A fork.
If I go that way I would probably prefer to simply start from the original, pre-JEDI, version of the source instead of trying to polish the turd it has become.

[Anders Melander 1785]

In the end, I had to abandon PUCU as the author never bothered to react to my bug reports.

Instead, I tried to adapt the JEDI implementation: I removed all dependencies and fixed the worst bugs and performance issues. Finally, I ran the code against my unit tests. The result was a big disappointment; While it didn't crash like PUCU, it failed even more of the test cases. The test suite uses the 19,000 NFC (compose) and NFD (decompose) normalization test cases published by the Unicode consortium.

 

So back to square one again. Comparing the algorithms used by the JEDI library against countless (I've looked at over a hundred) other Unicode libraries didn't reveal the cause. They all used the same algorithms. Blogs and articles that described the algorithm also matched what was being done. I was beginning to suspect that Unicode's own test cases were wrong, but then I finally got around to reading the actual Unicode specification where the rules and algorithms are described, and guess what - Apart from Unicode's own reference library and a few others, they're all doing it wrong.

 

I have now implemented the normalization functions from scratch based on the Unicode v15 specs and all tests now pass.

The functions can be found here, in case anyone needs them: https://gitlab.com/anders.bo.melander/pascaltype2/-/blob/master/Source/PascalType.Unicode.pas#L258

Note that while the functions implement both canonical (NFC/NFD) and compatible (NFKC/NFKD) normalization, only the canonical variants have been tested as they are the only ones I need.