Unicode normalization

[Anders Melander 1358]

I'm writing a shaper for complex text layout and for that, I need to do Unicode decomposition and composition (NFD and NFC normalization).
 

Does anyone know of a Pascal library that can do this?

 

I have the following basic requirements:

• Open source with a workable license (i.e. not GPL).
• Cross platform (i.e. not tied to Windows or whatever).
• Operate on UFC4/UTF-32 strings.
• Based on the standard Unicode data tables.
• Must have the ability to update tables when new Unicode tables are published.
• Must support both NFD decomposition and NFC composition.

 

So far I have found the following candidates:

Delphi Unicode libraries

• PUCU Pascal UniCode Utils Libary
  🔵 Origin: Benjamin Rosseaux.
✅ Maintained: Yes.
✅ Maintained by author: Yes.
✅ License: Zlib.
⛔ Readability: Poor. Very bad formatting.
✅ Performance: The native string format is UCS4 (32-bit).
✅ Features: Supports decomposition and composition.
✅ Dependencies: None.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
⛔ Table format: Generated inline arrays and code.
✅ Completeness: All Unicode tables are available.
✅ Hangul decomposition: Yes.
✅ Customizable: Required data and structures are exposed.
✅ Unicode level: Currently at Unicode v15.
⛔ Unicode normalization test suite: Fail/Crash
   
• FreePascal RTL
  🔵 Origin: Based on code by Inoussa Ouedrago.
✅ Maintained: Yes.
🔵 Maintained by author: No.
🔵 License: GPL with linking exception.
✅ Readability: Good. Code is clean.
✅ Performance: Code appears efficient.
⛔ Features: Only supports decomposition. Not composition.
✅ Dependencies: None.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
✅ Table format: Generated arrays in include files.
⛔ Completeness: Only some Unicode tables are available.
⛔ Hangul decomposition: No.
⛔ Customizable: Required data and structures are private.
✅ Unicode level: Currently at Unicode v14.
⛔ Unicode normalization test suite: N/A; Composition not supported.
   

• JEDI jcl
  🔵 Origin: Based on Mike Lischke's Unicode library.
🔵 Maintained: Sporadically.
🔵 Maintained by author: No.
✅ License: MPL.
✅ Readability: Good. Code is clean.
⛔ Performance: Very inefficient. String reallocations. The native string format is UCS4 (32-bit).
✅ Features: Supports decomposition and composition.
⛔ Dependencies: Has dependencies on a plethora of other JEDI units.
✅ Data source: Unicode tables are generated from official Unicode data files. Source for converter provided.
✅ Table format: Generated resource files.
🔵 Unicode level: Currently at Unicode v13.
✅ Completeness: All Unicode tables are available.
✅ Hangul decomposition: Yes.
⛔ Customizable: Required data and structures are private.
⛔ Other: Requires installation (to generate the JEDI.inc file).
🔵 Unicode normalization test suite: Unknown

 

The FPC implementation has had the composition part removed so that immediately disqualifies it and the JEDI implementation, while based on an originally nice and clean implementation, has gotten the usual JEDI treatment so it pulls in the rest of the JEDI jcl as dependencies. I could clean that up but it would amount to a fork of the code and I would prefer not to have to also maintain that piece of code.

That leaves the PUCU library and I currently have that integrated and working - or so I thought... Unfortunately, I have now found a number of severe defects in it and that has prompted me to search for alternatives again.

 

Here's the project I need it for: https://gitlab.com/anders.bo.melander/pascaltype2

[Lars Fosdal 1485]

Is there Unicode support in @Alexander Sviridenkov's https://delphihtmlcomponents.com/index.html ?

[Anders Melander 1358]

51 minutes ago, Lars Fosdal said:

Is there Unicode support in @Alexander Sviridenkov's https://delphihtmlcomponents.com/index.html ?

Doesn't really matter:

2 hours ago, Anders Melander said:

I have the following basic requirements:

• Open source with a workable license (i.e. not GPL). 

 

[Fr0sT.Brutal 814]

20 hours ago, Anders Melander said:

I could clean that up but it would amount to a fork of the code and I would prefer not to have to also maintain that piece of code.

Why not extract only necessary external requirements to custom low-fat units and leave the unit of interest as is? Or there's too much deps?

[Anders Melander 1358]

1 minute ago, Fr0sT.Brutal said:

Why not extract only necessary external requirements to custom low-fat units and leave the unit of interest as is? Or there's too much deps?

I guess that's one way to remove the dependencies. I hadn't thought of that.

 

But I would still have to clean up the remaining source and rewrite parts of it, so again: A fork.
If I go that way I would probably prefer to simply start from the original, pre-JEDI, version of the source instead of trying to polish the turd it has become.