Test your product with UTF-8 beta setting

[Tommi Prami 130]

Problem very poorly explained at : https://quality.embarcadero.com/browse/RSP-21814

Please vote for it.

 

Now we have seen many cases of this that customers have bought new computer and this setting has been on as default. And then at least our software stops working.

 

Hope that you'll test this in on your own environments and products. and report your findings on that Ticket. I'll try to get on this also day, and report more exact findings.

 

-Tee-

[Kryvich 165]

Never seen this option. Thanks for the heads up.

P.S. Is this done to make non-Unicode programs Unicode-compatible? I think Microsoft was late with this solution for 15 years at least.

[Fr0sT.Brutal 900]

Interesting option. I wonder what it changes from the app's POV.

3 minutes ago, Kryvich said:

P.S. Is this done to make non-Unicode programs Unicode-compatible? I think Microsoft was late with this solution for 15 years at least.

I doubt ancient apps will be able to use Unicode flawlessly even with this option.

Edited March 19, 2020 by Fr0sT.Brutal

[Tommi Prami 130]

10 minutes ago, Kryvich said:

P.S. Is this done to make non-Unicode programs Unicode-compatible? I think Microsoft was late with this solution for 15 years at least.

True dat.

[Tommi Prami 130]

This was tested in 10.3 But not with the latest. Anyone in 10.4 Beta should test this, on different kinds of apps.

 

And if there are problems, report issues inside beta-channels.

[Kryvich 165]

To support this option, every PAS file in a project should be saved in UTF-8 format (not ANSI!). Otherwise you'll get a bunch of compiler warnings.

 

[Uwe Raabe 2057]

23 minutes ago, Kryvich said:

To support this option, every PAS file in a project should be saved in UTF-8 format (not ANSI!).

For new units you can enforce that with this registry setting (example for Delphi  10.3):

Quote

HKEY_CURRENT_USER\Software\Embarcadero\BDS\20.0\Editor]
"DefaultFileFilter"="Borland.FileFilter.UTF8ToUTF8"

 

[Lars Fosdal 1792]

This could use a QP to expose it in the regular Options dialogs.

[Kryvich 165]

I have just write a little command line utility to convert your ANSI file to UTF-8.

https://github.com/Kryuski/AnsiToUtf8

[Uwe Raabe 2057]

42 minutes ago, Lars Fosdal said:

This could use a QP to expose it in the regular Options dialogs.

https://quality.embarcadero.com/browse/RSP-15589

[Kryvich 165]

56 minutes ago, Uwe Raabe said:

"DefaultFileFilter"="Borland.FileFilter.UTF8ToUTF8"

It's strange that "Borland" still here in Delphi 10.3.3. 🙂 

[Kryvich 165]

45 minutes ago, Lars Fosdal said:

This could use a QP to expose it in the regular Options dialogs.

I think this option is not necessary, if Delphi 10.4 will set UTF-8 encoding by default for all new source files.

[Tommi Prami 130]

7 minutes ago, Kryvich said:

I think this option is not necessary, if Delphi 10.4 will set UTF-8 encoding by default for all new source files.

Known fact or hoping for that?

 

-Tee-

[Kryvich 165]

@Tommi Prami Just a hope. I haven't access to the 10.4 beta.

 

Note: When UTF-8 is used as a default code page in Windows, if you use national characters in a AnsiChar constant, they should be encoded in #$nn format. Otherwise you'll get "W1061 Narrowing given WideChar constant (#$0401) to AnsiChar lost information".

program AnsiCharConst;
{$APPTYPE CONSOLE}
{$R *.res}
const
  AnsiCharYo1: AnsiChar = 'Ё';
  AnsiCharYo2: AnsiChar = #$A8; // 'Ё' in ANSI 1251 Code Page;
begin
  Writeln('#', Byte(AnsiCharYo1), ', #', Byte(AnsiCharYo2));
  Readln;
end.

This program shows: "#129, #168".

 

Edited March 19, 2020 by Kryvich

[Dave Nottage 557]

44 minutes ago, Kryvich said:

It's strange that "Borland" still here in Delphi 10.3.3.

So is CodeGear

[Rollo62 536]

Thanks god the new managements didn't kill all old references in code and IDE as first step, causing all kinds of weired issues.

[Kryvich 165]

Final conclusion after setting of UTF-8 as a default code page for Windows: if your program actively uses ANSI strings with national characters, you cannot just recompile the program and hope that it will work correctly. You will have to carefully check the code, and use AnsiStrings with a code page (i.e. AnsiString(1251)) or RawByteStrings where appropriate.

Edited March 19, 2020 by Kryvich

[Fr0sT.Brutal 900]

1 hour ago, Kryvich said:

When UTF-8 is used as a default code page in Windows, if you use national characters in a AnsiChar constant, they should be encoded in #$nn format.

Those who use chars > #127 in AnsiChar constant are digging a trap for themselves anyway.

I think this option is a good reason to check usage of Ansi encodings in projects. BTW, I recently found a very sly bug caused by ACP that only revealed itself on a system with an ACP different than expected. Luckily the check for reproducibility didn't require any OS change, just setting System.DefaultSystemCodePage to another value.

Edited March 19, 2020 by Fr0sT.Brutal