Delphi 11.2 vs 10.4.2 locale issue

[aehimself 396]

Hello,

 

I am in the process of verifying if we can update our environment to use Delphi 11.2 instead of the current 10.4.2. However compilation was successful, even the very basic tests failed because of an incorrect date format detected by Delphi.

 

Reproduction is really easy, just execute a single line:

TFormatSettings.Create(1038);

Result on Delphi 11.2:

 

On 10.4.2:

 

So 11.2 correctly detects the date separator as "." in the locale, however the very next line will call this method, which will replace all separators with a forward slash instead:

  procedure FixDateSeparator(var DateFormat: string);
  var
    P: PChar;
    InsideLiteral: Boolean;
  begin
    InsideLiteral := False;
    P := PChar(DateFormat);
    if P = nil then
      Exit;
    while P^ <> #0 do
    begin
      if P^ = '''' then
        InsideLiteral := not InsideLiteral;
      if (P^ = Separator) and (not InsideLiteral) then
        P^ := '/';
      Inc(P);
    end;
  end;

So because 10.4.2 could not detect the separator, it returns the correct date format. 11.2 detects the separator, ruining the date format instead. The WinAPi definition is the same, buffer is different in the implementation:

function GetLocaleInfo; external kernel32 name 'GetLocaleInfoW';

// Delphi 10.4.2 MSWindows implementation
function GetLocaleChar(Locale, LocaleType: Integer; Default: Char): Char;
var
  Buffer: array[0..1] of Char;
begin
  if GetLocaleInfo(Locale, LocaleType, Buffer, 2) > 0 then
    Result := Buffer[0] else
    Result := Default;
end;

// Delphi 11.2 MSWindows implementation
function GetLocaleChar(Locale, LocaleType: Integer; Default: Char): Char;
var
  Buffer: array[0..2] of Char;
begin
  if GetLocaleInfo(Locale, LocaleType, Buffer, 3) > 0 then
    Result := Buffer[0] else
    Result := Default;
end;

I can not wrap my head around this. I believe it should be a faulty FixDateSeparator implementation, as it makes no sense to change the correct separators to incorrect ones based on the selected locale.

 

Is this an issue in Delphi and I should raise a ticket or maybe I'm missing something...?

[Fr0sT.Brutal 900]

The reason for "fixing" is that they use slash as "current date separator" special char. When function encounters it, it replaces slash to current DateSeparator. This way date format becomes independent from date separator change so user can just modify date separator to get desired format instead of changing whole date format.

Regarding the separator itself, probably it couldn't be retrieved from the system (as https://learn.microsoft.com/en-us/windows/win32/intl/locale-sdate says, this constant is deprecated) so it gets value of slash.

BTW, considering this

Quote

A custom locale might not have a single, uniform separator character. For example, a format such as "12/31, 2006" is valid.

the whole logic of these manipulations seem to break on a format like above.

I did some tests and seems SDATE constant is taken from the first separator of a short date format. However it's easy to surprise RTL code easily by defining short date as "dd ,-;MM\:;yyyy". This way even the widened buffer of D11 won't be enough to hold the separator and it will take slash value though actual date format won't contain any slash at all

Edited November 28, 2022 by Fr0sT.Brutal

[Fr0sT.Brutal 900]

Elementary case of breaking things working since W7 and I suppose any Delphi: set OS date format to "dd-MM/yyyy". Result of "DateToStr(Now)" will be "28-11-2022" not "28-11/2022" as everyone could expect

[aehimself 396]

41 minutes ago, Fr0sT.Brutal said:

When function encounters it, it replaces slash to current DateSeparator. This way date format becomes independent from date separator change so user can just modify date separator to get desired format instead of changing whole date format.

I don't know if I follow you here. Does it mean that I must never use FormatSettings.ShortDateFormat as it is, only FormatSettings.ShortDateFormat.Replace('/', FormatSettings.DateSeparator)?

 

The reason I'm asking is that 3rd-party controls (like DevExpress) started to show (and save) dates differently just because of compiling under D11. That makes me think that either the above statement is incorrect, or all 3rd-party controls handle this incorrectly.

To correct this I had to manually correct FormatSettings.ShortDateFormat but that immediately breaks all logic which relies on TFormatSettings.Create and StrToDate - which our tool is using a lot.

 

54 minutes ago, Fr0sT.Brutal said:

Regarding the separator itself, probably it couldn't be retrieved from the system (as https://learn.microsoft.com/en-us/windows/win32/intl/locale-sdate says, this constant is deprecated) so it gets value of slash.

The reason Delphi 10.4.2 could not get it is because Embarcadero didn't provide a buffer big enough. As both versions are calling the same WinApi function, changing the buffer to array[0..1] retrieving will fail on D11 aswell; calling RaiseLastOSError will reveal the reason:

 

Maybe I should have been more precise. I personally don't care about the date separator alone, only the malformation of ShortDateFormat because of it, which seem to affect appearance and logic.

[Fr0sT.Brutal 900]

15 hours ago, aehimself said:

Does it mean that I must never use FormatSettings.ShortDateFormat as it is, only FormatSettings.ShortDateFormat.Replace('/', FormatSettings.DateSeparator)?

No, no, I'd say you rather should never use DateSeparator as it is unreliable (it was bulletproof when only one-char separators were possible and both separators were the same - which is not true anymore). The FormatSettings.ShortDateFormat field is transformed to internal RTL format that will be handled by RTL routines. You should bother with its contents only if you have custom (non-RTL) formatting routines.

15 hours ago, aehimself said:

The reason Delphi 10.4.2 could not get it is because Embarcadero didn't provide a buffer big enough. As both versions are calling the same WinApi function, changing the buffer to array[0..1] retrieving will fail on D11 aswell; calling RaiseLastOSError will reveal the reason:

Well, you should investigate why the 2-char buffer became insufficient for date sep. What result that WinAPI function returns if you call it directly (or just step into call to it from RTL)?

And what exactly means " 3rd-party controls handle this incorrectly "?

[aehimself 396]

On 11/29/2022 at 8:27 AM, Fr0sT.Brutal said:

Well, you should investigate why the 2-char buffer became insufficient for date sep. What result that WinAPI function returns if you call it directly (or just step into call to it from RTL)?

And what exactly means " 3rd-party controls handle this incorrectly "?

It returns a dot, a space and #0 (which makes sense) and DevExpress's date editor started to misbehave... normally it showed correct format but when you clicked in it it showed "____ / " and accepted years only.

Spent my last day trying to understand how this works and why some hacks were needed at some places. And I think I finally understood.

 

The issue is NOT with Delphi 11.2 but some random unit changing some properties in the global FormatSettings variable (probably as an attempt to fix the wrongly detected separator). Once I found and got rid of that, VCL controls started to work but some (previously fine) calls to StrToDate started to throw exceptions. The reason is, Emba reworked the logic of date and time parsing, which is more strict in the later. While having a short date format of "yyyy. mm. dd." could parse "2022.11.30" in 10.4.2, Delphi 11.2 actually requires the final separator.

 

Loads of experiments and this knowledge resulted a custom DateUtils unit to attempt to unify the behavior if the application is compiled with 10.4.2 and 11.2, I'll still have to roll back some local fixes tomorrow and start to use the helper instead... after that we'll see if my idea works.

[Fr0sT.Brutal 900]

15 hours ago, aehimself said:

It returns a dot, a space and #0

So you actually have 2-char separator which is exactly the case that makes RTL functions behave strangely.

In fact, this reveals a bunch of bugs to report to QC, as RTL functions are not ready for such flexible formats. The 11's version of the function is not correct either, it will break on 3-char separator.