Surprising behavior of .ToUpper

[Hans♫ 73]

We have received reports from our Turkish users that our app is failing. Our debugging comes down to this simplified code:

lStr1 := 'This';
lStr2 := 'THIS';
if lStr1.ToUpper = lStr2.ToUpper then
begin
  //This code is NOT executed if local is Turkish!
end;

This happens in both Delphi 11.3 and Delphi 12, and it happens on all platforms: Windows, MacOS, Android and iOS.

The reason is that the uppercase edition of the letter "i" becomes "İ", which is not the same as "I", while the uppercase edition of "I" remains unchanged.

 

The problem is solved by using "SameText", but I still find that the above behavior is strange.

[Lars Fosdal 1732]

Is it Turkish text being converted to uppercase?

Is ToUpper using the Turkish locale?

[Hans♫ 73]

Just now, Lars Fosdal said:

Is it Turkish text being converted to uppercase?

Is ToUpper using the Turkish locale?

No it is English text that is hardcoded in our app, just like in the example. But .ToUpper uses the OS locale (i.e. Turkish), which means that the uppercase edition of "i" is not "I".

[Lars Fosdal 1732]

You can specify the locale as a parameter to ToUpper.
https://docwiki.embarcadero.com/Libraries/Athens/en/System.SysUtils.TStringHelper.ToUpper

See also ToUpperInvariant.

[Anders Melander 1639]

As designed. Look at the source of TStringHelper.ToUpper in SysUtils.

Try using ToUpperInvariant instead.

[Kas Ob. 91]

Hi,

 

I always test my text and DB applications for international clients, not only international but when ever a user might be input text and locale might be a concern, which is almost always the case, with https://github.com/xupefei/Locale-Emulator

 

Get familiar with that emulator mainly because it really save your time on changing OS settings (non-Unicode and interface) and combine that with virtual keyboard or just use copied text from Google Translate using Chrome and FireFox, don't use Edge !

 

Never used TStringHelper though in my code, mostly due the old Delphi versions, so try to reproduce this on your developing machine.

 

Hope that help.

[uligerhardt 18]

Turkish has two kinds of I - one with dot and one without. Their case-handling is special. See Dotless I - Wikipedia.

Edited February 6 by uligerhardt

[Uwe Raabe 1889]

It boils down to the fact that 'THIS' just is not the upper case of 'This' on every locale. In case this is a requirement of the application, then ToUpper is not the right choice. Unfortunately ToUpperInvariant may not be either, although it will probably work for pure English text strings.

 

For most locales upper case for ASCII characters is what one would expect, but that doesn't hold true for Turkish.

[DPStano 15]

function LCMapStringEx(const Text: string; Locale: LCID; MapFlag: DWORD): string;
var
  pDes: PChar;
  len, len2: Integer;
begin
  if Text = '' then
  begin
    Result := '';
    Exit;
  end;
  len := Length(Text);
  len2 := len * 2 + 2;
  GetMem(pDes, len2); //half -> full
  try
    FillChar(pDes^, len2, 0);
    LCMapString(Locale, MapFlag, PChar(Text), len, pDes, len2 - 1);
    Result := string(pDes);
  finally
    FreeMem(pDes);
  end;
end;

Result := LCMapStringEx(Text, LANG_TURKISH, LCMAP_UPPERCASE);

Edited February 6 by DPStano

[ertank 21]

4 hours ago, Hans♫ said:

We have received reports from our Turkish users that our app is failing.

As per documentation TStringHelper.ToUpper uses locale specific case conversion.

As per documentation UpperCase or SameText uses 7-bit ASCII conversion. Only letters between 'a' and 'z' are converted to uppercase.

 

Since you are using ToUpper, uppercase of 'This' in Turkish locale is 'THİS' (upper case I letter with dot at top). Moreover, lower case for 'THIS' in Turkish locale is 'thıs' (small caps i letter without dot at top).

 

I cannot remember off my mind right now but there are other languages where some characters small and upper cases do not follow ASCII type of conversion.

[Lars Fosdal 1732]

OP specified hardcoded English text, so ToUpperInvariant or an English locale for ToUpper should do the trick.

[Brandon Staggs 185]

https://languagelog.ldc.upenn.edu/nll/?p=73