How to read an .ini file of unknown encoding a FormatSettings?

[Tom F 83]

I'm trying to untangle a mess I created years ago in a macOS FMX program (formerly 32-bit and now Universal ARM 64-bit,)  I've always used the current version of Delphi (now 12) and this app goes back 5+ years, so some of these files were created in earlier versions of Delphi and on older 32-bit versions of macOS.

For years, my app created an IniFile for my program's use only, using TInifile.WriteDateTime.
 

My two mistakes:
 

1. I didn't specify the encoding of the IniFile, which can vary.
    
2. WriteDateTime (AFAIK) uses the global FormatSettings, which can vary. 

 

So, I'm trying to figure out a way to reliably read the IniFile's TDateTime regardless of the defaults in effect when it was created or when it is being read. 

 

I'm planning for my app to try to open the file in TMemIniFile with different encodings (starting with the machine's default) and read the date with TMemIniFile.ReadString. I will then TryStrToDateTime with different FormatSettings (starting with the machine's default) until I get valid dates.  This seems a bit brute force, but I don't know what else to do.  My app is almost entirely US, with a few Canadian, European, Australian, and Asian users, so I hope to cover the majority of cases with a few formats.  I know this may not give me 100% success.

 

I've attached a sample of an .ini file I'm trying to reliably read. Here it is in UltraEdit in hex and characters:

 

 

I've never spent much time with Unicode, but I think that the BOM FF FE here means it's Unicode.

 

But when I open the attached .ini file, ReadString fails to find the Section and Identifier.  I've tried different encodings in the TMemIniFile.Create. (I'm testing this code in FMX Windows 32.)

I feel like I'm chasing my tail at this point.  Am I missing something?  Any suggestions on how to proceed?

 

procedure TForm1.Button1Click(Sender: TObject);
var
  IniFileDateTimeStr: String;
  IniFile: TMemIniFile;
begin
  IniFile := TMemIniFile.Create('C:\Junk\Test.Ini', TEncoding.Unicode); // I have tried other encodings too
  IniFileDateTimeStr := IniFile.ReadString('Version', 'SourceFileDate', 'Missing');
  ShowMessage(IniFileDateTimeStr); //<--- Always shows 'Missing'
end;

 

 

Test.ini

Edited December 20, 2023 by Tom F

[Der schöne Günther 316]

I made the same mistake some time back, but why don't you keep reading the existing local file as you always have? The local format settings or encodings surely have not changed?

 

Then, save it to a new file with fixed encoding and date/time (and float!) format and finally delete the old file. At each startup, your app can first look for the "old" file format, and then the new one.

[Tom F 83]

8 hours ago, Der schöne Günther said:

The local format settings or encodings surely have not changed?

It surprised me to discover that sometimes those settings are changed by users!

That's why I want my code to process this file (and others) regardless of machine settings.

 

[Remy Lebeau 1394]

15 hours ago, Tom F said:

I'm planning for my app to try to open the file in TMemIniFile with different encodings (starting with the machine's default)

The problem with that approach is that you may not get a decoding error, so you wouldn't really know which encoding "works" right away. You might just end up with garbage when reading data.

15 hours ago, Tom F said:

and read the date with TMemIniFile.ReadString. I will then TryStrToDateTime with different FormatSettings (starting with the machine's default) until I get valid dates.

And what do you consider to be "valid"?  Because different formats can produce results without errors, but just won't be what you are expecting.  Can you parse the string first to make an educated guess about its likely format before you then try to decode it as a TDateTime?

15 hours ago, Tom F said:

 

  

I've never spent much time with Unicode, but I think that the BOM FF FE here means it's Unicode.

Yes, and more than that, its the BOM for UTF-16LE specifically.  So you can use TEncoding.Unicode when loading the file.  Actually, internally TMemIniFile reads the file into a TStringList first before parsing it, and TStringList is smart enough to look for a BOM when no TEncoding is specified, so in theory TMemIniFile should be able to load your file in the correct encoding provided a BOM is always present and TMemIniFile.Encoding is nil.

 

However...

16 hours ago, Tom F said:

But when I open the attached .ini file, ReadString fails to find the Section and Identifier.  I've tried different encodings in the TMemIniFile.Create. (I'm testing this code in FMX Windows 32.)

The file you have provided has a UTF-8 BOM, not a UTF-16 BOM.  So your example code would need to use TEncoding.UTF8 instead of TEncoding.Unicode (or, don't specify the encoding at all and let TMemIniFile figure it out for you).

[Tom F 83]

Thanks, @Remy Lebeau. I always value your input.

I can't get the simple code below to write and then successfully read a date on macOS Sonoma 14.2 on a MacMini (ARM).  XCode version 15. Using Delphi version 12.

It works fine on Windows 10 running as a 32-bit or 64-bit app.  But on macOS, the TryStrToDateTime call returns false

I don't doubt that I'm doing something wrong, but I can't figure out what.
 

procedure TForm1.btnCreateClick(Sender: TObject);
var
  MemIniFile: TMemIniFile;
  Filename: String;
  DateStr: String;
begin
  Filename := IncludeTrailingPathDelimiter(TPath.GetDocumentsPath) + 'Test123.ini';
  MemIniFile := TMemIniFile.Create(Filename, TEncoding.Unicode );
  try
    DateStr := DateTimeToStr(Now);
    MemIniFile.WriteString('Version', 'SourceFileDate', DateStr);
    MemIniFile.UpdateFile;
  finally
    MemIniFile.Free;
  end;

end;

procedure TForm1.btnReadClick(Sender: TObject);
var
  Filename: String;
  MemIniFile: TMemIniFile;
  IniFileDateTimeStr: String;
  DateTimeFromIniFile: TDateTime;
begin
  Filename := IncludeTrailingPathDelimiter(TPath.GetDocumentsPath) + 'Test123.ini';
  MemIniFile := TMemIniFile.Create(Filename, TEncoding.Unicode );
  try
    IniFileDateTimeStr := MemIniFile.ReadString('Version', 'SourceFileDate', 'Missing');
  finally
    MemInifile.Free;
  end;

  if IniFileDateTimeStr = 'Missing' then
    ShowMessage('IniFile entry was missing')
  else
    if TryStrToDateTime(IniFileDateTimeStr, {out} DateTimeFromIniFile) then
      ShowMessage('Inifile contained date: ' + DateTimeToStr(DateTimeFromIniFile))
    else
      ShowMessage('TryStrToDateTime failed when called with ' + IniFileDateTimeStr);
end;

I've included the project source in an attached zip file.  

I've also attached the .ini file from the Mac.


 

 

Test123.ini

Complete project source.zip

[Remy Lebeau 1394]

20 minutes ago, Tom F said:

I can't get the simple code below to write and then successfully read a date on macOS Sonoma 14.2 on a MacMini (ARM).  XCode version 15. Using Delphi version 12.

...

I've also attached the .ini file from the Mac.

When I look at that .ini file, there is a Unicode character U+202F (Narrow No-Break Space) in between "11:15:12" and "AM".  I think TryStrToDateTime() only supports the ASCII space character U+0020, so likely can't handle that other type of space character and so fails the parse.

Edited December 20, 2023 by Remy Lebeau

[Tom F 83]

Thanks, Remy.  I'm glad it wasn't something totally obvious I was not doing! 😉

Even the simple code below (which doesn't use an inifile) fails.  I'd call that a genuine bug worthy of my reporting to EMB. 

Thanks again for your help on this and so many other things.  

 

procedure TForm1.Button1Click(Sender: TObject);
var
  DateStr: String;
  DateTime: TDateTime;
begin
  DateStr := DateTimeToStr(Now);
  if TryStrToDateTime(DateStr, {out} DateTime) then
    ShowMessage('Ok result: ' + DateTimeToStr(DateTime))
  else
    ShowMessage('TryStrToDate failed');
end;

 

[Remy Lebeau 1394]

8 minutes ago, Tom F said:

Even the simple code below (which doesn't use an inifile) fails.  I'd call that a genuine bug worthy of my reporting to EMB.

Did you verify that DateTimeToStr() is actually producing a String that uses U+202F instead of U+0020?  I can't imagine the RTL ever using U+202F on purpose, so it is likely coming from the OS configuration when the RTL reads in the OS's locale info.

 

All the more reason NOT to rely on OS locales when saving your data.  Use a more reliable format, like ISO-8601.

Edited December 20, 2023 by Remy Lebeau

[Tom F 83]

1 hour ago, Remy Lebeau said:

All the more reason NOT to rely on OS locales when saving your data.  Use a more reliable format, like ISO-8601.

Thanks for adding that reference, Remy.  I wasn't aware of Delphi's System.DateUtils.DateToISO8601 and siblings, which is exactly what I needed!!!

https://docwiki.embarcadero.com/Libraries/Alexandria/en/System.DateUtils.DateToISO8601 

[DelphiUdIT 176]

I add to this discussion (which talks about formats in INI files) that a problem similar to that of the date also exists in the writing and reading of float values (i.e. numbers with decimal points). It is necessary to use the TFormatSettings and set the correct values (which are those of the files and not of the language). The example is when you go to write a value in an environment set in Italian (where the "comma" is the decimal separation sign) and perhaps then read it in an environment set in English (where instead it is the "dot ").
I override ReadFloat and WriteFloat of TIniFile to correct that to always read and write the right values.

Of course it is for compatibility and maintenance of the old programs, the new ones use a local database to store the data.

Edited December 20, 2023 by DelphiUdIT

[Rollo62 536]

Maybe this is also interesting background info:
https://www.ionos.com/digitalguide/websites/web-development/iso-8601/

 

https://wiert.me/2020/10/21/delphi-get-timestamp-as-iso8601-string-for-use-in-filenames/

https://wiert.me/2011/08/18/iso-8601-date-time-and-datetime-in-delphi-was-simple-example-to-show-datetime-now-in-iso-8601-format-on-ideone-com-online-c-compiler-debugging-tool/

Edited December 21, 2023 by Rollo62