Integer overflow in tStringList.SaveToFile

[Jud 1]

I'm getting an integer overflow error when writing more than about 2^30 characters in a tStringList using
SaveToFile in 64-bit Delphi 10.4.1.  What I think is happening is that it is using two bytes per character
and this is putting some number over 2^31 and overflowing an integer.  

Is this one of the things that they haven't made 64-bit yet?

If there are under about 1 billion characters, it works.  If there are more, it crashes. 

 

When running the EXE file, it just stops running with no message.

 

In the IDE, it takes
you to:
procedure _IntOver;
{$IFDEF PUREPASCAL}
begin
  ErrorAt(Byte(reIntOverflow), ReturnAddress);
end;

in the system unit.

Which brings up another of my pet peeves: 64-bit Delphi has never handled integer overflows properly.

A sample program is attached.

SampleProgram.txt

Edited December 2, 2020 by Jud

[David Heffernan 2345]

2 hours ago, Jud said:

Which brings up another of my pet peeves: 64-bit Delphi has never handled integer overflows properly

In what way? Can you be more specific. 

 

As for the main problem, I'm not surprised. I'd expect that you'd be better off using a stream writer to write such a huge collection to a stream. I'd submit a bug report to QP, in case on doesn't already exist, and then write some helper code to workaround the issue. 

[Jud 1]

Just now, David Heffernan said:

In what way? Can you be more specific. 

 

As for the main problem, I'm not surprised. I'd expect that you'd be better off using a stream writer to write such a huge collection to a stream. I'd submit a bug report to QP, in case on doesn't already exist, and then write some helper code to workaround the issue. 

Are you asking about my pet peeve about 64-bit Delphi not handling integer overflows?  It has always had this flaw in the 64-bit version.  If you have an integer overflow in the EXE, it just stops, with no error message,  If you are running it in the IDE, it doesn't take you to the line the error, as it does in the 32-bit version, and, as far as I know, it does with all other runtime errors.  An integer overflow in the 64-bit version in the IDE takes you to the obscure place in my message.

 

When I have such an error in a 64-bit program, if I can I switch to 32-bit to see where the error is.  But many times that is not possible because the 64-bit version has to be used, usually because of memory.  And then it usually takes hours and hours to find the source code with the error.

[David Heffernan 2345]

Is your executable compiled with the overflow checks compiler option enabled? 

[Lajos Juhász 293]

22 minutes ago, David Heffernan said:

Is your executable compiled with the overflow checks compiler option enabled? 

It's quite easy to test. I've tested using:
 

procedure TForm1.Button1Click(Sender: TObject);
var i: integer;
begin
  i:=1024;
  while true do
    i:=i*i;
  ShowMessage('Completed. The result is:'+i.ToString);
end;

 

When tested using the 64 bit the call stack is:

:00007FF965893E49 ; C:\WINDOWS\System32\KERNELBASE.dll
System._RaiseAtExcept(???,???)
System.SysUtils.ErrorHandler(???,$70AE52)
System.ErrorAt(5,$70AE52)
System._IntOver
mainFRM.TForm1.Button1Click(???)
Vcl.Controls.TControl.Click
Vcl.StdCtrls.TCustomButton.Click
Vcl.StdCtrls.TCustomButton.CNCommand(???)
System.TObject.Dispatch((no value))

 

But it's not bad, double click on the mainFRM.Tform1.Button1Click and the debugger focues where the 32 bit debugger does, so it's just an extra step.

Edited December 2, 2020 by Lajos Juhász

[Lajos Juhász 293]

Now back to the original problem. This is the limitation of Delphi strings. You cannot allocate string larger than: (MaxInt - SizeOf(StrRec)) div SizeOf(WideChar) = 1073741815.

 

Unfortunately, internally TStringList first tries to convert its content to string and using TEncoding to a byte array. That's why it is not suitable to work with large volume of text.

[Guest]

9 hours ago, Lajos Juhász said:

It's quite easy to test. I've tested using:

procedure TForm1.Button1Click(Sender: TObject);
var i: integer;
begin
  i:=1024;
  while true do
    i:=i*i;
  ShowMessage('Completed. The result is:'+i.ToString);
end;

...

 

RAD Studio 10.3.3 Arch

• "i" value is "0" on second passed, and... until infinity!
• none message error or crash on app (32/64 or Console test)
• with RANGE and OVERFLOW check, then, I have the overflow message!

hug

Edited December 2, 2020 by Guest

[FPiette 382]

28 minutes ago, emailx45 said:

with RANGE and OVERFLOW check, then, I have the overflow message!

So everything looks OK no ?

 

[Lajos Juhász 293]

Quote

So everything looks OK no ?

Technically there is no problem. However it could be improved:

 

1.) When _NewUnicodeString failes due to CharLength instead of _IntOver it should raise a more meaningful exception. For example: You cannot create string length of %d.

2.) The overflow occurs in GetTextStr. When debugging a 32 bit application you get the following call stack:

:766f9ab2 KERNELBASE.RaiseException + 0x62
:41169900 
:41169900 
System.Classes.TStrings.SaveToStream($3033D38,???)
System.Classes.TStrings.SaveToFile(???,nil)
System.Classes.TStrings.SaveToFile(???)
mainFRM.TForm1.Button2Click($2FCEEA0)
Vcl.Controls.TControl.Click
Vcl.StdCtrls.TCustomButton.Click

 

SaveToStream is focused, this almost ok as this method calls GetTextStr. I've tried again to run the 64 bit version and I get a different call stack this time:

System._RaiseAtExcept(???,???)
System.SysUtils.ErrorHandler(???,$40FA0A)
System.ErrorAt(5,$40FA0A)
System._IntOver
System._NewUnicodeString(1092000000)
System._UStrFromPWCharLen('',nil {#0},???)
System.Classes.TStrings.GetTextStr
System.Classes.TStrings.SaveToStream($302DD30,$302CDD0)
System.Classes.TStrings.SaveToFile('d:\temp\StringTest.txt',nil)
System.Classes.TStrings.SaveToFile(???)
mainFRM.TForm1.Button2Click(???)

The IDE focuses System._IntOver instead of System._NewUnicodeString.  

[Guest]

I don't know, I don't know ... my knowledge of assembly (for analize) is nil!  

 

But, I think that the report about _IntOver, in 64 bits, is valid, after all, the error happens exactly at the point where the value (integer) advances over the allowed range!

Thus, the error should not be reported on the ShowMessage line (... i.ToString), because it should not be used due to the exception thrown before it! And, not even it is inside a "Try ... Finally" block.

 

So, if this is being taken into account in the 32-bit build, then this is where another error is.

Does that make sense to you or am I completely wrong?

[Guest]

about "Disabling Optmization by default = {O-}"  

• see on Help System:  

  Other than for certain debugging situations, you should never have a need to turn optimizations off. All optimizations performed by the Delphi compiler are guaranteed not to alter the meaning of a program. In other words, the compiler performs no "unsafe" optimizations that require special awareness by the programmer. This option can only turn optimization on or off for an entire procedure or function. You cannot turn optimization on or off for a single line or group of lines within a routine.

Edited December 2, 2020 by Guest

[Guest]

my test : 

• RAD Studio 10.3.3 Arch
• CONSOLE 32 / 64bits test
• SSD 120GB Sandisk  

 

 

code source 

program WriteStingsError;

/// USER CODE WITH SOME CHANGES...

{$APPTYPE CONSOLE}
(* Windows 64-bit platform
  It takes about 20 seconds to write the file to an SSD.

  number := 20637552 - works

  Number := 21000000 - Integer overflow

  IDE stops on line in system unit:
  procedure _IntOver;
  {$IFDEF PUREPASCAL}
  begin
  ErrorAt(Byte(reIntOverflow), ReturnAddress);
  end;

  I believe it may be caused in:
  system.classes, procedure TStrings.SaveToStream *)

{$R *.res}
// .... R+,Q+,O-  // removed by my test
{ R+ }
{ Q+ }

uses
  System.SysUtils,
  System.classes,
  Vcl.Dialogs,
  System.Diagnostics;

var
  i, number  : integer;
  s          : string;
  sList      : tStringList;
  FileName   : tFileName;
  lStartTimer: TStopWatch;

begin
  lStartTimer := TStopWatch.StartNew;
  //
  FileName := '.\StringTest.txt';
  s        := '';
  //
  for i := 1 to 50 do
    s   := s + 'A';
  //
  number := 20637552; // OK
  //
  // number := 21000000; // causes integer overflow, even in 64-bit
  sList := tStringList.Create;
  //
  try
    for i := 1 to number do
      sList.add(s);
    //
    writeln('Writing ', length(sList[0]) * number, ' characters');
    //
    try
      // at 32bits, the error occurr at:
      // System.Classes.pas, line 6815 =    SetString(Result, nil, Size); // Out Memory
      sList.SaveToFile(FileName); // integer overflow if this line is present
    except
      on E: Exception do
        ShowMessage('error :' + E.ClassName + sLineBreak + E.Message);
    end;
    //
  finally
    if not(sList = nil) then
      sList.free;
    //
    lStartTimer.Stop;
    //
    writeln('Time Elapsed: ' + lStartTimer.ElapsedMilliseconds.ToString);
    writeln('File written to ', FileName);
    writeln('test done, number = ' + number.ToString);
    readln;
  end;

end.

 

hug

Edited December 2, 2020 by Guest

[David Heffernan 2345]

3 minutes ago, emailx45 said:

if not(sList = nil) then sList.free;

You don't need the if. You know that sList is not nil. Also, Free already includes a nil check. Also, there is the <> operator for not equals. 

[Guest]

Sorry! I Like this!  

[Guest]

to open the "StringTest.txt" until the Notepad++ is asking help..... 

 

Edited December 2, 2020 by Guest

[Jud 1]

17 hours ago, David Heffernan said:

Is your executable compiled with the overflow checks compiler option enabled? 

Yes, it is, and that makes no difference with an integer overflow in 64-bit mode.

[Jud 1]

17 hours ago, Lajos Juhász said:

It's quite easy to test. I've tested using:
 

procedure TForm1.Button1Click(Sender: TObject);
var i: integer;
begin
  i:=1024;
  while true do
    i:=i*i;
  ShowMessage('Completed. The result is:'+i.ToString);
end;

 

When tested using the 64 bit the call stack is:

:00007FF965893E49 ; C:\WINDOWS\System32\KERNELBASE.dll
System._RaiseAtExcept(???,???)
System.SysUtils.ErrorHandler(???,$70AE52)
System.ErrorAt(5,$70AE52)
System._IntOver
mainFRM.TForm1.Button1Click(???)
Vcl.Controls.TControl.Click
Vcl.StdCtrls.TCustomButton.Click
Vcl.StdCtrls.TCustomButton.CNCommand(???)
System.TObject.Dispatch((no value))

 

But it's not bad, double click on the mainFRM.Tform1.Button1Click and the debugger focues where the 32 bit debugger does, so it's just an extra step. 

 

My test case is different, but if I go down to "WriteStringsError.WriteStringsError and click on that, it does take me to the source line after the line that caused the error, so that helps.

[Jud 1]

17 hours ago, Lajos Juhász said:

Now back to the original problem. This is the limitation of Delphi strings. You cannot allocate string larger than: (MaxInt - SizeOf(StrRec)) div SizeOf(WideChar) = 1073741815.

 

Unfortunately, internally TStringList first tries to convert its content to string and using TEncoding to a byte array. That's why it is not suitable to work with large volume of text.

 

But with the 64-bit platform, why not increase it (use NativeInt).  Most computers these days have more than 4GB in them and hard drives 4TB and larger, with large files, are common.

[Jud 1]

7 hours ago, FPiette said:

So everything looks OK no ?

 

Not for me. I have never gotten the integer overflow error when compiling in 64-bit mode, and this goes back to the first version with 64-bit platform.

[Guest]

Quote

Not for me. I have never gotten the integer overflow error when compiling in 64-bit mode, and this goes back to the first version with 64-bit platform.

• In my tests, in 64bits dont occurr any error by EIntOverFlow!
• Only in 32bits, we have the Out Memory because:  "strings" -- NOTE: until NOTEPAD++ crash when try open your file-resulted.txt with more than size 1GB with "AAAAAAAAAAAAAAA....."
• I have 16GB RAM and CPU 4x8 i7 4770K and is not enought for open the file, because, the software used try read the file whole on memory... Software like MSWord or similar, do paging to open big-files!

• System.Classes.pas, line 6815 =    SetString(Result, nil, Size); // Out Memory

• as I show in my sample above!

 

http://docwiki.embarcadero.com/Libraries/Sydney/en/System.SysUtils.EIntOverflow

 

Did you try in your system?

[Jud 1]

10 minutes ago, emailx45 said:

• In my tests, in 64bits dont occurr any error by EIntOverFlow!
• Only in 32bits, we have the Out Memory because:  "strings" -- NOTE: until NOTEPAD++ crash when try open your file-resulted.txt with more than size 1GB with "AAAAAAAAAAAAAAA....."
• I have 16GB RAM and CPU 4x8 i7 4770K and is not enought for open the file, because, the software used try read the file whole on memory... Software like MSWord or similar, do paging to open big-files!

• 
  System.Classes.pas, line 6815 =    SetString(Result, nil, Size); // Out Memory

• as I show in my sample above!

 

http://docwiki.embarcadero.com/Libraries/Sydney/en/System.SysUtils.EIntOverflow

 

Did you try in your system?

 

[Jud 1]

11 minutes ago, emailx45 said:

• In my tests, in 64bits dont occurr any error by EIntOverFlow!
• Only in 32bits, we have the Out Memory because:  "strings" -- NOTE: until NOTEPAD++ crash when try open your file-resulted.txt with more than size 1GB with "AAAAAAAAAAAAAAA....."
• I have 16GB RAM and CPU 4x8 i7 4770K and is not enought for open the file, because, the software used try read the file whole on memory... Software like MSWord or similar, do paging to open big-files!

• 
  System.Classes.pas, line 6815 =    SetString(Result, nil, Size); // Out Memory

• as I show in my sample above!

 

http://docwiki.embarcadero.com/Libraries/Sydney/en/System.SysUtils.EIntOverflow

 

Did you try in your system?

In my development system I have 32GB and an i7.  There are 13 computers here, and their memory ranges from 32GB to 512GB.

 

Yes, you get the out of memory error in 32-bit mode, which is why I have to use 64-bit mode.  The integer overflow occurs in 64-bit mode when using tStrings.SaveToFile method.

 

The tStringList help page says that tStringList can hold up to 2,147,483,647 strings.    But if you have even 1,200,000,000 strings with even one character per sting, you will get the integer overflow error when trying to use the SaveToFile method.

And it isn't because of Windows limit on file size, which is much larger. 

 

 

[Guest]

here, I have a file created with your code with 1GBytes of size!

I tryed open in Notepad++ / Wordpad and the app stay for many times and when I clicked it the app crash!

this files very big for open all in one time!

 

maybe using another software i can open, but I dont have software for this in my pc!

[Jud 1]

5 minutes ago, emailx45 said:

here, I have a file created with your code with 1GBytes of size!

I tryed open in Notepad++ / Wordpad and the app stay for many times and when I clicked it the app crash!

this files very big for open all in one time!

 

maybe using another software i can open, but I dont have software for this in my pc!

EditPad Pro can open files bigger than 4GB. 

[Guest]

try this new approach:

 

 

program WriteStingsError2;

{$APPTYPE CONSOLE}
{$R *.res}
{ R+ }
{ Q+ }

uses
  System.SysUtils,
  System.classes,
  Vcl.Dialogs,
  System.Diagnostics;

var
  i, z, number: integer;
  s           : string;
  lStartTimer : TStopWatch;
  //
  myFile   : TextFile;
  lPlatform: string;

begin
  ReportMemoryLeaksOnShutdown := true;
  //
  {$IF Defined(WIN64)}
  lPlatform := 'MSWindows 64bits';
  {$ELSE}
  lPlatform := 'MSWindows 32bits';
  {$ENDIF}
  //
  lStartTimer := TStopWatch.StartNew;
  //
  s.Create('A', 50);
  //
  number := 20637552; // OK
  //
  // number := 21000000; // causes integer overflow, even in 64-bit
  //
  AssignFile(myFile, '.\myFileWithStrings.txt');
  try
    Rewrite(myFile); // create and open it!
    //
    writeln(lPlatform + ', writing string ' + (number * 50).tostring + ' string on ' + '.\myFileWithInteger.txt');
    //
    try
      for i := 1 to number do
      begin
        for z := 1 to 50 do
          writeln(myFile, s);
        //
        if (i mod 1000000) = 0 then // 1.000.000 each...
          writeln('was writed ' + (50 * i).tostring + ' strings');
      end;
    except
      on E: Exception do
        ShowMessage('error ...');
    end;
  finally
    CloseFile(myFile);
    //
    lStartTimer.Stop;
    //
    writeln('Time Elapsed: ' + lStartTimer.ElapsedMilliseconds.tostring);
    // writeln('File written to ', FileName);
    writeln('test done, number = ' + number.tostring);
    readln;
  end;

end.

 

hug

Edited December 3, 2020 by Guest