Load a String from a file - returns strange char set

[bernhard_LA 0]

I'm almost  sure that I already  used the  code below without any  issues  in previous application. Now a added this  to a new app and   

dumped the Loadstring  variable into a memo with 

 memo.lines.add (  Loadstring ) ; 

All text is now displayed using chines char's , the file I#m reading for sure is ASCII and  only latin chars .....

What went wrong ?

 

 

 var LoadString: AnsiString;

FS: TFileStream;
begin
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
  try
    SetLength(LoadString, (FS.Size div SizeOf(Char)));
    if FS.Size > 0 then
      FS.Read(Pointer(LoadString)^, FS.Size);
  finally
    FS.Free

;

Edited November 6, 2019 by bernhard_LA

[Vandrovnik 214]

if FS.Size > 0 then
      FS.Read(LoadString[low(LoadString)], FS.Size);

[bernhard_LA 0]

thanks for helping  -  found also my mistake : 

 

after change  from 

 

 

var LoadString: String;

 

to

 

var LoadString: AnsiString;

 

code is working,  with a typecast  String(Loadstring)  ;  but  now different  issue .... this will now work for compiling  under LINUX  😞 

 

Edited November 6, 2019 by bernhard_LA

[Alexander Elagin 143]

Like this?

 

var
  S: String;
  F: TFileStream;
  B: TBytes;
begin
  S := '';
  F := TFileStream.Create('test.txt', fmOpenRead);
  try
    if F.Size > 0 then begin
      SetLength(B, F.Size);
      F.Read(B[0], Length(B));
      S := TEncoding.ANSI.GetString(B);
      WriteLn('S = ', S);
    end;
  finally
    F.Free;
  end;
end;

 

[Remy Lebeau 1394]

Why are you using AnsiString at all?  You should be using (Unicode)String instead, since that is what the TMemo expects.

 

In any case, Char is WideChar in D2009+, so SizeOf(Char) is 2 not 1, so you are allocating memory for your AnsiString for only 1/2 of the file data, but then reading the full file into that memory.  So you are going to corrupt surrounding memory.  Use SizeOf(AnsiChar) instead, which is 1 so you can just drop the SizeOf() altogether.

var
  LoadString: AnsiString;
  FS: TFileStream;
begin
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
  try
    SetLength(LoadString, FS.Size);
    FS.ReadBuffer(Pointer(LoadString)^, FS.Size);
  finally
    FS.Free;
  end;
  Memo.Lines.Add(String(LoadString));
end;

Alternatively, there are other options, such as TMemoryStream.LoadFromFile():

uses
  System.Classes;

var
  LoadString: AnsiString;
  MS: TMemoryStream;
begin
  MS := TMemoryStream.Create;
  try
    MS.LoadFromFile(FileName);
    SetString(LoadString, PAnsiChar(MS.Memory), MS.Size);
  finally
    MS.Free;
  end;
  Memo.Lines.Add(String(LoadString));
end;

Or TStreamReader:

uses
  System.Classes, System.SysUtils;

var
  LoadString: String;
  Reader: TStreamReader;
begin
  Reader := TStreamReader.Create(FileName, TEncoding.ANSI);
  try
    LoadString := Reader.ReadToEnd;
  finally
    Reader.Free;
  end;
  Memo.Lines.Add(LoadString);
end;

Or TFile.ReadAllText():

uses
  System.IOUtils, System.SysUtils;

var
  LoadString: String;
begin
  LoadString := TFile.ReadAllText(FileName, TEncoding.ANSI);
  Memo.Lines.Add(LoadString);
end;

 

 

Edited November 6, 2019 by Remy Lebeau

[Cristian Peța 103]

Or directly if you don't need to add but to replace.

  Memo.Lines.LoadFromFile(FileName, TEncoding.ASCII);

 

[aehimself 396]

Memo.Lines.LoadFromFile suffers from performance issues if the file is n x 10 MB, especially if it is on a network location. I changed most of my sensitive methods to use Streams instead:

  fs := TFileStream.Create('filename.txt', fmOpenRead + fmShareDenyNone);
  Try
   Memo.Lines.BeginUpdate;
   Memo.Lines.LoadFromStream(fs);
  Finally
   FreeAndNil(fs);
   Memo.Lines.EndUpdate;
  End;

 

[David Heffernan 2345]

1 hour ago, aehimself said:

Memo.Lines.LoadFromFile suffers from performance issues if the file is n x 10 MB, especially if it is on a network location. I changed most of my sensitive methods to use Streams instead:

  fs := TFileStream.Create('filename.txt', fmOpenRead + fmShareDenyNone);
  Try
   Memo.Lines.BeginUpdate;
   Memo.Lines.LoadFromStream(fs);
  Finally
   FreeAndNil(fs);
   Memo.Lines.EndUpdate;
  End;

 

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream. 

[aehimself 396]

3 minutes ago, David Heffernan said:

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream. 

Code is from D7-era. Maybe it worked different back then...? In 10.2 .LoadFromFile effectively does the same (only without the .BeginUpdate and .EndUpdate).

But I remember that I had to change due to insanely long loading times.

[David Heffernan 2345]

4 hours ago, aehimself said:

Code is from D7-era. Maybe it worked different back then...? In 10.2 .LoadFromFile effectively does the same (only without the .BeginUpdate and .EndUpdate).

But I remember that I had to change due to insanely long loading times.

I don't think that there was any difference regarding the streams. I think you've just mixed up the begin/end update with the streams and got a bit of cargo cult. FWIW your try/finally is not right.

[aehimself 396]

2 hours ago, David Heffernan said:

FWIW your try/finally is not right.

...and this is from an era when I was already using Try ... Finally blocks!

Seriously, most of my first codes make me want to give up development and be a baker instead.

[Fr0sT.Brutal 900]

TStringList first reads whole file to a buffer, recodes it to string and then splits it up to lines. While it's the simplest method, it could consume damn much memory (up to 5*FileSize when reading one-byte encoding). So for large sizes one usually prefers some custom read/split implementation

[Remy Lebeau 1394]

On 11/8/2019 at 5:43 AM, David Heffernan said:

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream.

Yes, it does:

procedure TStrings.LoadFromFile(const FileName: string);
var
  Stream: TStream;
begin
  Stream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    LoadFromStream(Stream);
  finally
    Stream.Free;
  end;
end;

procedure TStrings.LoadFromFile(const FileName: string; Encoding: TEncoding);
var
  Stream: TStream;
begin
  Stream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    LoadFromStream(Stream, Encoding);
  finally
    Stream.Free;
  end;
end;

procedure TStrings.LoadFromStream(Stream: TStream);
begin
  LoadFromStream(Stream, nil);
end;

procedure TStrings.LoadFromStream(Stream: TStream; Encoding: TEncoding);
var
  Size: Integer;
  Buffer: TBytes;
begin
  BeginUpdate;
  try
    Size := Stream.Size - Stream.Position;
    SetLength(Buffer, Size);
    Stream.Read(Buffer, 0, Size);
    Size := TEncoding.GetBufferEncoding(Buffer, Encoding, FDefaultEncoding);
    SetEncoding(Encoding); // Keep Encoding in case the stream is saved
    SetTextStr(Encoding.GetString(Buffer, Size, Length(Buffer) - Size));
  finally
    EndUpdate;
  end;
end;

Quote

TStringList first reads whole file to a buffer, recodes it to string and then splits it up to lines.

You mean TStrings, not TStringList.  TMemo.Lines (aka the TMemoStrings class) derives from TStrings directly, not from TStringList.  But yes, TStrings.LoadFromStream() behaves the way you describe (see above), TMemoStrings does not override that behavior.

Quote

While it's the simplest method, it could consume damn much memory (up to 5*FileSize when reading one-byte encoding). So for large sizes one usually prefers some custom read/split implementation

Yes, the way TStrings.LoadFromStream() was implemented is really inefficient for large amounts of text.

Edited November 11, 2019 by Remy Lebeau