Jump to content
bernhard_LA

Load a String from a file - returns strange char set

Recommended Posts

I'm almost  sure that I already  used the  code below without any  issues  in previous application. Now a added this  to a new app and   

dumped the Loadstring  variable into a memo with 

 memo.lines.add (  Loadstring ) ; 

All text is now displayed using chines char's , the file I#m reading for sure is ASCII and  only latin chars .....

What went wrong ?

 

 

 var LoadString: AnsiString;

FS: TFileStream;
begin
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
  try
    SetLength(LoadString, (FS.Size div SizeOf(Char)));
    if FS.Size > 0 then
      FS.Read(Pointer(LoadString)^, FS.Size);
  finally
    FS.Free

;

Edited by bernhard_LA

Share this post


Link to post

thanks for helping  -  found also my mistake

 

after change  from 

 

 

var LoadString: String;

 

to

 

var LoadString: AnsiString;

 

code is working,  with a typecast  String(Loadstring)  ;  but  now different  issue .... this will now work for compiling  under LINUX  😞 

 

Edited by bernhard_LA

Share this post


Link to post

Like this?

 

var
  S: String;
  F: TFileStream;
  B: TBytes;
begin
  S := '';
  F := TFileStream.Create('test.txt', fmOpenRead);
  try
    if F.Size > 0 then begin
      SetLength(B, F.Size);
      F.Read(B[0], Length(B));
      S := TEncoding.ANSI.GetString(B);
      WriteLn('S = ', S);
    end;
  finally
    F.Free;
  end;
end;

 

  • Thanks 1

Share this post


Link to post

Why are you using AnsiString at all?  You should be using (Unicode)String instead, since that is what the TMemo expects.

 

In any case, Char is WideChar in D2009+, so SizeOf(Char) is 2 not 1, so you are allocating memory for your AnsiString for only 1/2 of the file data, but then reading the full file into that memory.  So you are going to corrupt surrounding memory.  Use SizeOf(AnsiChar) instead, which is 1 so you can just drop the SizeOf() altogether.

var
  LoadString: AnsiString;
  FS: TFileStream;
begin
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
  try
    SetLength(LoadString, FS.Size);
    FS.ReadBuffer(Pointer(LoadString)^, FS.Size);
  finally
    FS.Free;
  end;
  Memo.Lines.Add(String(LoadString));
end;

Alternatively, there are other options, such as TMemoryStream.LoadFromFile():

uses
  System.Classes;

var
  LoadString: AnsiString;
  MS: TMemoryStream;
begin
  MS := TMemoryStream.Create;
  try
    MS.LoadFromFile(FileName);
    SetString(LoadString, PAnsiChar(MS.Memory), MS.Size);
  finally
    MS.Free;
  end;
  Memo.Lines.Add(String(LoadString));
end;

Or TStreamReader:

uses
  System.Classes, System.SysUtils;

var
  LoadString: String;
  Reader: TStreamReader;
begin
  Reader := TStreamReader.Create(FileName, TEncoding.ANSI);
  try
    LoadString := Reader.ReadToEnd;
  finally
    Reader.Free;
  end;
  Memo.Lines.Add(LoadString);
end;

Or TFile.ReadAllText():

uses
  System.IOUtils, System.SysUtils;

var
  LoadString: String;
begin
  LoadString := TFile.ReadAllText(FileName, TEncoding.ANSI);
  Memo.Lines.Add(LoadString);
end;

 

 

Edited by Remy Lebeau
  • Like 4
  • Thanks 1

Share this post


Link to post

Memo.Lines.LoadFromFile suffers from performance issues if the file is n x 10 MB, especially if it is on a network location. I changed most of my sensitive methods to use Streams instead:

  fs := TFileStream.Create('filename.txt', fmOpenRead + fmShareDenyNone);
  Try
   Memo.Lines.BeginUpdate;
   Memo.Lines.LoadFromStream(fs);
  Finally
   FreeAndNil(fs);
   Memo.Lines.EndUpdate;
  End;

 

Share this post


Link to post
1 hour ago, aehimself said:

Memo.Lines.LoadFromFile suffers from performance issues if the file is n x 10 MB, especially if it is on a network location. I changed most of my sensitive methods to use Streams instead:


  fs := TFileStream.Create('filename.txt', fmOpenRead + fmShareDenyNone);
  Try
   Memo.Lines.BeginUpdate;
   Memo.Lines.LoadFromStream(fs);
  Finally
   FreeAndNil(fs);
   Memo.Lines.EndUpdate;
  End;

 

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream. 

Share this post


Link to post
3 minutes ago, David Heffernan said:

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream. 

Code is from D7-era. Maybe it worked different back then...? In 10.2 .LoadFromFile effectively does the same (only without the .BeginUpdate and .EndUpdate).

But I remember that I had to change due to insanely long loading times.

Share this post


Link to post
4 hours ago, aehimself said:

Code is from D7-era. Maybe it worked different back then...? In 10.2 .LoadFromFile effectively does the same (only without the .BeginUpdate and .EndUpdate).

But I remember that I had to change due to insanely long loading times.

I don't think that there was any difference regarding the streams. I think you've just mixed up the begin/end update with the streams and got a bit of cargo cult. FWIW your try/finally is not right.

Share this post


Link to post
2 hours ago, David Heffernan said:

FWIW your try/finally is not right.

...and this is from an era when I was already using Try ... Finally blocks!

Seriously, most of my first codes make me want to give up development and be a baker instead.

  • Haha 2

Share this post


Link to post

TStringList first reads whole file to a buffer, recodes it to string and then splits it up to lines. While it's the simplest method, it could consume damn much memory (up to 5*FileSize when reading one-byte encoding). So for large sizes one usually prefers some custom read/split implementation

Share this post


Link to post
On 11/8/2019 at 5:43 AM, David Heffernan said:

This looks a bit bogus to my eyes. Why would this change anything. Surely LoadFromFile use a file stream.

Yes, it does:

procedure TStrings.LoadFromFile(const FileName: string);
var
  Stream: TStream;
begin
  Stream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    LoadFromStream(Stream);
  finally
    Stream.Free;
  end;
end;

procedure TStrings.LoadFromFile(const FileName: string; Encoding: TEncoding);
var
  Stream: TStream;
begin
  Stream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    LoadFromStream(Stream, Encoding);
  finally
    Stream.Free;
  end;
end;

procedure TStrings.LoadFromStream(Stream: TStream);
begin
  LoadFromStream(Stream, nil);
end;

procedure TStrings.LoadFromStream(Stream: TStream; Encoding: TEncoding);
var
  Size: Integer;
  Buffer: TBytes;
begin
  BeginUpdate;
  try
    Size := Stream.Size - Stream.Position;
    SetLength(Buffer, Size);
    Stream.Read(Buffer, 0, Size);
    Size := TEncoding.GetBufferEncoding(Buffer, Encoding, FDefaultEncoding);
    SetEncoding(Encoding); // Keep Encoding in case the stream is saved
    SetTextStr(Encoding.GetString(Buffer, Size, Length(Buffer) - Size));
  finally
    EndUpdate;
  end;
end;
Quote

TStringList first reads whole file to a buffer, recodes it to string and then splits it up to lines.

You mean TStrings, not TStringList.  TMemo.Lines (aka the TMemoStrings class) derives from TStrings directly, not from TStringList.  But yes, TStrings.LoadFromStream() behaves the way you describe (see above), TMemoStrings does not override that behavior.

Quote

While it's the simplest method, it could consume damn much memory (up to 5*FileSize when reading one-byte encoding). So for large sizes one usually prefers some custom read/split implementation

Yes, the way TStrings.LoadFromStream() was implemented is really inefficient for large amounts of text.

Edited by Remy Lebeau

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×