Jump to content
JohnLM

Streams - Writing and Reading data or text from

Recommended Posts

I am a beginner just learning a few things about Stream, how to Write and Read from it. 

 

For instance, given the code snippet below, I can write and read a text string into a TEdit. This is bare basic without Try/Finally. 

 

* code snippet extracted from one of my projects. 

var s: string;

procedure TForm1.btnStrToStreamClick(Sender: TObject); // save/write to a stream
begin
  s := 'this is a test.';
  strmSize := length(s);  // 15 chars
  strm.Write(s, strmSize);
end;

procedure TForm1.btnStrmToStringClick(Sender: TObject); // load/read into a tedit control
begin
  strm.Read(s, strmSize); // read back the 15 chars
  eb2.Text := s;          // show in tedit control
end;

 

I was wondering how FireDAC's tmemtable stores the data when the user uses the memtable.SaveToStream method.

 

I am asking because I would like to Paste some data into a dbgrid.  I don't mean to past it directly into the dbgrid.  I mean to pass the data in the clipboard into a stream that a TMemTable can deal with via Streams.  A lot of times when I have various sources that I copy/select from (web page selections, notepad, excel, ms access, etc.) and I would like to just Paste it into a dbgrid via the database memtable

 

The process flow would go something like this: (selected data -> clipboard -> [custom format/structure for dest] -> a stream -> database -> viewing control ie dbgrid) 

 

So, I would like to know if anyone knows what the structure used in the .SaveToStream is, so that I can build the same and send the clipboard contents over to it.

 

Edited by JohnLM
updated the section on flow.

Share this post


Link to post

This is not an answer, sorry.

 

But i think you need some pointers to begin with, say this code 

procedure TForm1.btnStrmToStringClick(Sender: TObject); // load/read into a tedit control
begin
  strm.Read(s, strmSize); // read back the 15 chars
  eb2.Text := s;          // show in tedit control
end;

here reside few problems, first how TStream works, stream's position need to be adjusted, in such case like above you need to use strm.Position := 0 before Read, also you need to make sure that s (the string) will have the right length to receive the data, so SetLength(s,strmSize) also needed, in your case it did work because s is not local variable.

 

As i said this is not an answer, but i think you need to be aware of TPersistent and their usage, also what is serialization and what options you have in Delphi/Pascal to serialize data, i recommend to search the net, not because we are lazy here but your question is very wide and a full answer will require a book to cover, 

Please start here with this Remy great answer 

https://stackoverflow.com/questions/14763635/can-i-serialize-a-delphi-tpersistent-as-a-field-of-tcomponent-using-the-default

 

Again please read about the subject more, and then refine your questions and we are here to help !

Share this post


Link to post
12 hours ago, JohnLM said:

I am a beginner just learning a few things about Stream, how to Write and Read from it. 

 

For instance, given the code snippet below, I can write and read a text string into a TEdit. This is bare basic without Try/Finally. 

 

* code snippet extracted from one of my projects. 


var s: string;

procedure TForm1.btnStrToStreamClick(Sender: TObject); // save/write to a stream
begin
  s := 'this is a test.';
  strmSize := length(s);  // 15 chars
  strm.Write(s, strmSize);
end;

procedure TForm1.btnStrmToStringClick(Sender: TObject); // load/read into a tedit control
begin
  strm.Read(s, strmSize); // read back the 15 chars
  eb2.Text := s;          // show in tedit control
end;

 

The code will not work as intended. Streams store bytes, not characters, and the size of a character is 2 bytes in Unicode-enabled Delphi versions. To calculate the number of bytes to write you have to use

  strmSize := Length(s) * Sizeof(char);

TStream.Read and TStream.Write are also somewhat deceptive and excellent traps for beginners :classic_cool:. They use untyped parameters and those do not work as you obviously expect from your code snippets. The compiler passes the address of the variable s to the method, not its content. So your strm.Write statement writes strmsize bytes starting from the address of s, but since strings in Delphi are reference types that address only holds a pointer pointing at the actual content of the string. You have to use

  strm.Write(s[1], strmSize);

to write the content of the string.

Reading it back has to follow the same pattern, and you have to position the stream to the start first and make sure the string you read into has enough memory allocated. Assuming the stream contains only the one string you have written into it this would look like this. I use a local string variable here to make things clearer, globals just confuse things:

 

procedure TForm1.btnStrmToStringClick(Sender: TObject); // load/read into a tedit control
var
  LTarget: string;
begin
  strm.Position := 0;
  SetLength(LTarget, strm.Size div sizeof(char))
  strm.Read(LTarget[1], Length(LTarget)* sizeof(char));
  eb2.Text := LTarget;
end;

Things get more complex if you want to store more than one string into the same stream. In such a case it is advisable to store the length of a string (as integer) before the string content, so you can read the length first to figure out how to size the target string and how many bytes to read into it.

 

Delphi has useful helper classes like TStringReader and TStringWriter to make such stuff easier. And if you only have to handle text you can put it into a TStringlist and use the list's SaveToStream and LoadFromStream methods to avoid all the hassle completely.

 

  • Like 4

Share this post


Link to post

Thanks Peter.  You helped to clear up a few things.  I remember you and a few others from TeamB from the days of old, back in the 90's/2000's, seeing your many replies.  I always wondered about the origin of TeamB, you know, how it came to be and all.  You guys were everywhere, at least in my Yahoo and Google searches in those days.  I learned a lot and also forgot a lot, LOL.  I had many personal life matters that took over my Turbo Pascal and Delphi hobbies and left me on many programming hiatuss over the years. 

 

Edited by JohnLM

Share this post


Link to post

I had other questions but am pressed for time for work.   But real quick.  I see TMemoryStream has .Clear and TStream do not.  Is there a similar way to clear streams when using TStream ?

 

In this learning portion of streams and in the test app I am using in this topic, I have been using the following setup: 

 

var
  strm: TStream;  // <-- 
  memstrm: TMemoryStream;

... 
... 

procedure tform1.FormCreate...
begin
  strm: TMemoryStream.Create; // <--
  memstrm: TMemoryStream.Create;
end;

... 
procedure clearmem;
begin
  MemStrm.clear; // reset to nothing and start over
  strm. ?        // 
end;
... 

procedure TForm1.FormDestroy(Sender: TObject);
begin
  strm.Free;
  memstrm.Free;
end;

TIA

 

Edited by JohnLM

Share this post


Link to post
10 hours ago, JohnLM said:

I had other questions but am pressed for time for work.   But real quick.  I see TMemoryStream has .Clear and TStream do not.  Is there a similar way to clear streams when using TStream ?

Setting the stream.size to 0 usually is the equivalent of Clear but that depends on the implementation of the actual TStream descendent you are using.

 

Btw.: TeamB was a group of users offering support to other users on the old Borland newsgroups. As recognition they received goodies like free product licences, a bit like the current Embarcadero MVP program, and we had dedicated support partners at Borland. All that slowly died off after Codegear took over, unfortunately.

  • Sad 1

Share this post


Link to post
On 9/15/2023 at 4:32 PM, JohnLM said:

Im a beginner just learning a few things about Stream, how to Write and Read from it.

One way to learn Stream is:

 

  1. Add a TMemo to a form in the Designer.
  2. Add lines of text to Memo1 in ObjectInspector usings Lines property editor.
  3. Select the Memo1 in design and cut/paste into notepad.
  4. Tinker with property settings streamed out in notepad.
  5. Stream the adjusted Memo1 back into the designer by pasting notepad text.
  6. Wow. 

 

Next lesson 

Place cut out of design window into code by pasting inside a buttonclick.

modify the pasted so the runtime creates the memo at runtime.🙂

...

Database can use record with fixed string size on one hand and can use JSON for streaming needs. 

 

TStream  used like TStrings in procedure arguments. MS is passed as a TStream in the TreeView.SavetoStream.

 

procedure TForm3.Button5Click(Sender: TObject);
var
  MS: TmemoryStream;
begin
  MS := TMemoryStream.Create;
  Try
    TV.SaveToStream(MS);  // TV isa TreeView
    MS.Position := 0;
    Memo1.Lines.LoadFromStream(MS);
  Finally
    MS.Free;
  End;
end;

 

 

 

 

Edited by Pat Foley
Add TStream argument example

Share this post


Link to post
On 9/17/2023 at 4:07 AM, PeterBelow said:

Btw.: TeamB was a group of users offering support to other users on the old Borland newsgroups. As recognition they received goodies like free product licences, a bit like the current Embarcadero MVP program, and we had dedicated support partners at Borland. All that slowly died off after Codegear took over, unfortunately.

But at least we get some of that back via Embarcadero MVP! :classic_biggrin:

Share this post


Link to post
7 hours ago, Die Holländer said:

Your site (Lebeau Software) is not available anymore.

It is working fine for me.

7 hours ago, Die Holländer said:

Are you planning to continue with the site?

Yes.  My hosting provider did perform maintenance on my site's server yesterday, so maybe they messed something up, or have already resolved the issue.

Share this post


Link to post

Update on this project endeavor. . .

 

RE: creating a Hex Dump feature - Reading stream data and converting to a hex-dump output.

 

I finally managed to get this bare-bones of a hex-dump feature added to this test app as part of this project, where I am trying to figure out if it's possible to obtain the data behind or inside the memtable's stream and then be able to read/write back and forth data to the stream. 

 

I struggled with this output shown below. I had trouble understanding ASCII and printable chars based on raw data as seen below.  I searched for several days on the ASCII code and Code Page and Unicode, and so on.  To be honest, I still do not fully understand it all but I only wanted to get a display like the one shown below. 

 

1 - 0015 through 1217 is the Byte position in row number format, in steps of 16 bytes into the stream and is representative of the size of the stream (from TMemTable), or strm.size;
2 - first line, and so on down the list, -- 41..00 is the conversion from Byte to Hex
3 - first line, and so on down the list, -- "ADBS...." is the char representation.

4 - the dots "." that you see are supposedly non-printable chars

5 - the TMemTable's is connected to a dbgrid.  The grid has 4 fields: (IMS, ItemNo, Category, Vendor) - those fieldnames are showing in the hex dump below.  This data is pulled from the memtabl's stream and processed.

 

note:  I did the best I could in the time frame I have currently in creating this feature.  And, please don't expect it to follow any standard hex dump editor perfectly.  I have not compared it to any of those as yet.  But for something like this, its a first-time for me without blueprints or someone else's code. 

0015: 41 44 42 53 10 00 00 00 06 02 00 00 FF 00 01 00  - ADBS............
0031: 01 FF 02 FF 03 04 00 08 00 00 00 6D 00 65 00 6D  - ...........m.e.m
0047: 00 31 00 05 00 08 00 00 00 6D 00 65 00 6D 00 31  - .1.......m.e.m.1
0063: 00 06 00 00 00 00 00 07 00 00 08 00 32 00 00 00  - ............2...
0079: 09 00 00 FF 0A FF 0B 04 00 06 00 00 00 49 00 4D  - .............I.M
0095: 00 53 00 05 00 06 00 00 00 49 00 4D 00 53 00 0C  - .S.......I.M.S..
0111: 00 01 00 00 00 0E 00 0D 00 0F 00 01 10 00 01 11  - ................
0127: 00 01 12 00 01 13 00 01 14 00 01 15 00 06 00 00  - ................
0143: 00 49 00 4D 00 53 00 FE FF 0B 04 00 0C 00 00 00  - .I.M.S..........
0159: 49 00 74 00 65 00 6D 00 4E 00 6F 00 05 00 0C 00  - I.t.e.m.N.o.....
0175: 00 00 49 00 74 00 65 00 6D 00 4E 00 6F 00 0C 00  - ..I.t.e.m.N.o...
0191: 02 00 00 00 0E 00 16 00 17 00 14 00 00 00 0F 00  - ................
0207: 01 10 00 01 11 00 01 12 00 01 13 00 01 14 00 01  - ................
0223: 15 00 0C 00 00 00 49 00 74 00 65 00 6D 00 4E 00  - ......I.t.e.m.N.
0239: 6F 00 18 00 14 00 00 00 FE FF 0B 04 00 10 00 00  - o...............
0255: 00 43 00 61 00 74 00 65 00 67 00 6F 00 72 00 79  - .C.a.t.e.g.o.r.y
0271: 00 05 00 10 00 00 00 43 00 61 00 74 00 65 00 67  - .......C.a.t.e.g
0287: 00 6F 00 72 00 79 00 0C 00 03 00 00 00 0E 00 16  - .o.r.y..........
0303: 00 17 00 14 00 00 00 0F 00 01 10 00 01 11 00 01  - ................
0319: 12 00 01 13 00 01 14 00 01 15 00 10 00 00 00 43  - ...............C
0335: 00 61 00 74 00 65 00 67 00 6F 00 72 00 79 00 18  - .a.t.e.g.o.r.y..
0351: 00 14 00 00 00 FE FF 0B 04 00 0C 00 00 00 56 00  - ..............V.
0367: 65 00 6E 00 64 00 6F 00 72 00 05 00 0C 00 00 00  - e.n.d.o.r.......
0383: 56 00 65 00 6E 00 64 00 6F 00 72 00 0C 00 04 00  - V.e.n.d.o.r.....
0399: 00 00 0E 00 16 00 17 00 14 00 00 00 0F 00 01 10  - ................
0415: 00 01 11 00 01 12 00 01 13 00 01 14 00 01 15 00  - ................
0431: 0C 00 00 00 56 00 65 00 6E 00 64 00 6F 00 72 00  - ....V.e.n.d.o.r.
0447: 18 00 14 00 00 00 FE FE FF 19 FE FF 1A FE FF 1B  - ................
0463: FF 1C 1D 00 00 00 00 00 FF 1E 00 00 01 00 00 00  - ................
0479: 01 00 05 00 00 00 41 41 41 41 41 02 00 05 00 00  - ......AAAAA.....
0495: 00 42 42 42 42 42 03 00 05 00 00 00 43 43 43 43  - .BBBBB......CCCC
0511: 43 FE FE FE FE FE FF 1F FE FF 20 21 00 02 00 00  - C........ !....
0527: 00 FF 22 FE FE FE 0E 00 4D 00 61 00 6E 00 61 00  - ..".....M.a.n.a.
0543: 67 00 65 00 72 00 1E 00 55 00 70 00 64 00 61 00  - g.e.r...U.p.d.a.
0559: 74 00 65 00 73 00 52 00 65 00 67 00 69 00 73 00  - t.e.s.R.e.g.i.s.
0575: 74 00 72 00 79 00 12 00 54 00 61 00 62 00 6C 00  - t.r.y...T.a.b.l.
0591: 65 00 4C 00 69 00 73 00 74 00 0A 00 54 00 61 00  - e.L.i.s.t...T.a.
0607: 62 00 6C 00 65 00 08 00 4E 00 61 00 6D 00 65 00  - b.l.e...N.a.m.e.
0623: 14 00 53 00 6F 00 75 00 72 00 63 00 65 00 4E 00  - ..S.o.u.r.c.e.N.
0639: 61 00 6D 00 65 00 0A 00 54 00 61 00 62 00 49 00  - a.m.e...T.a.b.I.
0655: 44 00 24 00 45 00 6E 00 66 00 6F 00 72 00 63 00  - D.$.E.n.f.o.r.c.
0671: 65 00 43 00 6F 00 6E 00 73 00 74 00 72 00 61 00  - e.C.o.n.s.t.r.a.
0687: 69 00 6E 00 74 00 73 00 1E 00 4D 00 69 00 6E 00  - i.n.t.s...M.i.n.
0703: 69 00 6D 00 75 00 6D 00 43 00 61 00 70 00 61 00  - i.m.u.m.C.a.p.a.
0719: 63 00 69 00 74 00 79 00 18 00 43 00 68 00 65 00  - c.i.t.y...C.h.e.
0735: 63 00 6B 00 4E 00 6F 00 74 00 4E 00 75 00 6C 00  - c.k.N.o.t.N.u.l.
0751: 6C 00 14 00 43 00 6F 00 6C 00 75 00 6D 00 6E 00  - l...C.o.l.u.m.n.
0767: 4C 00 69 00 73 00 74 00 0C 00 43 00 6F 00 6C 00  - L.i.s.t...C.o.l.
0783: 75 00 6D 00 6E 00 10 00 53 00 6F 00 75 00 72 00  - u.m.n...S.o.u.r.
0799: 63 00 65 00 49 00 44 00 0E 00 64 00 74 00 49 00  - c.e.I.D...d.t.I.
0815: 6E 00 74 00 33 00 32 00 10 00 44 00 61 00 74 00  - n.t.3.2...D.a.t.
0831: 61 00 54 00 79 00 70 00 65 00 14 00 53 00 65 00  - a.T.y.p.e...S.e.
0847: 61 00 72 00 63 00 68 00 61 00 62 00 6C 00 65 00  - a.r.c.h.a.b.l.e.
0863: 12 00 41 00 6C 00 6C 00 6F 00 77 00 4E 00 75 00  - ..A.l.l.o.w.N.u.
0879: 6C 00 6C 00 08 00 42 00 61 00 73 00 65 00 14 00  - l.l...B.a.s.e...
0895: 4F 00 41 00 6C 00 6C 00 6F 00 77 00 4E 00 75 00  - O.A.l.l.o.w.N.u.
0911: 6C 00 6C 00 12 00 4F 00 49 00 6E 00 55 00 70 00  - l.l...O.I.n.U.p.
0927: 64 00 61 00 74 00 65 00 10 00 4F 00 49 00 6E 00  - d.a.t.e...O.I.n.
0943: 57 00 68 00 65 00 72 00 65 00 1A 00 4F 00 72 00  - W.h.e.r.e...O.r.
0959: 69 00 67 00 69 00 6E 00 43 00 6F 00 6C 00 4E 00  - i.g.i.n.C.o.l.N.
0975: 61 00 6D 00 65 00 18 00 64 00 74 00 41 00 6E 00  - a.m.e...d.t.A.n.
0991: 73 00 69 00 53 00 74 00 72 00 69 00 6E 00 67 00  - s.i.S.t.r.i.n.g.
1007: 08 00 53 00 69 00 7A 00 65 00 14 00 53 00 6F 00  - ..S.i.z.e...S.o.
1023: 75 00 72 00 63 00 65 00 53 00 69 00 7A 00 65 00  - u.r.c.e.S.i.z.e.
1039: 1C 00 43 00 6F 00 6E 00 73 00 74 00 72 00 61 00  - ..C.o.n.s.t.r.a.
1055: 69 00 6E 00 74 00 4C 00 69 00 73 00 74 00 10 00  - i.n.t.L.i.s.t...
1071: 56 00 69 00 65 00 77 00 4C 00 69 00 73 00 74 00  - V.i.e.w.L.i.s.t.
1087: 0E 00 52 00 6F 00 77 00 4C 00 69 00 73 00 74 00  - ..R.o.w.L.i.s.t.
1103: 06 00 52 00 6F 00 77 00 0A 00 52 00 6F 00 77 00  - ..R.o.w...R.o.w.
1119: 49 00 44 00 10 00 4F 00 72 00 69 00 67 00 69 00  - I.D...O.r.i.g.i.
1135: 6E 00 61 00 6C 00 18 00 52 00 65 00 6C 00 61 00  - n.a.l...R.e.l.a.
1151: 74 00 69 00 6F 00 6E 00 4C 00 69 00 73 00 74 00  - t.i.o.n.L.i.s.t.
1167: 1C 00 55 00 70 00 64 00 61 00 74 00 65 00 73 00  - ..U.p.d.a.t.e.s.
1183: 4A 00 6F 00 75 00 72 00 6E 00 61 00 6C 00 12 00  - J.o.u.r.n.a.l...
1199: 53 00 61 00 76 00 65 00 50 00 6F 00 69 00 6E 00  - S.a.v.e.P.o.i.n.
1215: 74 00 0E 00 43 00 68 00 61 00 6E 00 67 00 65 00  - t...C.h.a.n.g.e.
1217: 73 00                                            - s.

 

Edited by JohnLM
spelling/typo, added bullet 5

Share this post


Link to post
7 hours ago, JohnLM said:

note:  I did the best I could in the time frame I have currently in creating this feature.  And, please don't expect it to follow any standard hex dump editor perfectly.  I have not compared it to any of those as yet.  But for something like this, its a first-time for me without blueprints or someone else's code. 

You did a nice job there, also there is no standard hex dump, for the most used cases, have a look here https://www.asciitable.com/

The range is [32..126] , and for the extended it will be [32..126,128..254], and that it.

 

8 hours ago, JohnLM said:

1 - 0015 through 1217 is the Byte position in row number format,

One thing though about this, the first column should be the offset (or what you called the position), meaning the first raw should be 0000 and the next 0016.

 

Another thing is refactor that piece of code, into completely agnostic input, meaning let the new refactored code take as input a pointer and size, TMemoryStream as example has Stream.Position returns a pointer to memory where the data reside, you can use it with a string when the content are corrupted to check what exactly went wrong, it could be a function return a string, and you can use it as your own code in separated unit (your own library), you will reuse that function in the future with different projects.

 

In my own code i have the same, but sometimes i replace the offset with the address, or show both, but you rarely need the address itself.

Share this post


Link to post

Thank you for your positive feedback. 

 

I failed to mention and add another bullet, to describe the field values showing in the above-posted hex dump output, the values (AAAAA, BBBBB, CCCCC), this was the data I stored inside the dbgrid that I .SaveToStream and then copied all its content into the bufArray variable for processing.

 

The one CON I see in this hex dump feature is that it is slow. The method I used to process and then update the memo control is quite slow, about 1.25 secs to go through a 1.3k to 1.5k byte array to show the hex dump data. This includes using the .beginupdate and .endupdate settings. 

 

In order to show a clean output, I am using several functions: 

 

PadZero(N: longint; Len: integer): string; -- show 0000, 0001, 0999, ... etc
ByteToIntChar(b: byte): char; -- shows ascii 32..126 chars
PadSpaceRt(s: string; Len: integer): string; -- aligns the ascii char pain

 

The above gets processed in the inner for-loop: (for row do.., for col do..)

 

But at least it works, and that is what matters. 

Share this post


Link to post
3 hours ago, JohnLM said:

about 1.25 secs to go through a 1.3k to 1.5k byte array to show the hex dump data.

:classic_blink:

That is very slow.

 

3 hours ago, JohnLM said:

In order to show a clean output, I am using several functions: 

 

PadZero(N: longint; Len: integer): string; -- show 0000, 0001, 0999, ... etc
ByteToIntChar(b: byte): char; -- shows ascii 32..126 chars
PadSpaceRt(s: string; Len: integer): string; -- aligns the ascii char pain

 

The above gets processed in the inner for-loop: (for row do.., for col do..)

That is the reason of this slowness.

Drop all of these, make sure you are using one, yes one loop !

PadZero is not needed at all as you will use IntToHex with like 2 digits,

if while..do as one loop is too complicated for first write, then you use one while with increase over 16 and inner for/while with 16, the best approach is one while..do though.

You can consider this as learning adventure, you build the content on two strings at the same time, the hex and the printed char then combine them at the end of each line, adding them to the result, read a byte once and get its hex value and its printable character.

 

ByteToIntChar is not neede just use char(x) or char(pX^),, or simple Char(oneC).

PadSpaceRt honestly ,i never heard of it but can imagine its job, simple concatenation with + is more than enough, in fact is faster than any function you think of in Delphi.

 

Share this post


Link to post

A few minutes after my post above, I made the discovery of the cause of slowness.  Next, I will look into figuring out how to time the output speed in seconds or ms when I figure that out.  I want to show that as part of a process time somewhere in the output window.  Then, I can test the efficiency of various ideas. 

 

Anyway, to explain the slowness..

 

In this line below, in the inner for/loop, I was testing some values and forgot to remove those part(s) of code after inc(index);  I REM them out as seen below, and the output is now instant.  Those code fragments were two TStaticText controls that I was using to debug the index counts I was having trouble with.  TStaticText displays values without interruption while TMemo updates in real-time during for/loops.  If I use plain TEdit controls, then the updates display at the end of the TMemo updates. So, I don't use tedit's for this purpose during debugging real-time outputs. 

 

for row do. .

  for col do. .

    inc(index); //st1.Caption := inttostr(index); st2.Caption := inttostr(x);

  // col

// row

 

(I am going to branch this hex dump feature into another separate project--will call it HexdumpViewer or something--and will add some other interesting ideas to it.  I have a few already and want to try them out soon.)

 

I will look at your other ideas/suggestions, thanks. 

 

Edited by JohnLM
typo / cleanup

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×