TCP Receiving Binary File

[Bob4231 2]

Hello, I'm new to the forum.

 

I recently found ICS Component Suite, it looks amazing  and would like to use to receive a binary file (jpg picture) at computer, from a ESP32 WiFi device.

I'm stick with the SendFile example from the UserMade section. I'm running on Delphi XE10.2.

 

example is almost untouched.

I open the port at RcvFile and start listening. Then, send the file name & size on the ESP32. Then the RcvFile go to binary mode, and begin to receive the bytes,

File bytes received ok, but when I try to open the file (picture) always a small top section is correct but the rest is scrambled.

 

I tried to insert a delay when sending the packets at the WiFi device side, but result is the same. My packet size is 1436 bytes.

 

The I also tried between the example itself to send a picture file using SendFile  -->  RcvFile  examples, but also the same result.

 

Is this me or it has been happen to someone else..?

 

 

I tried to follow up thru the socket receive event but still too complicated for me  :S

 

 

Trying to well understand the line:

 

Count := Receive(Rcvd + WritePtr, FileSize - WritePtr);

 

I would like to receive each bytes pack and append to a TFileStream directly, instead of putting in memory, as I tried to send a 1Gb file and the original example failed because out of memory.

 

Thanks In Advance!

 

Edited April 8, 2019 by Bob4231
misspell

[Angus Robertson 520]

2 hours ago, Bob4231 said:

I recently found ICS Component Suite, it looks amazing  and would like to use to receive a binary file (jpg picture) at computer, from a ESP32 WiFi device.

I'm stick with the SendFile example from the UserMade section. I'm running on Delphi XE10.2.

 

The user made samples are 15 to 20 years old and few probably work with the latest version of ICS. You should really be looking at OverbyteIcsBinCliDemo.dpr in Samples\delphi\SocketDemos which is more up to date. But still does not use a stream. 

 

So perhaps look at a new sample OverbyteIcsIpStmLogTst.dpr in \Samples\delphi\sslinternet which does receive binary streams of unlimited size. 

 

Angus

 

[FPiette 363]

The problem with this demo is a Unicode issue. The author made it with a Delphi version before Unicode was introduced. This demo assume a char in an AnsiChar (8 bits) while for years char are Unicode (16 bits).

This is not difficult to fix. The buffer used to send/receive binary data must by an array of byte instead of an array of char.

For the filename, you have to choose if you want to send Unicode filename or ASCII filename and convert appropriately.

Do you need more help to fix the sample?

 

François Piette

ICS Authour

Embarcadero MVP

 

[FPiette 363]

I have fixed the demo to work with all Unicode enabled Delphi versions (2009 and up).

It will no more work with older ascii only versions.

I have tested with Delphi 10.3.1 rio.

The file sendfile.zip has been updated on the website, you can download again from usermade page at http://www.overbyte.be

 

Let me know if it works for you.

[Bob4231 2]

Woow really appreciated; thank you a lot.

I'm traveling but will test tonight at the hotel !

 

It worked great; file now copies perfectly well (localhost).

 

Will test with the ESP32 once in home  : )

 

 

Edited April 9, 2019 by Bob4231
update

[Bob4231 2]

 

Francois thank you for excellent ICS components and all the time you have been dedicated to the Delphi development.

It is indeed a highly optimized source code; I need to learn more in detail. 

 

The SendFile example you fixed works outstanding; the ESP32 can send the files without problem. I'm now optimizing the packet size at the ESP32 side.

 

As Angus pointed, I'm studying also the OverbyteIcsBinCliDemo, I need to be more familiar with the pointers things  : )

This is a good example to learn.

 

Thank you!

 

[Bob4231 2]

Hello!

I have been testing with the OverbyteIcsBinCliDemo and the OverbyteIcsTcpSrv examples.

 

From what I have learned, when the Client asks for binary data, one need to send the next request:

binary [size] + #13#10

 

So it is recognized at the Server example and sends back the following answer, followed by the Id-Header and the payload.

                Client.SendStr('200 OK Binary ' + IntToStr(I) +
                           ' bytes requested'#13#10);

The Client looks for the "200 Ok Binary" substring to understand the binary transfer will follow up. This change the receiving mode to Binary.

The first binary data consists of a 4 byte ID and a 4 byte data size.

  THdrRec = record
    case Integer of
      0: (
        ID1     : Byte;
        ID2     : Byte;
        ID3     : Byte;
        ID4     : Byte;
        SizeLo  : Word;
        SizeHi  : Word);
      1: (
        ID    : Longint;
        Size  : LongInt);
  end;

Then, the Server example will send as many bytes as the Size indicated to the Client, ended with an "E" character.

 

Have been working with the examples to use for a file transfer between the ESP32 and the computer.

I have used the 4 bytes header to send the file size. 

 

I have adjusted the ESP32 side for a binary message write of 2048 bytes. The OverbyteIcsBinCliDemo receives the data chunks and I direct it to a filestream so the file is built. All this works great at this point. So far with the BinCliDemo fairly modified to save the file, I get a file transfer speed of 100 Kbyte/second (around 800 kbps). Files receives ok, no corruption. This a really exciting experience to see how the file 'downloads' from the ESP32.

 

Now I'm at the step to improve the speed for the data transfer. I have tried to increase the ESP32 buffer size but this will not work as the MTU is limited. I tried different sizes anyway and with higher values the time increases as errors are produced as packet re-transmission or duped packets. 

 

If the data transmission speed is limited by the healthy handshaking of the TCP client connection and MTU limitation, I come with my beginner idea of using two client connections to send the half of the file over each one. My thoughts are; if I can process each independent client data on a separate buffers/process, I would be able to get the two halves of the file (say 1 mbyte + 1 mbyte) and join in the correct order.

 

This way, each client will keep handling its packet sizes and Ack's, the Router has enough bandwidth to handle this and more, and for the ESP32 side I have ready about how to send to 2 clients the same file, half one, half the other.

For the Delphi side I expect to receive each part in a memory buffer 'same time' and put together at the end.

 

The thing is; I'm now wondering how to work at the BinCliDemo to receive the two clients simultaneously. From the ICS library's, as it works asynchronously looks it is possible. But as the example is, it handles received data from one client only. or at least, that is what I think.

 

I looked around the code over and over and still do not have a good start point. Ideas come and go but when i looks I have found a way, further thinking invalidates the idea...  : )

 

I also look in the first example of the FileSend/FileReceive where the connected Client can be indexed, say as "Client[0]", "Client[1]", etc.

 

Probably I'm trying to do in the worst way?

 

Question is,

Could you bring some direction, general advice how/where to start, to handle two client connections data independently...?

 

PD: Forgot to say, the idea is to have the two clients on separated TCP ports. Say port 23 and port 24 so each one handle its own packet processing.

 

 

 

Edited May 27, 2019 by Bob4231
update

[Angus Robertson 520]

> Now I'm at the step to improve the speed for the data transfer. I have tried to increase the ESP32 buffer size

The ESP32 microcontroller is a small, very cheap device not designed for speed (it's used in mains smart plugs controlled by Alexa).  From an ICS perspective do not mess with MTU or TCP settings, just make sure you have a static receive buffer of 64K or something from which you write to a file in chunks of that size. Memory files are not always a good idea, since they may come from paged memory. 

 

> I'm now wondering how to work at the BinCliDemo to receive the two clients simultaneously.

Most ICS servers derive from TWSocketServer, where you put all your code into a client class and can handle hundreds of clients on the same port.  If you really want clients on different ports, generally you need one server per port, although TWSocketServer will also listen on multiple ports such as 80 and 443 for HTTP and HTTPS.  Look at the OverbyteIcsTcpSrvIPv6 sample which is the multi-client  binary file transfer version that will work with your client. 

 

Angus

[Bob4231 2]

Ok, thanks! 

I agree about forget about MTU modifications.

 

Will keep learning on TWSocketServer all this week.

Also to build the example at here to better understand the class and how to handle the events.