TBinaryPacket - a wrapper class to read out sent binary data over network

[aehimself 388]

Hello all,

 

I've seen discussions about how to properly receive binary packets through the network. I even had my share of mistakes with my first implementation so I decided to create a class to do the heavylifting: receiving data chunks and assembling the packets. Feel free to use, improve and build upon.

 

Keep in mind that

1, Packet is referred to the data sent by the client, not TCP packets!

2, The code is NOT thread safe!

3, Memory allocation is inefficient and can be improved. In my case (really low amount of packets) it's not an issue but the internal data storage easily can be replaced with a stream for example

uBinaryPacket.pas

[Fr0sT.Brutal 897]

TMemoryStream? 🙂

[aehimself 388]

4 hours ago, Fr0sT.Brutal said:

TMemoryStream? 🙂

MemoryStream, BytesStream, you can even use a FileStream if you feel like it. For exotic coders even a 1024*768 bitmap would do.

 

All I meant is that it is a choke point for consideration. In my case this was more than sufficient.

[Lars Fosdal 1730]

A couple of observations:
1. TPacketLength = Word;
The first thing you write to the stream is the Packet length as a word.

PPacketLength(@inData[0])^ := datalen; - but datalen is an integer.

TBytes can be larger than 64K, so there are some safeguards missing.

 

2. If you are transmitting to a different byte order system, the PacketLength needs to be marshalled/demarshalled.

[aehimself 388]

2 minutes ago, Lars Fosdal said:

A couple of observations:
1. TPacketLength = Word;
The first thing you write to the stream is the Packet length as a word.

TBytes can be larger than 64K, so there are some safeguards missing.

 

2. If you are transmitting to a different byte order system, the PacketLength needs to be marshalled/demarshalled.

Appending the length is included in the class method TBinaryPacket.AppendSize(Var inData: TBytes);

As for the size limit I do have a check but it's in the sending method of my client application. You are absolutely right, it makes more sense here though!

  // If the buffer exceeds the maximum length allowed, raise an error as it can not be sent!
  If Length(buf) > TPacketLength.MaxValue Then Raise ETCPPortError.Create('Buffer overflow, cannot send ' + Length(buf).ToString + ' bytes!');

I personally have no experience with different byte order systems... You mean some systems (OSes..?) store a word like LLHH and an other like HHLL? I always thought that it's language dependent.

In this case yes, the code is not ready for it yet. It can be achieved by adding one more byte to the beginning... 0 if sender is a LLHH system, 1 if sender is a HHLL system. The receiving part then can decide if we need to switch the order upon reading a packet out.

 

Thank you for this input! Although I used this code to send and receive data from Windows to Windows it's always good to know that there are cases which I have to consider before using it in a future cross-platform project.

[Lars Fosdal 1730]

Well, come to think of it - it is rare to see big-endian CPUs these days so perhaps marshalling is overkill.

 

I would probably have had a fixed header block with

header size byte

header version byte

datasize word

data...

 

I guess you do compression in a layer outside of this one?

 

[aehimself 388]

Actually no, compression is not performed in the project in which this class was used.

Data packet is a JSON object which gets serialized and encrypted... size is usually between 80-140 bytes, which only increases after compression.

 

The reason I didn't include a version in the header is the packet itself: if a new version adds a new field this easily can be detected during deserialization.

 

I know the code is not completely universal. I kept it as simple as possible to do it's job in this context and nothing more - keeping the logic easy to read and understand. If your packets are usually larger than 64k, it's enough to change TPacketLength = UInt64; for example, the rest of the code should take care about this automatically.

 

P.s.: I just read your first answer again and realized that I'm indeed copying an Integer in a TPacketLength which is definitely not right! I'll fix this and move the size check from my program to this class so I can reupload. I should also mention the size limit in the "Keep in mind" section I assume 🙂

 

P.s.p.s.: There was a network loss between just the other day and about 90k of buffer was received by the server upon reestablishment containing a number of ~140 byte packets. The code worked flawlessly, separated and processed each of these without errors. So yeah, I'm happy 🙂

Edited June 28, 2021 by aehimself