NetHTTPCLient, NetHTTPRequest and CURL

[sfrazor 3]

In the past I've used easy curl/libcurl in C to accomplish some tasks.  I am novice when it comes to this so forgive me if I construct my questions incorrectly.

 

I'm trying to understand how to accomplish the following in native Delphi using the TNetHTTPClient/NetHTTPRequest  component.  Rather than post a bunch of embarrassing code, I'll post some C Code best I can recall that I've worked through in the past and see if I can accomplish it in Delphi.  I don't mind reading but I didn't see anything that addressed this on a simple level I can understand.  

 

If I'm understanding, there is nothing CURL is doing here that a  HTTP Client can't accomplish.

 

In 'C' 

//first some curl basic setup....  then:

curl_mimepart *part;

 

curl_addpart(mime);

curl_mime_name(part, "data");

curl_mime_filename(part, "filename");  //there is no filename but curl is odd and needs it to send as if binary data if I remember correctly.

curl_mime_data(part, databuf, data_len);

curl_mime_type((part, "application/octet-stream"); // NetHTTPClient1.ContentType:= 'application/octet-stream' ??

curl_easy_setopt(curol, CURLOPT_MIMEPOST, mime); // NetHTTPRequest1.MethodString:= 'POST' ??

curl_easy_perform(curl);

 

What am I trying to accomplish?  Sending a simple binary data buffer as if it were a file.  

 

[Dmitry Arefiev 99]

If you are talking about Windows, and you want to use libcurl from TNetHttpClient, and you are using RAD Studio 11 Update 2, and you have System.Net.HttpClient.*.pas source code. Then with a simple modification of System.Net.HttpClient.Linux.pas, you can use it on Windows. Is it what you want ?

[Remy Lebeau 1263]

Using TNetHTTPRequest, you can do something like this:

type
  TReadOnlyMemoryBufferStream = class(TCustomMemoryStream)
  public
    constructor Create(APtr: Pointer; ASize: NativeInt);
    function Write(const Buffer; Count: Longint): Longint; override;
  end;

constructor TReadOnlyMemoryBufferStream.Create(APtr: Pointer; ASize: NativeInt);
begin
  inherited Create;
  SetPointer(APtr, ASize);
end;

function TReadOnlyMemoryBufferStream.Write(const Buffer; Count: Longint): Longint;
begin
  Result := 0;
end;

...

DataStream := TReadOnlyMemoryBufferStream.Create(databuf, data_len);
try
  PostData := TMultipartFormData.Create;
  try
    PostData.AddStream('data', DataStream, 'filename', 'application/octet-stream');
    NetHTTPRequest1.Post(url, PostData);
  finally
    PostData.Free;
  end;
finally
  DataStream.Free;
end;

Using Indy's TIdHTTP instead, you can do this:

DataStream := TIdReadOnlyMemoryBufferStream.Create(databuf, data_len);
try
  PostData := TIdMultiPartFormDataStream.Create;
  try
    PostData.AddFormField('data', 'application/octet-stream', '', DataStream, 'filename');
    IdHTTP1.Post(url, PostData);
  finally
    PostData.Free;
  end;
finally
  DataStream.Free;
end;

 

Edited October 17, 2022 by Remy Lebeau

[zed 10]

There are Delphi wrappers for libcurl (for example: mormot2), so you can easily port your existing C code to Delphi.

 

 

[sfrazor 3]

Sorry I started this and abandoned it.  This is a learning exercise for me and my employer made me do real work 🙂

 

So a couple of things.

 

zed:  I downloaded the mormot2 and  am going to take it for a test drive.  We'll see what happens there.

Dmitry: I haven't looked at System.Net.HttpClient.Linux.pas yet but I will.

Remy:  I gave your example code a try.  My test server doesn't like the Postdata I'm sending.  I didn't realize  how little I know about this until now.  

The server expects <4 byte len><GZIP binary data> like this:  https://10.1.1.1:8443/<4 byte len><GZIP binary data> as a POST.  At the moment I'd settle for the successful unzip of Postdata so I can see if I munged it some how.  But its not getting that far.

The 4 bytes get mangled.  It is supposed to be byte count of the unzipped data size for a validation check.  The GZIP is a mime context with JSON etc within it.  I look at the buffer in debug before the Post.  It looks good there.

 

It all works as intended with curl ez.  When I build the DATA with Delphi  I stuff the 4 byte uint value at the head of the data which is a delphi Datastream.  Verify its correct.  Then append the binary zipped data (mime) to that Datastream and send it.

 

The server spits out some odd value for the first 4 bytes so the GUNZIP fails the byte check.  Is the entire content of the  NetHTTPRequest1.Post(url, PostData) getting endcoded?  I don't want that.  So that's what I'm investigating now.

 

I'll throw together some sample code next.  It'll be easier to see what I'm doing wrong.

 

 

[Remy Lebeau 1263]

2 hours ago, sfrazor said:

Remy:  I gave your example code a try.  My test server doesn't like the Postdata I'm sending.  I didn't realize  how little I know about this until now.  

The server expects <4 byte len><GZIP binary data> like this:  https://10.1.1.1:8443/<4 byte len><GZIP binary data> as a POST.

It expects that in the URL itself?  That is extremely odd.  But even so, that doesn't change anything I said earlier.  The code I provided, for both TNetHTTPClient and TIdHTTP, sends a POST in 'multipart/form-data' format, same as curl's CURLOPT_MIMEPOST option does.  This is a standardized format.  So, what exactly is the server complaining about this when using the code I gave you?

2 hours ago, sfrazor said:

It all works as intended with curl ez.

Then please show the RAW request that curl is actually transmitting (which you can get by using the CURLOPT_VERBOSE and CURLOPT_DEBUGFUNCTION options), compared to the RAW requests that TNetHTTPClient and TIdHTTP are transmitting (not sure how to get that info with TNetHTTPClient, but for TIdHTTP you can assign any TIdLog... component to the TIdHTTP.Intercept property).

2 hours ago, sfrazor said:

The 4 bytes get mangled.

In what way, exactly?

2 hours ago, sfrazor said:

When I build the DATA with Delphi  I stuff the 4 byte uint value at the head of the data which is a delphi Datastream.  Verify its correct.  Then append the binary zipped data (mime) to that Datastream and send it.

Oh, so that data is not in the URL, but actually in the POST body? That makes more sense.  But even so, the code I gave you should be posting that data just fine.  Whatever bytes you put into the TStream will get transmitted to the server.

2 hours ago, sfrazor said:

The server spits out some odd value for the first 4 bytes so the GUNZIP fails the byte check.

Can you be more specific?

2 hours ago, sfrazor said:

Is the entire content of the  NetHTTPRequest1.Post(url, PostData) getting endcoded?

Of course, according to the MIME standards for the 'multipart/form-data' format.

2 hours ago, sfrazor said:

I don't want that.

Yes, you do, because that is what the original curl code is doing, too.

[sfrazor 3]

Remy,

Thanks for the details.   It helps.

 

This should go a bit further in clarifying my process.....

PostData: TMultipartFormData;
ms: TMemoryStream;
ResultString: TStringStream;
URL: string
CompressedWithBytes: string;;
CompressedLen: Cardinal; //unsigned  is required by the server

....

URL := 'https://10.1.1.1:8443/'

ms := TMemoryStream.create

ResultString := TStringStream.create;

PostDat := TMultipartFormData.create;

CompressedWithBytes := ZipAddJSON(JSONDATA); // Returns string with 4 bytes added to header (27 02 00 00 <GZIP-JSON data>)

CompressedLen := Length(CompressedWithBytes);

ms.write(CompressedWithBytes, CompressedLen);

Postdata.AddStream('data', ms, 'filename', 'application/octet-stream);

NetHttpRequest.Post(URL,PostData);

 

Server Side Python3

 

 

Receives HTTP(S) POST
...
filedata = request.files.get('data');

// verifies data keyword is there prints error if missing
// verifies application/octet-stream is there prints error if missing

data = file.stream.read()
file.close()

size = int.from_bytes(data[:4], 'little) <- this should be the 4 byte size of the uncompressed data
print(f"First four bytes: {size}")
First four bytes: 88434780 <--should be 27020000 which is the first 4 bytes added to the data before sending
data = zlib.decompress(data[4:], buffsize=int.from_bytes(data[:4], "little"))  <---------- fails                                 
                                       
excep zlib.error as e:
    raise HTTPError(400, f"Failed to decompress data: {e}") 

//Exception Prints:
//Error in app: Failed to decompress data: Error -3 while decompressing data: incorrect header check

 

I'll see if I can provide the curl raw data...  Its actually production code that send periodic updates to a server.  I also don't know how to do that with TNETHTTPCLient or TNETHTTPRequest 😞

 

Edited October 21, 2022 by sfrazor

[Remy Lebeau 1263]

3 hours ago, sfrazor said:

This should go a bit further in clarifying my process.....

Why are you using 'string' for binary data? Don't do that. Use TBytes/TArray<Byte> for that instead. And then you can wrap that with TBytesStream instead of TMemoryStream (avoiding an unnecessary copy)  Or, just pass your TMemoryStream into ZipAddJson() and let it Write() its bytes directly to the stream (again avoiding an unnecessary copy).

 

Either way, you are not Seek()'ing the stream back to Position 0 before calling AddStream().

[Fr0sT.Brutal 897]

Sniffers like Wireshark or Fiddler allow to see raw requests even with encrypted connection. They require installing their cert though. Otherwise you can run a HTTP=>HTTPS proxy and dump its traffic