httpclient POSTing binary data with TIdHTTP

[Mocte Sandoval 0]

Hi to all

 

I don't ask here often but now I need some help with Indy ( which I don't use often either btw ).

 

The scenario: 

 

I need to upload "documents" ( mostly PDF, Tiff, Jpg ) usually from a file to a third party HTTP server, I am doing that perfectly fine with code like this ( just a snippet ) :

var
  Params: TIdMultipartFormDataStream;
Begin
  ...
    Params.AddFile(fileName, fullFilePath, 'application/octet-stream');
    http.Post('http://x.x.x.x:8996/service', Params, responseStream);
  ...
End

The endpoint from the server side receives the file and answers with code 201 which is expected.

 

Now the problem, there are a lot of these "documents" stored on files on disk, each of these files contain a bunch of "documents" on a propietary format, which I'm succesfully extracting as an array of bytes, I can save them as individual files with TFileStream and view them with external tools, but I need to send these individual files to the http server, so for speeding up the process I want to avoid saving them to disk, so what i would like is to send the array of bytes straight to the server, here is an snippet from the code I am using so far:

procedure SendFromBytes( const buffer: array of byte );
var
  ...
  data, 
  idBoundary: String;
  ...
Begin
  ...
     idBoundary := '---someboundaryId';

      StreamResp := TStringStream.Create('',TEncoding.UTF8);
      streamSrc  := TMemoryStream.create;

      http.Request.ContentType := sContentTypeFormData + idBoundary;

      data := idBoundary + CRLF +
        'Content-Disposition: form-data; name="'+name+'"' + CRLF + CRLF;
      streamSrc.Write(data[1], Length(data));
      streamSrc.WriteBuffer(Buffer[0], length(Buffer));

      data := CRLF +  idBoundary;
      streamSrc.Write(data[1], Length(data));
      http.Post('http://x.x.x.x:8996/service', StreamSrc, streamResp);
...
End;

But here I'm getting 400 Bad request, so I'm not sending the message as is expected by the server 

 

Is there an Indy guru here who can help me find the cause? 

 

Edit: I'm on Delphi XE3

Edit 2: spelling on subject

Edited July 9, 2019 by Mocte Sandoval

[Attila Kovacs 631]

What happens if you set the stream's position to zero before posting?

[Mocte Sandoval 0]

1 hour ago, Attila Kovacs said:

What happens if you set the stream's position to zero before posting?

Just tested with the same result (400),  I didn't show it here but I have a TIdLogFile for logging the message, the data seems to be all in place, I suspect I am constructing the message in some way different from how TIdMultipartFormDataStream is processed by TidHttp and therefore it is possible I'm not sticking to the http spec for a multipart/form-data post but I'm failing to see where.
 

[Attila Kovacs 631]

Hm. Maybe the missing closing '--' suffix on the last boundary.

Edited July 9, 2019 by Attila Kovacs

[Mocte Sandoval 0]

Thank you Attila, I stripped that in my search of the correct way, added again with no luck.

 

I'm attaching a sample of what the log looks like, I deleted a great part of its contents because the documents are confidential data but maybe some expert eyes can notice something I'm missing.

 

BTW I think I missed the last line on the log  {"error":"multipart: NextPart: EOF"}  definitely there is something missing there.

test.txt

Edited July 9, 2019 by Mocte Sandoval
spelling

[Attila Kovacs 631]

umm, if  "boundary=---2-1d1a304dc9" in the header, shouldn't be the boundary -----2-1d1a304dc9 in the body? And closing with -----2-1d1a304dc9--?

Or just omit a -- in the header.

Edited July 9, 2019 by Attila Kovacs

[Mocte Sandoval 0]

Ok, I got rid of all initial "-"  so boundary is now e.g. boundary=4024b236a3a1 on header,  --4024b236a3a1 in body, and closing --4024b236a3a1--

 

And all is working great now ! 

 

Thank you so much Attila this was driving me nuts. 

 

Edit: How can I mark this topic as resolved?

Edited July 9, 2019 by Mocte Sandoval
Mark as solved

[Attila Kovacs 631]

Glad to hear that.

 

3 minutes ago, Mocte Sandoval said:

How can I mark this topic as resolved?

You can't.

And you will get it resolved the second time tomorrow by others 😉

It's ok.

[Remy Lebeau 1436]

6 hours ago, Mocte Sandoval said:

Now the problem, there are a lot of these "documents" stored on files on disk, each of these files contain a bunch of "documents" on a propietary format, which I'm succesfully extracting as an array of bytes, I can save them as individual files with TFileStream and view them with external tools, but I need to send these individual files to the http server, so for speeding up the process I want to avoid saving them to disk, so what i would like is to send the array of bytes straight to the server, here is an snippet from the code I am using so far

Why not continue using TIdMultipartFormDataStream for that?  It has an overloaded AddFormField() method that lets you specify a TStream for the input data.  You could store your bytes in a TMemoryStream or TBytesStream, or use Indy's own TIdMemoryBufferStream, etc.  For example:

procedure SendFromBytes( const buffer: array of byte );
var
  Strm: TIdMemoryBufferStream;
  Params: TIdMultipartFormDataStream;
begin
  ...
  Strm := TIdMemoryBufferStream.Create(PByte(buffer), Length(buffer));
  try
    Params := TIdMultipartFormDataStream.Create;
    try
      Params.AddFormField('name', 'application/octet-stream', '', Strm, 'filename');
      http.Post('http://x.x.x.x:8996/service', Params, responseStream);
    finally
    end;
  finally
    Strm.Free;
  end;
  ...
end;

36 minutes ago, Mocte Sandoval said:

Ok, I got rid of all initial "-"  so boundary is now e.g. boundary=4024b236a3a1 on header,  --4024b236a3a1 in body, and closing --4024b236a3a1--

Yes, whatever value is specified in the "boundary" header, in the body it needs to have a leading "--" in front of each new MIME part, and a trailing "--" at the end of the last MIME part.  TIdMultipartFormDataStream handles that internally for you.

Edited July 10, 2019 by Remy Lebeau

[Mocte Sandoval 0]

11 minutes ago, Remy Lebeau said:

Why not continue using TIdMultipartFormDataStream for that?  It has an overloaded AddFormField() method that lets you specify a TStream for the input data.  You could store your bytes in a TMemoryStream or TBytesStream, or use Indy's own TIdMemoryBufferStream, etc.

Yes, whatever value you specify in the "boundary" header, you need to add a leading "--" to in the body, and a trailing "--" at the end.

 

Good to know, I'll give it a try tomorrow, thank you Remy