TSslHttpRest and multipart/form-data

[ogalonzo 2]

Is there an example of how to send an image with TSslHttpRest with content as multipart/form-data? I'm a little confused reading closest example that I could find (with THttpCli BTW).

 

Thanks in advance. 

[Angus Robertson 574]

Uploading files is built into the TSslHttpRest component. 

 

The main OverbyteIcsHttpRestTst sample has a menu option Upload File, select MIME Multipart, specify the Upload File Name, POST or PUT, and any REST parameters needed, it just works. 

 

The OverbyteIcsSnippets sample has a much simpler demo where you click a single button 'HTTP POST Upload File' which runs a single function to upload a file to one of my servers. 

 

Angus

 

[ogalonzo 2]

Ok I'll try it. Thank you very much!

[ogalonzo 2]

Well, I tried and no luck. I mean I tried using just the example (OverbyteIcsHttpRestTst), and it didn't work, Surely is something that I'm doing wrong. Also I re-read my original question and I didn't a good job describing my problem: 

 

I'm trying to implement an API from a biometric (face, finger and card) control device. I'm trying to register a face, which requires to send a request to 

 

{{host}}/ISAPI/Intelligent/FDLib/FaceDataRecord?format=json

 

Content is multipart/form-data. Body has to specify two parts, one named "data" which contains a JSON, another one named "image" which contains full local path to the image to be sent for biometric processing. I tried using the REST parameters for this, couldn't make it work, always received status 400 (bad request). I'm in the dark here. 

 

Cloud you please point me in the right direction? whatever help you can give me is greatly appreciated.

 

Thanks in advance. 

 

 

[Angus Robertson 574]

The TSslHttpRest file upload feature is designed to replicate a web browser uploading a file from a web page using a Submit command.  

 

You can see how the MIME content is built in the TSslHttpRest.RestRequest function at line 2910 in OverbyteIcsSslHttpRest.pas, you'll need to build a similar post stream before calling the component with your own Json requirements. 

 

However, your description of needing a 'full local path to the image' seems strange, unless your REST request is to a server on the same PC, normally you'd expect to send JPG image data or something, not a file name,  

 

Angus

 

 

 

[ogalonzo 2]

Ok, tried again and failed miserably. I'll try to describe what I'm using in Postman, and if you can point me to what to use in ICS that would be more than enough: 

 

In Postman, Body, then form-data I see two rows:

 

1) Key=data, Value=JSON

2) Key=image, value=<filename> (you're right, is just the filename)

 

Do I define those two as parameters in TSslHttpRest?

 

Additionally (from what I understood from the example you mentioned earlier) I have to generate by hand the boundary part, is that correct?

 

Thanks in advance. 

[Angus Robertson 574]

I don't know anything about Postman.  If you are not sending a binary file, there is no point in using MIME.  It's just a simple REST request.

 

I need to see what HTTP data Postman creates, not what the GUI says, to make any sensible suggestions.

 

Angus

 

[Kas Ob. 121]

Sorry to interfere, but i think the OP question wasn't clear enough.

 

He is asking if ICS hoes support MIME with multipart/form-data , that protocol allow to send multiple data with different encoding/(content type) with a name for each, like you can send "1.gif", "text.txt" and "conf.xml" in same POST or same response, also the content encoding for each doesn't need to be Base64 encoded, it can be binary.

 

The protocol for web is here

https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

 

and some extra resource https://www.sobyte.net/post/2021-12/learn-about-http-multipart-form-data/

 

It is defined in https://www.ietf.org/rfc/rfc2045.txt

 

Very similar if not identical is used with emails and SOAP, it is gaining popularity each day, as many sites allows you to upload multiple image from one form post.

 

Hope that cleared the OP question as i have no idea if ICS support this.

[Angus Robertson 574]

Thanks, effectively this site seems to be requiring the same parameters as a web browser submit form, rather than the more normal content used by REST APIs. 

 

The ICS TRestParam class can already generate seven different content types, Json, UrlEnc, XML, CSV etc, so I need to add FormData as an eighth content type, I'll look at it next week.  

 

Effectively this will be a more generalised version of the file uploading code I already recommended, but without actually uploading a file which it seems was a red herring.

 

Angus

 

[ogalonzo 2]

I changed to TSslHttpCli, and coded this:

 

// Note: current process I'm implementing comprises two steps: first one is sending a request to take a photo on the biometric device, then the
// device answers with the photo embedded in the request. From here it starts the  following code, which tries to emulate more or less what I saw
// in your example: 

      if (d.PartCount > 0) then
        Begin
          f:=d.Part(0);

          { Found one part, check type }
          if Assigned(f) and (f.ContentType = 'image/jpeg') then
            begin
              s:=ExtractFilePath(Application.ExeName) + 'capture.jpg';
              f.SaveToFile(s);

              strBdry := '-----------------------------' + IntToHex(Random(MaxInt), 8) + IntToHex(Random(MaxInt), 8);

              SslHttpCli1.ContentTypePost:= 'multipart/form-data; boundary=' + strBdry;
              SslHttpCli1.Connection := 'Keep-Alive';
              
              MimeHeader:=strBdry + sLineBreak +
                        'Content-Disposition: form-data; name="data"' + sLinebreak +
                        '{' + slineBreak +
                        '    "faceLibType": "blackFD",' + slineBreak +
                        '    "FDID": "1",' + slineBreak +
                        '    "FPID": "2000",' + slineBreak +
                        '    "featurePointType":"face"' + slineBreak +
                        '}' + slineBreak +
                        strBdry + sLineBreak +
                        'Content-Disposition: form-data; name="image" filename="capture.jpg"' + sLineBreak;

              MimeFooter:=strBdry + '--' + sLineBreak;

              SslHttpCli1.SendStream:=TMultiPartFileReader.Create (s, MimeHeader, MimeFooter);
              SslHttpCli1.SendStream.Position:=0;

              SslHttpCli1.URL:='https://' + STR_HOST + '/ISAPI/Intelligent/FDLib/FaceDataRecord?format=json';

              SslHttpCli1.Post();
            end;
        End;

      d.Free();
    end;

However I get an AV on SslHttpCli1.Post();. Any idea what I'm doing wrong?

[FPiette 382]

10 hours ago, ogalonzo said:

However I get an AV on SslHttpCli1.Post();. Any idea what I'm doing wrong?

You should mention where EXACTLY you get the AV. The debugger should tells you. Reproduce here the source lines around the AV.

[Kas Ob. 121]

              SslHttpCli1.ContentTypePost:= 'multipart/form-data; boundary=' + strBdry;
              SslHttpCli1.Connection := 'Keep-Alive';
              
              MimeHeader:=strBdry + sLineBreak +
............
                        strBdry + sLineBreak +
                        'Content-Disposition: form-data; name="image" filename="capture.jpg"' + sLineBreak;

              MimeFooter:=strBdry + '--' + sLineBreak;

              SslHttpCli1.SendStream:=TMultiPartFileReader.Create (s, MimeHeader, MimeFooter);

The "boundary=" value should be less two dashes from the boundary in header and footer.

 

The first answer on this SO question is just perfect

https://stackoverflow.com/questions/4526273/what-does-enctype-multipart-form-data-mean

[Angus Robertson 574]

The TMultiPartFileReader class you have used opens and converts file content to MIME base64, for a file upload. 

 

Earlier, you said you didn't want to upload the file.   But I'm still waiting for you to show an actual example of the data you are trying to POST.

 

Angus

 

[Angus Robertson 574]

Assuming you don't want to upload a file, you should change the code you showed to remove TMultiPartFileReader, just build a single MIME block from your header and footer, change capture.jpg to your full file name in s. and use the string as raw parameters in the REST call, which will automatically create the SendStream for the request. 

 

RestRequest(httpPost, URL, False, MyMimeHeaders);

 

Angus

 

[ogalonzo 2]

Finally managed to solve it. I was wrong about the image, I had to send it but in binary form. Final code is this, unnecesary parts removed:

 

{msCont = TMemoryStream
bwCont = TBinaryWriter
fs = TFileStream}

              s:='capture.jpg';

              strMIMEBoundary:=IntToHex(Random(MaxInt), 8);

              { Note: my device requires digest authentication, change or delete next three lines if yours does not }
              HttpCli1.ServerAuth:=httpAuthDigest;
              HttpCli1.Username:='xxxx';
              HttpCli1.Password:='xxxx';
              
              HttpCli1.ExtraHeaders.Add('Connection=keep-alive');
              HttpCli1.ContentTypePost:= 'multipart/form-data; boundary=' + strMIMEBoundary;

              strMIMEBoundary:='--' + strMIMEBoundary;

              slBuf.LoadFromFile(ExtractFilePath(Application.ExeName) + 'data.json', TEncoding.UTF8);

              msCont:=TMemoryStream.Create();
              bwCont:=TBinaryWriter.Create(msCont);

              bwCont.Write(TEncoding.UTF8.GetBytes(strMIMEBoundary + sLineBreak + 'Content-Disposition: form-data; name="data"' + sLinebreak + sLinebreak));
              bwCont.Write(TEncoding.UTF8.GetBytes(slBuf.Text));
              bwCont.Write(TEncoding.UTF8.GetBytes(strMIMEBoundary + sLineBreak));
              bwCont.Write(TEncoding.UTF8.GetBytes('Content-Disposition: form-data; name="image"; filename="capture.jpg"' + sLineBreak));
              bwCont.Write(TEncoding.UTF8.GetBytes('Content-Type: image/jpeg' + sLineBreak + sLinebreak));
              
              { Create stream from image }
              fs:=TFileStream.Create(s, fmOpenRead);
              fs.Position:=0;

              { Dump entire image to content }
              msCont.CopyFrom(fs, fs.Size);

              fs.Free();

              { Write MIME end marker }
              bwCont.Write(TEncoding.UTF8.GetBytes(sLineBreak + strMIMEBoundary + '--' + sLineBreak));

              bwCont.Free();

              ss:=TStringStream.Create();

              msCont.Position:=0;

              HttpCli1.SendStream:=msCont;
              HttpCli1.SendStream.Position:=0;
              HttpCli1.RcvdStream:=ss;

              ss.Position:=0;

              HttpCli1.URL:='http://' + STR_HOST + '/ISAPI/Intelligent/FDLib/FaceDataRecord?format=json';

              HttpCli1.Post();

              ss.Free()
            end;
        End

Thanks to everyone for your time. 

[FPiette 382]

14 hours ago, ogalonzo said:

Finally managed to solve it. I was wrong about the image, I had to send it but in binary form. Final code is this, unnecesary parts removed:

Thanks for your feedback. It's important for the community to have the final resolution.

[Angus Robertson 574]

The ICS TSslHttpRest component now has proper support for multipart/form-data parameters, using TRestParams, available in SVN and the overnight zip.  There is a new content type for FormData

and a new method AddItemFile that allows one or more files to be added as parameters. 

 

The  OverbyteIcsSnippets sample has two new HTTP file uploads buttons, one doing a simple file upload, the second a form-data upload, which set-ups the parameters as follows:

 

MyJsonParams := TRestParams.Create(self);
MyJsonParams.PContent := PContJson;
MyJsonParams.AddItem('FileTitle', mytitle);
MyJsonParams.AddItem('FileName', myfile);
SslHttpRest.RestParams.PContent := PContFormData;
SslHttpRest.RestParams.AddItem('FileTitle', mytitle);
SslHttpRest.RestParams.AddItemA('JsonBlock', MyJsonParams.GetParameters, true);
SslHttpRest.RestParams.AddItemFile('FileName', mysrcfile, 0);  
SslHttpRest.RestParams.AddItem('Submit', 'SubmitFile');
SslHttpRest.HttpUploadStrat := HttpUploadNone;  
StatCode := SslHttpRest.RestRequest(httpPOST, myurl, False, '');  

 

This part was relatively straight forward, but testing proved interesting, particularly with an 8Gbyte file.  The ICS application web server read uploaded data in a memory buffer, so that was changed for file stream above a certain size, likewise TRestParams needed a file stream for size.  Then our form handling code needed updating for character sets as recommended by the latest RFC with a new parameter display web page in the ICS web server samples for testing.  

 

So a long process, but much improved REST functionality.

 

Angus