Indy OnWork

[mikerabat 20]

Dear all!

 

Our Software supports new uploading large files (100MB + ) to a server. To get a nice feedback we wanted

to have some kind of progress telling the user how far we are already with the upload.

For the upload we use the TidHTTP component + an openssl io handler.

 

To get the progress working we used the OnWork event that is published by the component.

While we were using the component for testing in our internal network (which is quite fast) the

progress seemed ok but when we tested that on a slow network (around 2MBit) we saw that the first 99% were

very fast (20MB per second +) and then the last bytes "hang" until the full file was transmitted. Basically the

progress bar showed a progress of 99% for around 4minutes until the file was finally uploaded.

 

What am I missing here? How can I determine how much data is transmitted to the server?

 

 

The code involved is:
 

function CreateLibertyHTTP( method : TIdSSLVersion = sslvTLSv1_2; proxySet : ISuperObject = nil ) : TIdHTTP;
var sslIOHandler : TIdSSLIOHandlerSocketOpenSSL;
begin
     Result := TIdHTTP.Create(nil);
     sslIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(Result);
     sslIOHandler.SSLOptions.Method := method;
     
     Result.IOHandler := sslIOHandler;

     if (proxySet <> nil) and (proxySet.S['Server'] <> '') then
     begin
          Result.ProxyParams.ProxyServer := proxySet.S['Server'];
          if proxySet.I['Port'] <> 0 then
             Result.ProxyParams.ProxyPort := proxySet.I['Port'];
          if proxySet.s['Username'] <> '' then
          begin
               Result.ProxyParams.ProxyUsername := proxySet.s['Username'];
               Result.ProxyParams.ProxyPassword := proxySet.s['Pwd'];
          end;
          Result.ProxyParams.BasicAuthentication := proxySet.B['BasicAuthentication'];
     end;
end;

httpReq := CreateLibertyHTTPReq( some params here );
DL.LLog(dlInfo, 'Liberty upload');
                         libertyParams.Clear;
                         libertyParams.Add('SID=' + SessionId);
                         libertyParams.Add('blockearly=1');
                         libertyParams.Add('csrfToken=' + csrfToken);

                         // eg 'http://192.168.0.193:81/tom24x/uploadFile?SID=45784095&blockearly=1'
                         s := httpReq.Post(LibServer+'tom24x/uploadFile', libertyParams, IndyTextEncoding_UTF8);
                         if Pos('error', LowerCase(s)) > 0 then
                            raise Exception.Create('Error initializing upload');

                         uplFileResp := SO(s);

                         fUploadFileSize:=FileSizeByName(DawFileName);
                         //s:='?SID='+SessionId+'&blockearly=1&overwrite=1';  // overwrite recording in case it's already uploaded
                         Stream := TStringStream.Create('');
                         Params := TIdMultipartFormDataStream.Create;
                         httpReq.OnWork:=LibertyUploadProgress;
                         try
                            Params.AddFile('pdf', DawFileName, 'application/octet-stream');
                            // New in 2.9.3: The Uploadfile issues a token which can be used as session id (one time thing)
                            // so the param does no harm at all.
                            s := LibServer+'rec/UploadRecording.exe?SID=' + uplFileResp.S['session'] + '&blockearly=1&overwrite=1';
                            // e.g. 'http://192.168.0.193:81/rec/UploadRecording.exe?SID=45784095&blockearly=1&overwrite=1'
                            try
                               httpReq.Post(s, Params, Stream);
                            except
                                  on E: Exception do
                                     raise Exception.Create('Sending DAW file failed: ' + E.Message);
                            end;
                            DL.LLog(dlInfo, 'Liberty says: '+Stream.DataString);

 

procedure TfrmDBBrowser.LibertyUploadProgress(ASender: TObject; AWorkMode: TWorkMode;
  AWorkCount: Int64);
function BytesToStr( numBytes : Int64 ) : string;
begin
     if numBytes > 1000000 then
     begin
          Result := Format('%.2f MB', [numBytes/(1000*1000)]);
     end
     else if numBytes > 10000 then
     begin
          Result := Format('%d kB', [numBytes div 1000]);
     end
     else
         Result := intToStr( numBytes ) + ' B';
end;
begin
     if fSxProgress.Cancelled
     then
         Abort
     else
     begin
          // note: the upload file size may be a bit higher than the actual file size due to header overhead
          fSxProgress.lblTopLabel.Caption := fLibertyUplLabel + ' ' + BytesToStr( Min(fUploadFileSize, AWorkCount ) ) + '/' + BytesToStr( fUploadFileSize );
          fSxProgress.lblTopLabel.Update;
          fSxProgress.SetProgress(-1, MulDiv(AWorkCount, 100, fUploadFileSize));
     end;
end;

 

kind regards

   Mike

 

 

Edited October 28, 2021 by mikerabat
Added code to clarify

[mikerabat 20]

Strange enough on a second installation (aka our test site) the OnWork property works as expected... uploading around 1MByte per second

and not waiting at all at the end. It looks like there is some kind of proxy in between that first receives the full message - relays that to the

actual server - and waits until something comes back... I have though never heard of something like this and they couldn't clarify that either...

Is there actually some kind of hardware around that could reproduce such a behaviour?

 

[Remy Lebeau 1394]

6 hours ago, mikerabat said:

To get the progress working we used the OnWork event that is published by the component.

You are ignoring the OnWorkBegin and OnWorkEnd events.  In particular, the OnWorkBegin event will tell you the actual total number of bytes that are about to be sent.  You should be using that value to calculate your progress in the OnWork event, not limiting your progress to just the file size, which will obviously be smaller than the actual transmission.  You are correct that there is going to be some extra overhead due to the formatting of the HTTP request, and your progress calculation is not reflecting those extra bytes.

 

Also, you are ignoring the AWorkMode parameter of the events, which will tell you whether the event are being triggered when sending the request to the server or when reading the response from the server.  You don't want the latter confusing your progress calculations.

6 hours ago, mikerabat said:

How can I determine how much data is transmitted to the server?

Keep in mind that sockets are buffered by default.  When TIdHTTP is "sending" data to the server, it is really just putting the data into a kernel buffer.  The actual data transmission happens in the background as the kernel sees fit to do so, and there is no way to track the actual transmissions (without hooking into the network hardware, anyway).  So, what you are really reporting progress for is the rate in which you are able to queue data for transmission, not the actual transmission itself.  If the buffer fills up, because the other party hasn't acknowledged transmitted data yet (ie, you are "sending" faster than the peer is reading), then queuing of new data will block until buffer space clears up as older data gets acknowledged.

 

For most apps, this is adequate.  But, if needed, you can try increasing the size of the kernel buffer (Indy doesn't have a property for that, but you can use the TIdHTTP.Socket.Binding.SetSockOpt() method to assign a value to the socket's SO_SNDBUF option manually), or you can try disabling some of the kernel's buffering by setting the TIdHTTP.UseNagle property to false.

 

[Guest]

Your caption to this topic made me think "Indy Components" FLOSS went "startup". Funny.