Using TIdHTTP in a thread

[GG2023 0]

Hello,

 

I am using TIdHTTP with SSL IO to send/receive REST API requests.

 

At some point I get Read timed out and at this point the thread seems to be stuck in-limbo.

 

I tried to recreate the HTTP object, but seems like that does not help - how do I reset the component after Read timed out exception?

 

I am using simple Get and Post methods. I have keep-alive option set, now I am wondering if it would help to remove it...

 

Thanks a lot.

Gabi

Edited April 22, 2023 by GG2023

[Dalija Prasnikar 1341]

It is impossible to say what is going on without seeing your code.

[GG2023 0]

The thread itself looks like this

 

while not Terminated do begin

      //do something with TXXX object that affects X, Y, Z
      vTerminate := X - Y < Z;

      if vTerminate then begin
         PostLog(0, 'Thread is about to be terminated.');
         UpdateThread;
         self.Terminate;
         end
      else begin
         PostAlive;
         Sleep(FSystemConfig.ThreadSleep)
      end
   end //while not terminated

 

Now the TXXX creates and uses TIdHTTP like below

 

procedure TXXX.CreateHTTP;
begin
   FHTTP                    := TIdHTTP.Create(NIL);
   FHTTP.Request.Connection := 'keep-alive';
   FHTTP.IOHandler          := CreateSSLIOHandler;
   FHTTP.ConnectTimeout     := 10000;
   FHTTP.ReadTimeout        := 30000;
end;

// -----------

function TXXX.CreateSSLIOHandler: TIdSSLIOHandlerSocketOpenSSL;
begin
   result                        := TIdSSLIOHandlerSocketOpenSSL.Create;
   result.SSLOptions.Method      := sslvTLSv1_2;
   result.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
   result.SSLOptions.Mode        := sslmClient;
   result.SSLOptions.VerifyMode  := [];
   result.SSLOptions.VerifyDepth := 0;
   result.OnStatusInfoEx         := self.OnStatusInfoEx;
end;
 

procedure TXXX.OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
begin
   SSL_set_tlsext_host_name(AsslSocket, FHTTP.Request.Host)
end;

 

And the actual HTTP processing is the following

 

vJsonToPost := {....some json goes here}

   PostLog(0, vJsonToPost);

   vJsonToSend := TStringStream.Create(vJsonToPost);
   try
      FHTTP.Request.ContentType     := 'application/json';
      FHTTP.Request.ContentEncoding := 'utf-8';
      while TRUE do begin
         try
            vId := '';
            vResult := FHTTP.Post(vUrl, vJsonToSend);
            if vResult > '' then begin
               ///working with vResult
            end;
         except
            on E: Exception do begin
               PostLog(1, 'Exception in SubmitIOCBuyOrder: ' + E.Message);
               //FGotException := True;
               PostLog(100, FHTTP.Response.RawHeaders.Text);

               //if Read time out
               //if FHTTP.Response.RawHeaders.Text = '' then
               if E.Message = 'Read timed out.' then
                  RecreateHTTP;
            end
         end;

         if vClientOrderId > '' then
            if CheckIfOrderIsThere(vClientOrderId) then begin
               PostLog(0, 'New order is going to be posted with status: ' + upperCase(vStatus));
               PostOrder;
               break
            end;
      end //while TRUE
   finally
      vJsonToSend.Free
   end;

 

CheckIfOrderIsThere does FHTTP.Get, if that matters.

[Dalija Prasnikar 1341]

I am sorry, but snippets of code where it is not clear what happens at which point and how everything is created and called are also not enough to figure out what exactly is wrong with your code.

 

One of the things does sound suspicious: you say the thread is stuck in the limbo when timeout occurs and you have Sleep call which will cause thread to sleep if it is not terminated, Maybe there is a problem?

[Remy Lebeau 1263]

On 4/21/2023 at 10:49 PM, GG2023 said:

At some point I get Read timed out and at this point the thread seems to be stuck in-limbo.

Can you be more specific about what "stuck in-limbo" actually means?

On 4/21/2023 at 10:49 PM, GG2023 said:

I tried to recreate the HTTP object, but seems like that does not help

Recreating the TIdHTTP and TIdSSLIOHandlerSocketOpenSSL objects would work fine.  How does it "not help" exactly?

On 4/21/2023 at 10:49 PM, GG2023 said:

how do I reset the component after Read timed out exception?

If you don't want to recreate the objects, you should be able to just Disconnect() the TIdHTTP and Clear() its IOHandler.InputBuffer if it has any unread data in it before the timeout occurred.

On 4/21/2023 at 10:49 PM, GG2023 said:

I am using simple Get and Post methods. I have keep-alive option set, now I am wondering if it would help to remove it...

That won't make any difference.  Besides, TIdHTTP should already be disconnecting the socket if a read error occurs.  So really, you might only need to clear the InputBuffer.

On 4/23/2023 at 2:39 AM, GG2023 said:

   result.SSLOptions.Method      := sslvTLSv1_2;

   result.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];

You should not use the Method and SSLVersions properties together.  They are mutually exclusive, setting one updates the other.  In fact, don't even use the Method property at all.

On 4/23/2023 at 2:39 AM, GG2023 said:

procedure TXXX.OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT; const AType, AMsg: String);

begin
   SSL_set_tlsext_host_name(AsslSocket, FHTTP.Request.Host)
end;

You don't need to do that manually at all.  TIdSSLIOHandlerSocketOpenSSL calls SSL_set_tlsext_host_name() internally for you, and has done so since 2016.

On 4/23/2023 at 2:39 AM, GG2023 said:

   vJsonToSend := TStringStream.Create(vJsonToPost);

   try
      FHTTP.Request.ContentType     := 'application/json';
      FHTTP.Request.ContentEncoding := 'utf-8';

'utf-8' is not a valid value for the 'Content-Encoding' request header.  You need to use the 'charset' attribute of the 'Content-Type' header instead (ie, the Request.Charset property).  Also, you are not telling the TStringStream to use UTF-8, so it will use the OS default instead, which is ANSI on Windows and UTF-8 on Posix, so you will mismatch your JSON data if it ever contains non-ASCII characters.  Use this instead:

vJsonToSend := TStringStream.Create(vJsonToPost, TEncoding.UTF8);
...
FHTTP.Request.ContentType := 'application/json';
FHTTP.Request.Charset := 'utf-8';

On 4/23/2023 at 2:39 AM, GG2023 said:

               //if Read time out

               //if FHTTP.Response.RawHeaders.Text = '' then
               if E.Message = 'Read timed out.' then
                  RecreateHTTP;

The correct way to detect a read timeout is to check if the caught exception is an instance of the EIdReadTimeout class.  Don't rely on parsing the exception's Message:

if E is EIdReadTimeout then

For that matter, why would you want to recreate the objects only on a read timeout, and not on other errors?
 

[GG2023 0]

Dear Remy,

 

I appreciate your input, I have found the issue in my code.

 

Regarding OnStatusInfoEx this is pretty strange, I had to add this piece to several other RESP providers in order for the code to work. Most likely I am still missing something here.

 

Have a good weekend and thank you.

[Remy Lebeau 1263]

7 hours ago, GG2023 said:

Regarding OnStatusInfoEx this is pretty strange, I had to add this piece to several other RESP providers in order for the code to work. Most likely I am still missing something here.

I'm guessing that code is simply old and can now be removed.