TSslHttpRest - 429 (Too Many Requests) - resend request

[Anxich 0]

Hello,

 

I am writing an application that is using an third party API. Now, this API has limits of requests that can be made within a minute. So I would like to add a timeout and then for my application to retry the request again. I've wrote a piece of code like that:

 

unit cSslHttpRestASync;

interface

uses
  OverbyteIcsSslHttpRest, Classes, OverbyteIcsHttpProt;

const
  MaxRetryAmount = 10;

type
TSslHttpRestASync = class(TSslHttpRest)
  private
    procedure OnRequestDoneAsync(Sender    : TObject;
                            Request   : THttpRequest;
                            ErrorCode : Word);
  protected
    FDone : Boolean;
    FError : Boolean;
    FRetryCount : Integer;
  public
    property Done : Boolean read FDone;
    property Error : Boolean read FError;

    constructor Create(AOwner: TComponent); override;
end;

implementation

uses
  cUtils, SysUtils, DateUtils;

{ TSslHttpRestAsync }

constructor TSslHttpRestAsync.Create(AOwner: TComponent);
begin
  inherited Create(AOwner);

  OnRequestDone := OnRequestDoneAsync;
  FRetryCount := 0;
end;

procedure TSslHttpRestAsync.OnRequestDoneAsync(Sender: TObject; Request: THttpRequest; ErrorCode: Word);
var
  SslHttpRestASync : TSslHttpRestASync;
  I : Integer;
  RetryAfter : Integer;
begin
  SslHttpRestASync := Sender as TSslHttpRestASync;

  if ((SslHttpRestASync.StatusCode = 200) or (SslHttpRestASync.StatusCode = 201)) then begin
    FDone := True;
  end else begin
    if FRetryCount = MaxRetryAmount then begin
      FError := True;
    end;

    if SslHttpRestASync.StatusCode = 429 then begin
      SaveToLog('FRetryCount: ' + IntToStr(FRetryCount) + sLineBreak + 'URL: ' + SslHttpRestASync.URL + sLineBreak + SslHttpRestASync.RcvdHeader.Text);

      for I := 0 to SslHttpRestASync.RcvdHeader.Count - 1 do begin
        if Pos('Retry-After: ', SslHttpRestASync.RcvdHeader[I]) <> 0 then begin
          RetryAfter := StrToInt(Copy(SslHttpRestASync.RcvdHeader[I], Pos(' ', SslHttpRestASync.RcvdHeader[I]) + 1, Length(SslHttpRestASync.RcvdHeader[I])));
        end;
      end;

      Sleep(RetryAfter * 1000);
      GetASync;
    end else begin
      Inc(FRetryCount);
      Sleep(500);
      GetASync;
    end;
  end;
end;

end.

 

Now, the problem is, I am getting same header over and over in my logs. The date matches the date of first request - like the request isn't sent again and I am just being served same information.

 

What am I doing wrong here? I'm completely lost.

Edited May 12, 2020 by Anxich

[Angus Robertson 577]

Not looked closely at your code, only got as far as sleep in an event handler which is very bad design. You are also recursively starting a new request from the event, which calls the event again...

 

To use async functionality properly, you should use a timer and triggers.  ICS provides such functions in  OverbyteIcsUtils, ie IcsGetTrgSecs, IcsGetTrgMins, IcsTestTrgTick which are used in OverbyteIcsSslMultiWebServ1.pas as an example.  You set a trigger in the event to how every many seconds you want to wait, then test it in a timer triggering once a second or slower, then start the next request. 

 

Angus

 

[FPiette 385]

Maybe there is a caching device between your application and the server. This device always returns the same result. Just a guess.

[Anxich 0]

3 hours ago, Angus Robertson said:

Not looked closely at your code, only got as far as sleep in an event handler which is very bad design. You are also recursively starting a new request from the event, which calls the event again...

 

To use async functionality properly, you should use a timer and triggers.  ICS provides such functions in  OverbyteIcsUtils, ie IcsGetTrgSecs, IcsGetTrgMins, IcsTestTrgTick which are used in OverbyteIcsSslMultiWebServ1.pas as an example.  You set a trigger in the event to how every many seconds you want to wait, then test it in a timer triggering once a second or slower, then start the next request. 

 

Angus

 

Thanks, that worked fine 🙂

Can you explain to me why this way works, but mine didn't? Or let me know where to find answer to that? I'd really appreciate that. I want to understand what I've done.

[Angus Robertson 577]

As I said, you are recursively starting a new HTTP request from the within an event called by that request, you need to start the next request outside the event, either from a timer as I suggested, or by sending a message to a function that starts the next request.  Sleep stops messages being processed so defeats the concept of event driven programming, it should only generally only be used in threads that don't do anything else.

 

Angus

 

 

 

[Anxich 0]

5 hours ago, Angus Robertson said:

As I said, you are recursively starting a new HTTP request from the within an event called by that request, you need to start the next request outside the event, either from a timer as I suggested, or by sending a message to a function that starts the next request.  Sleep stops messages being processed so defeats the concept of event driven programming, it should only generally only be used in threads that don't do anything else.

 

Angus

 

 

 

Thank you, that shed some light on this for me.

 

I am having a problem with this approach though as well, probably I am missing something again. It seems that POST methods do not have the params I've given them in first place when I am resending the query.

procedure TSslHttpRestASync.OnTimer(Sender: TObject);
begin
  if IcsTestTrgTick64(FResendTrigger) then begin
    FResendTrigger := Trigger64Disabled;

    DoRequestASync(FRequestType);
  end;
end;

GET request seem to be working just fine, however I am adding parameters directly to URL with these requests. When executing POST requests, I am using code like this.

FSslHttpRest[I].RestRequest(httpPOST, FSslHttpRest[I].URL, True, FJson[I].AsJSon());

Any ideas? Thanks for your help so far 🙂

[Tntman 14]

If you are using third party api read the docs there.. They will maybe tell you how much requests are you allowed to send in some amount of time.. Also when you get response from server in 99% of cases they will return amount of requests that you have left to perform in some amount of seconds.. Best way is to use program called postman to simulate requests and see whats going on, it will give you detailed information about everything.. My answer is maybe not what you are looking for but i think that postman will help you to understand better what's actually happening instead of using delphi code to figure out.. After successful tests in postman you implement and write delphi code, that's how i am doing usually 

Edited May 13, 2020 by Tntman

[Anxich 0]

11 minutes ago, Tntman said:

If you are using third party api read the docs there.. They will maybe tell you how much requests are you allowed to send in some amount of time.. Also when you get response from server in 99% of cases they will return amount of requests that you have left to perform in some amount of seconds.. Best way is to use program called postman to simulate requests and see whats going on, it will give you detailed information about everything.. My answer is maybe not what you are looking for but i think that postman will help you to understand better what's actually happening instead of using delphi code to figure out.. After successful tests in postman you implement and write delphi code, that's how i am doing usually 

Hey, I've been using Postman all along, but the API I am working with is very badly done (government job). Official docs say 6000 requests per minute, in reality it's 60 requests per minute on search methods, 1000 on getters etc. So I am working a way here to make it work universally.

 

Retry-After is implemented in my code as well, just I didn't share it here. Also - said API - does not always send that in header...

Edited May 13, 2020 by Anxich

[Angus Robertson 577]

You should be using the onHttpRestProg event with DebugLeve=DebugBody so you can actually see any error messages and what is really happening.  But please don't post long logs here, no more than a few pertinent lines. 

 

Angus

 

[Anxich 0]

46 minutes ago, Angus Robertson said:

You should be using the onHttpRestProg event with DebugLeve=DebugBody so you can actually see any error messages and what is really happening.  But please don't post long logs here, no more than a few pertinent lines. 

 

Angus

 

Yes, the params are gone when reissuing a request the way I did it.

 

"Content-Length: 0"

[Anxich 0]

I've modified my timer like that after I looked through ICS source code.

 

procedure TSslHttpRestASync.OnTimer(Sender: TObject);
begin
  if IcsTestTrgTick64(FResendTrigger) then begin
    FResendTrigger := Trigger64Disabled;

    if FRequestType = httpPOST then
      FSendStream.Seek(0, soFromBeginning);

    DoRequestASync(FRequestType);
  end;
end;

It is now working all good, thank you all for help 🙂

[Angus Robertson 577]

You should be repeating RestRequest and not using DoRequestAsync so the request is properly initialised, missed that, other problems may occur attempting to make this a class. 

 

Angus