TIdHTTP SSL and error 'HTTP/1.1 403 Forbidden'

[Carlos Tré 9]

Dear All,

 

I'm not versed in HTTP a SSL, so excuse-me if I make any confusion.

A FMX project I'm working on demands GET, POST, FTP and email receiving and parsing capabilities.

 

I started with a simple GET do a HTTPS server, so I dropped a TIdHTTP and a TIdSSLIOHandlerSocketOpenSSL components in a bare form and configured as follows:

• IdHTTP1.Request.ContentType  := 'application/x-www-form-urlencoded';
• IdHTTP1.IOHandler := IdSSLIOHandler;
• IdSSLIOHandler1.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
• IdSSLIOHandler1.SSLOptions.SSLMode apparently has no impact

and then called IdHTTP1.Get(<https url>);

In the application executable folder there are 3 files: libeay32.dll, ssleay32.dll and the application executable.

The GET method raises an exception EIdHTTPProtocolException with the message 'HTTP/1.1 403 Forbidden', so I think I'm missing something very basic because the <https url> works in the browser, I just could't figure it out what that would be by going through both component's properties. Can you help me?

 

Also, a long time ago I needed to use Indy in a project, and there were paid support available. Is there nowadays? As I said before, this line of work is not my strong suit and I don't want to be the PITA I'll probably be - I think I'll have many more questions.

 

Best regards & TIA

Carlos

 

 

 

[Remy Lebeau 1421]

1 hour ago, Carlos Tré said:

The GET method raises an exception EIdHTTPProtocolException with the message 'HTTP/1.1 403 Forbidden', so I think I'm missing something very basic because the <https url> works in the browser,

The fact that you are even getting an HTTP 403 error at all means the SSL portion is working fine, since HTTPS is encrypted and you are able to see a decrypted response.  So the problem has to be related to something else.  Maybe you are not accessing the correct URL.  Maybe you are not authenticated properly with the server.  Maybe you are missing a client-side certificate.  We don't know, since we don't know your setup or what the server is expecting.

 

One thing you could try is capture the HTTP request the browser is sending (using your browser's built-in debugger, or an external HTTPS debug proxy like Fiddler), and then capture the HTTP request that TIdHTTP is sending (assign one of Indy's TIdLog... component to the TIdHTTP.Intercept property), and compare them for any differences.  If you don't see anything apparent at the HTTP layer (cookies, login credentials, etc), then the issue could be at the SSL layer (ie, certificates, etc)

Quote

I just could't figure it out what that would be by going through both component's properties. Can you help me?

You likely need to talk to the server admins about this, ask them what the server requires to get permission to access it.  Have them look at the HTTPS requests on their end to see why the server is rejecting them.

Quote

Also, a long time ago I needed to use Indy in a project, and there were paid support available. Is there nowadays?

You should contact AToZed to double-check, but I do not believe that paid support is still provided, at least for Indy.  Probably only for IntraWeb, if at all.

 

Edited May 13, 2020 by Remy Lebeau

[Guest]

 

11 hours ago, Carlos Tré said:

IdHTTP1.Request.ContentType  := 'application/x-www-form-urlencoded';

This should be a problem as you telling the server that your request has content type, then you use simple GET without payload.

1) check the working HTTP headers in the browser, either as Remy or google "browser developer tools network", 

2) ContentType in the request most likely to be used with POST ( or PUT ), but not GET, so check the method in the browser too.

 

edit additon:

May be the mistake is you used ContentType instead of Accept ?

Edited May 14, 2020 by Guest

[Remy Lebeau 1421]

9 hours ago, Kas Ob. said:

This should be a problem as you telling the server that your request has content type, then you use simple GET without payload.

It is not ideal, but it is not a problem, since the server will just ignore the 'Content-Type' request header in a GET request.

Quote

May be the mistake is you used ContentType instead of Accept ?

'application/x-www-webform-urlencoded' would not be a valid media type for an 'Accept' request header.

Edited May 14, 2020 by Remy Lebeau

[Carlos Tré 9]

Dear Remy,

 

Many thanks for your help, I'll try not to be PITA.

 

Quote

 

The fact that you are even getting an HTTP 403 error at all means the SSL portion is working fine, since HTTPS is encrypted and you are able to see a decrypted response. 

 

If SSL was not the problem I set out to find what could it be that I was missing. As I told before, I'm a newbie to internet programming, and was unaware that I should set a few more properties. I'm not completely there yet, but made some progress and now there is no authorization error. The response I get is encrypted and I don't know how to decrypt it. Is this another property I must  set in order to solve it? Or shall I do further processing?

 

As you probably must have noted by now, this is related to the problem I had with ICS, one that you kindly help me with. Having failed that POST attempt a few times I asked for and the webmaster set a lean and mean to get it working through a simple GET. And it did alright.

 

Discussing my lack of experience on this matter with a fellow programmer, she suggested the change to Indy, so here I am 🙂

 

This is the relevant part of the code that works

 

procedure TForm1.btnGetClick(Sender: TObject);
var
  LResp : string;
begin
  IdHTTP1.Request.Accept := 'text/html, image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*';
  IdHTTP1.Request.AcceptEncoding := 'gzip, deflate';
  IdHTTP1.Request.UserAgent := 'Mozilla/4.0';

  IdHTTP1.IOHandler := IdSSLIOHandler1;
  IdSSLIOHandler1.SSLOptions.SSLVersions := [sslvTLSv1_2];
  IdLogFile1.Open;
  try
    LResp := IdHTTP1.Get('https://www.trivial.com.br/envia_arq3.php?senha=violeta&nome=tre');
  finally
    IdLogFile1.Close;
  end;
  Memo1.Lines.Add(LResp);

 LResp should get 'tre' and I get '*)Jÿÿ=h' instead. Also, the log file created is empty, most certainly due to something I'm missing again. The only property set at run time is the file name in the component, and the IsHTTP1.Intercept is set to the logger.

 

Again, many thanks for all your help.

 

Edit:

 

The code began to work after setting the UserAgent property.

 

Edited May 14, 2020 by Carlos Tré
I forgot something

[Carlos Tré 9]

Dear Kas Ob,

 

Many thanks for help, very much appreciated.

 

Quote

This should be a problem as you telling the server that your request has content type, then you use simple GET without payload.

1) check the working HTTP headers in the browser, either as Remy or google "browser developer tools network", 

2) ContentType in the request most likely to be used with POST ( or PUT ), but not GET, so check the method in the browser too.

A precious lesson you taught me here, it will most certainly will be of help in the future as well. The browser debugger was instrumental here.

 

[Guest]

You are welcome.

 

It is always better to show one how to fish !

[Remy Lebeau 1421]

Quote

IdHTTP1.Request.AcceptEncoding := 'gzip, deflate';

The data you are seeing is not *encrypted*, it is *compressed*.  You are telling the server that you will accept responses in a compressed format (on top of encryption used by SSL/TLS), and the server is choosing to actually send a compressed response (you can verify that by looking at the TIdHTTP.Response.ContentEncoding property after the response arrives), however you have not setup TIdHTTP to actually decompress the compressed data for you, so you are seeing the compressed data as-is (well, after it has been String'ified, anyway).

 

Get rid of that assignment to the TIdHTTP.Request.AcceptEncoding property (unless you really want to try your hand at decompressing the data manually).  Instead, assign a TIdZLibCompressorBase-derived component, such as TIdCompressorZLib, to the TIdHTTP.Compressor property, and let TIdHTTP manage the AcceptEncoding property for you based on the compressor's capabilities.

Quote

Also, the log file created is empty, most certainly due to something I'm missing again.

Don't call the TIdLogFile's Open() and Close() methods directly.  Use its Active property instead, which will call Open()/Close() for you:

IdLogFile1.Active := True;
try
  LResp := IdHTTP1.Get('https://www.trivial.com.br/envia_arq3.php?senha=violeta&nome=tre');
finally
  IdLogFile1.Active := False;
end;

Calling Open() will open the log file you specify, but any sent/received data will not be logged to the file if Active is False.

Edited May 15, 2020 by Remy Lebeau

[Carlos Tré 9]

16 hours ago, Kas Ob. said:

You are welcome.

 

It is always better to show one how to fish !

Luckily I had the strength required do hold the rod. 🙂

[Carlos Tré 9]

11 hours ago, Remy Lebeau said:

Get rid of that assignment to the TIdHTTP.Request.AcceptEncoding property (unless you really want to try your hand at decompressing the data manually).  Instead, assign a TIdZLibCompressorBase-derived component, such as TIdCompressorZLib, to the TIdHTTP.Compressor property, and let TIdHTTP manage the AcceptEncoding property for you based on the compressor's capabilities.

Thank you very much, now it works like a charm. Next step is filling a form fields, I hope I can translate you English explanation into "Indysh" by myself. We''ll see. 🙂