TNetHTTPRequest Logon to Download

[egnew 5]

Hello:

'

I have been using Indy to download PDF files containing images of data collected by two of our third party providers.  It works fine with Indy but i am having a problem getting past the logon page when using TNetHttpClient and TNetHttpRequest.  The logon page loads corrected with a GET and the HTML is correct.  I then do a PUT with the username, password, and another required parameter.  But when I try to download a file, the download is the content of the logon page, not the file I requested.

 

What do I need to do to download from a website requiring a username and password with specific form requirements for obtaining a session?

 

Thanks

[Remy Lebeau 1421]

16 hours ago, egnew said:

I then do a PUT with the username, password, and another required parameter.

Why PUT and not POST?  Browsers don't use PUT for HTML-based webforms.

16 hours ago, egnew said:

But when I try to download a file, the download is the content of the logon page, not the file I requested.

Have you tried sniffing the HTTP traffic to see if there are significant differences between TIdHTTP's requests and TNetHTTPClient's requests?

16 hours ago, egnew said:

What do I need to do to download from a website requiring a username and password with specific form requirements for obtaining a session?

Does the logon use cookies?  Do you have cookies enabled in the TNetHTTPClient?

Edited September 6 by Remy Lebeau

[egnew 5]

I accidentally typed PUT as I am actually using POST.  I have not tried a sniffer yet as I have not installed it on my new computer.  I do have the information from accessing the site using Chrome's debugger and everything I am doing looks good.,  I will install the debugger if I can't get it working.   Yes I have AllowCookies set to true.

 

I have a unit which defines type TisNetHttp to handle all the details.  Pertinent sections of code are shown below.

 

In my main application I set a StringList named v_Params with the required values for the form I accessed using a GET.  I then call the routines and check the downloaded file. 

myHttp.GetText(v_LogonUrl);            // Fetches logon page
myHttp.Post(v_LogonUrl,v_Params);      // Send the form data
myHttp.GetFile(v_PDFUrl,v_FileName);   // Save the PDF

// TYPE
  TisNetHttp = Class
  private
    { Private declarations }
      f_NetHTTPClient: TNetHTTPClient;
      f_NetHTTPRequest: TNetHTTPRequest;
...

// CONSTRUCTOR
  f_NetHttpClient := TNetHttpClient.Create(nil);
  f_NetHttpClient.AllowCookies := True;
  f_NetHttpClient.HandleRedirects := True;
  f_NetHttpClient.OnAuthEvent := NetHttpClientAuthEvent;
  f_NetHttpRequest := TNetHttpRequest.Create(nil);
  f_NetHttpRequest.Client := f_NetHttpClient;
  f_NetHttpRequest.OnRequestCompleted := OnRequestCompleted;
  f_NetHttpRequest.OnRequestError := OnRequestError;

procedure TisNetHttp.GetFile (const p_Url, p_FileName: String);
// Similar to GetText below except content is saved to file
...

function TisNetHttp.GetText (const p_Url: String): String;
var
  v_Response: IHTTPResponse;
begin
  f_Error := ''; // Used by OnRequestError
  try
    v_Response := f_NetHTTPRequest.Get(p_Url);
    f_StatusCode := v_Response.StatusCode;
    f_StatusText := v_Response.StatusText;
    Result := v_Response.ContentAsString;
  except
    on E: Exception do
      begin
        Result := '';
        f_StatusCode := -1;
        f_StatusText := E.Message;
      end;
  end;
end;

procedure TisNetHttp.Post (const p_Url: String;
                             var p_Strings: TStrings);
var
  v_Response: IHTTPResponse;
begin
  f_Error := ''; // Used by OnRequestError
  try
    v_Response := f_NetHTTPRequest.Post(p_Url,p_Strings);
    f_StatusCode := v_Response.StatusCode;
    f_StatusText := v_Response.StatusText;
  except
    on E: Exception do
      begin
        f_StatusCode := -1;
        f_StatusText := E.Message;
      end;
  end;
end;

 

[Remy Lebeau 1421]

1 hour ago, egnew said:

I have not tried a sniffer yet as I have not installed it on my new computer.

You don't need to use an external sniffer with TIdHTTP, as it can log its own HTTP traffic directly. Simply assign a TIdConnectionIntercept-derived component to the TIdHTTP.Intercept property, such as one of the TIdLog... components.

 

I don't know if TNetHTTPClient has any logging capabilities, though (probably not).

Edited September 6 by Remy Lebeau

[egnew 5]

I have a working Indy version but I want to move everything over to the native components.  It is no problem for me to install a sniffer.

 

Am I using the correct steps in my procedures?

 

Thanks, Sidney

[egnew 5]

Also, I tried setting CustHeaders in the NetHttpClient.  I can also try that in the NetHTTPRequest but it didn't make a difference in the client.

  with f_NetHttpClient.CustHeaders do
  begin
    Clear;
    Add('Accept','*/*');
    Add('AcceptEncoding','gzip,deflate,br,zstd');
    Add('AcceptLanguage','en-US,en;q=0.9');
    Add('ContentType','application/x-www-form-urlencoded; charset=UTF-8');
    add('UserAgent','Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537..36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36');
  end;
 

  with f_NetHttpClient.CustHeaders do
  begin
    Clear;
    Add('Accept','*/*');
    Add('AcceptEncoding','gzip,deflate,br,zstd');
    Add('AcceptLanguage','en-US,en;q=0.9');
    Add('ContentType','application/x-www-form-urlencoded; charset=UTF-8');
    add('UserAgent','Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537..36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36');
  end;

The documentation only listed a few parameters as available for custom headers but there were many more headers in the chrome debugger.  Do limitations exist on what can be placed in the custom headers?

 

Also, the documentation mentions setting MethodString in the request and using the Execute method.  That didn't make sense to me when I needed to provide parameters to the POST.  Do you think that could be part of the problem?

 

[egnew 5]

I solved the problem by using Fiddler and found I was missing some parameters needed for the post.  I added code to scan the HTML to make sure all required parameters were added to the post.

 

Thanks for your suggestions.