Need help with IDhttp and Thread

[clubreseau 0]

Hi,

 

Here is what I would like to do.

a simple Form with a Tlistbox which would contain several URLs with a TButton and a TMemo.

 

To make it simple, I try as quickly as possible to go to all the URLs of my listbox and take the source page of each URL in my listbox and if in the page the word Welcome is there. then insert the url in the Tmemo

 

I have watched several tutorials that talk about Thread and Parallel Programming Library I can't seem to do this. I'm trying from this tutorial https://stackoverflow.com/questions/37123765/how-to-use-threads-with-idhttp-in-delphi-10 but it doesn't work.

 

Could someone give me a piece of code that will do the job.

 

Thank you and sorry for mu english

 

 

[Remy Lebeau 1397]

That StackOverflow post already showed working code.  If it is not working for you, then you are likely using it incorrectly, but we can't see the code you are using.  Please show YOUR actual code that you are having trouble with, and explain HOW it is not working for you exactly.

[clubreseau 0]

I use this code and it work

 

procedure TForm1.Button1Click(Sender: TObject);
var
  i:integer;
  lpath:string;

begin
  for i := 0 to ListBox1.Items.Count-1 do
  begin
    lPath := ListBox1.Items.Strings[i];
    download(lPath);
   end;
 end;

 procedure TForm1.Download(lpath: string);
 begin
    TTask.Create(
      procedure
      var
        HTTP : TidHTTP;
        IdSSL: TIdSSLIOHandlerSocketOpenSSL;
        content : String;
        URI: TIdURI;
        fmatch : TMatchCollection;
        regex2 : TRegEx;
        fmatchcount:string;
      begin
        HTTP := TIdHTTP.Create(nil);
        try
          HTTP.ReadTimeout := 30000;
          HTTP.HandleRedirects := True;
          IdSSL := TIdSSLIOHandlerSocketOpenSSL.Create(HTTP);
          IdSSL.SSLOptions.Method := sslvTLSv1_2;
          IdSSL.SSLOptions.Mode := sslmClient;
          HTTP.IOHandler := IdSSL;
          HTTP.AllowCookies:=true;
          HTTP.Request.UserAgent := 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36';
          HTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
          HTTP.Request.AcceptLanguage := 'en-GB,en;q=0.5';
          HTTP.Request.Connection := 'keep-alive';
          HTTP.Request.ContentType := 'application/x-www-form-urlencoded';
          HTTP.CookieManager := TIdCookieManager.Create(HTTP);
          URI := TIdURI.Create('http://localhost');
          try
            HTTP.CookieManager.AddServerCookie('habari=10283454', URI);
          finally
            URI.Free;
          end;
          Content := HTTP.Get(lpath);
        finally
          TThread.queue(TThread.Current,
          procedure
          begin

              regex2 := TRegEx.Create('Welcome', [roIgnoreCase]);

    fmatch := regex2.Matches(Content);
    fmatchcount:=IntToStr(fmatch.Count);


  if fmatchcount > '0' then

           listbox2.items.Add(lpath);
        
            //listbox2.Items.Add(lpath);
          end);
        end;

      end
    ).Start;
  end; 

Like this its slow how to past the 100 urls in my listbox faster ?

[Remy Lebeau 1397]

Did you try profiling your code to see where it is actually spending its time?  You are starting a new TTask thread for each ListBox item, but running 100 threads simultaneously will not be faster than processing 100 items in batches of, say, 4-8 threads at a time.  Creating more simultaneous threads does not mean the job will be completed faster.  If anything, doing so will slow it down, because the OS can only handle so much work simultaneously, the more threads you have running the more time the OS has to spend switching between them.  In general, you should not have more threads running than you have CPU cores.

 

Have you tried using TParallel.For() instead?  It uses a smaller pool of threads and will manage them according to its actual work load.  Or, at the very least, you can use the TTask constructor that allows your tasks to use threads from a TThreadPool that you create.

 

Also, don't pass TThread.Current to TThread.Queue().  That will link the queued operation to the thread, and if the thread terminates before the queued operation is performed, the operation will be canceled, and you won't see the thread's result in your UI.  Better to pass nil instead in this case.

 

Also, you are leaking the TIdHTTP object, and thus the TIdSSLIOHandlerSocketOpenSSL and TIdCookieManager objects.

 

Also, there is no need to invoke your TRegEx logic in the context of the main UI thread (I wouldn't even use TRegEx at all), it should be invoked in the context of the TTask thread instead.  Only the final ListBox addition (the only part of the thread code that actually touches the UI) should be queued, if it is to be performed at all.

Edited October 26, 2020 by Remy Lebeau

[clubreseau 0]

 TParallel.For() can you show me how ? for what I want !

 

thank you

 

I try this and its freeze my delphi

TParallel.&For(1,10,
  procedure(Index: Integer)
  begin

 TThread.Synchronize(TThread.CurrentThread,
        procedure
        begin
          Form1.Caption := 'Task Starting...';

        end);
  end);

 

I try this and work 

 

      TThread.Queue(TThread.CurrentThread,

        procedure
        begin
          Form1.Caption := 'Task Starting...';

        end);

   end

 

why Synchronize freeze and not Queue ?

 

then for what I want I need to use only Queue ?

 

ALso I try this

procedure TForm1.Button1Click(Sender: TObject);
  begin
  TParallel.&For(0, ListBox1.Items.Count-1,
    procedure(AIndex: Integer)
    var
      lPath: string;
      lHTTP: TIdHTTP;
      IdSSL: TIdSSLIOHandlerSocketOpenSSL;
      URI: TIdURI;
    begin
      TThread.Queue(nil,
        procedure
        begin
          Form1.Caption := 'Task Starting...';
          lPath := ListBox1.Items.Strings[AIndex];
        end);


      lHTTP := TIdHTTP.Create(nil);
      try

              lHTTP.ReadTimeout := 30000;
          lHTTP.HandleRedirects := True;
          IdSSL := TIdSSLIOHandlerSocketOpenSSL.Create(lHTTP);
          IdSSL.SSLOptions.SSLVersions:=  [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
          IdSSL.SSLOptions.Mode := sslmClient;
          lHTTP.IOHandler := IdSSL;
          lHTTP.AllowCookies:=true;
          lHTTP.Request.UserAgent := 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36';
          lHTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
          lHTTP.Request.AcceptLanguage := 'en-GB,en;q=0.5';
          lHTTP.Request.Connection := 'keep-alive';
          lHTTP.Request.ContentType := 'application/x-www-form-urlencoded';

          lHTTP.Get(lPath);
        except
          on E: EIdHTTPProtocolException do
          begin
            if E.ErrorCode = 404 then
            begin
              TThread.Synchronize(nil,
                procedure
                begin
                  Form1.ListBox2.Items.Add(lPath);
                end
              );
            end;
            Exit;
          end;
        end;

      TThread.Queue(nil,
        procedure
        begin
          Form1.Memo1.Lines.Add(lPath);
        end
      );
    end
  );
end;

this code give me error 1 or more error 0 exception(s);

if I click again I got this error: error 1 or more error 1 exception(s); #0 EldUnknownProtocol: UnknownProtocol

Edited October 26, 2020 by clubreseau

[Remy Lebeau 1397]

22 hours ago, clubreseau said:

 TParallel.For() can you show me how ? for what I want !

Try something like this:

procedure TForm1.Button1Click(Sender: TObject);
var
  i: integer;
  lPaths: array of string;
begin
  SetLength(lPaths, ListBox1.Items.Count);
  for i := 0 to ListBox1.Items.Count-1 do begin
    lPaths[i] := ListBox1.Items.Strings[i];
  end;

  TParallel.&For(0, ListBox1.Items.Count-1,
    procedure(AIndex: Integer)
    var
      lPath: string;
      lHTTP: TIdHTTP;
      IdSSL: TIdSSLIOHandlerSocketOpenSSL;
      URI: TIdURI;
    begin
      lPath := lPaths[AIndex];

      lHTTP := TIdHTTP.Create(nil);
      try
        lHTTP.ReadTimeout := 30000;
        lHTTP.HandleRedirects := True;
        IdSSL := TIdSSLIOHandlerSocketOpenSSL.Create(lHTTP);
        IdSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
        IdSSL.SSLOptions.Mode := sslmClient;
        lHTTP.IOHandler := IdSSL;
        lHTTP.AllowCookies := True;
        lHTTP.Request.UserAgent := 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36';
        lHTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
        lHTTP.Request.AcceptLanguage := 'en-GB,en;q=0.5';

        lHTTP.Get(lPath);
      except
        on E: EIdHTTPProtocolException do
        begin
          if E.ErrorCode = 404 then
          begin
            TThread.Queue(nil,
              procedure
              begin
                Form1.ListBox2.Items.Add(lPath);
              end
            );
          end;
          Exit;
        end;
      end;

      TThread.Queue(nil,
        procedure
        begin
          Form1.Memo1.Lines.Add(lPath);
        end
      );
    end
  );
end;

Alternatively:

function TForm1.Download(lPath: string): ITask;
begin
  Result := TTask.Run(
    procedure
    var
      lHTTP: TIdHTTP;
      IdSSL: TIdSSLIOHandlerSocketOpenSSL;
      URI: TIdURI;
    begin
      lHTTP := TIdHTTP.Create(nil);
      try
        lHTTP.ReadTimeout := 30000;
        lHTTP.HandleRedirects := True;
        IdSSL := TIdSSLIOHandlerSocketOpenSSL.Create(lHTTP);
        IdSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
        IdSSL.SSLOptions.Mode := sslmClient;
        lHTTP.IOHandler := IdSSL;
        lHTTP.AllowCookies := True;
        lHTTP.Request.UserAgent := 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36';
        lHTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
        lHTTP.Request.AcceptLanguage := 'en-GB,en;q=0.5';

        lHTTP.Get(lPath);
      except
        on E: EIdHTTPProtocolException do
        begin
          if E.ErrorCode = 404 then
          begin
            TThread.Queue(nil, // or Synchronize()
              procedure
              begin
                ListBox2.Items.Add(lPath);
              end
            );
          end;
          Exit;
        end;
      end;

      TThread.Queue(nil, // or Synchronize()
        procedure
        begin
          Memo1.Lines.Add(lPath);
        end
      );
    end
  );
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  i: integer;
  TaskList: TList<ITask>;
  T: ITask;
begin
  TaskList := TList<ITask>.Create;
  try
    for i := 0 to ListBox1.Items.Count-1 do begin
      T := Download(ListBox1.Items.Strings[i]);
      TaskList.Add(T);
    end;
    while not TTask.WaitForAll(TaskList.ToArray, 500) do begin
      CheckSynchronize();
      Update;
    end;
  finally
    TaskList.Free;
  end;
end;

Quote

I try this and its freeze my delphi

why Synchronize freeze and not Queue ?

TThread.Synchronize() is synchronous.  It sends a request to the main UI thread and waits for it to be processed before exiting back to the calling thread.  But the main UI thread is blocked waiting for the TParallel.For() loop to finish running, and so it cannot process TThread.Synchronize() requests in the meantime.  Deadlock occurs.  See Freeze when use TThread.Synchronize with TParallel or TTask.WaitForAll.

 

TThread.Queue() is asynchronous.  It posts a request to an internal queue and then exits immediately, allowing the calling code to continue running.  The main UI thread will process the queue at a later time, in this case after the TParallel.For() loop has finished.

Quote

ALso I try this

That won't work, because TThread.Queue() is asynchronous.  When you issue the request to read the ListBox string, it won't be retrieved in time for your TIdHTTP to actually use it.  What you could do instead is copy the ListBox strings to a local array before starting the TParallel.For() loop, and then you can access that array directly in your anonymous procedure without needing to synchronize access to it.  See the example above.

 

Otherwise, don't use TParallel.For() when you need to perform actions synchronously in the main UI thread.  See Using TThread.Synchronize with TTask.WaitForAll and the example above.

Quote

EldUnknownProtocol: UnknownProtocol

That is because your lPath variable has not been assigned a value yet when you pass it to TIdHTTP.Get().

Edited October 27, 2020 by Remy Lebeau

[clubreseau 0]

Wow Thank you Remy Lebeau for your time, Thank you for this documentation.

 

What more could you ask for a well documented example.

 

I'll read it to get it right, then try to build a loop with your explanations.

 

Thank you again for all this help.

Edited October 27, 2020 by clubreseau

[clubreseau 0]

Ok for me the second option is better for me. 

Only 1 problem, if my listbox contain 100 urls, going to 100 urls at the same time, this is make page time out.

How I can proced with 5 or 10 url at time ?

[Remy Lebeau 1397]

2 hours ago, clubreseau said:

Only 1 problem, if my listbox contain 100 urls, going to 100 urls at the same time, this is make page time out.

How I can proced with 5 or 10 url at time ?

Connect the TTask objects to a shared TThreadPool that limits how many threads can run at a time.  Or, use a shared Win32 semaphore to limit how many TTask threads can send HTTP requests at a time.

[clubreseau 0]

Ok TThreadPool hard to implement on the code.

[Remy Lebeau 1397]

4 hours ago, clubreseau said:

Ok TThreadPool hard to implement on the code.

Why is it hard?  TTask has constructors that take a TThreadPool as input.  Create a single TThreadPool, configure it as needed, such as MaxWorkerThreads, and then pass that pool to each TTask you create.

[clubreseau 0]

I try with

 

function TThreadPool.ShouldGrowPool: Boolean;
begin
  Result := {(FWorkerThreadCount < FMinLimitWorkerThreadCount) and }(FIdleWorkerThreadCount < FQueuedRequestCount) and
(FWorkerThreadCount < Self.FMaxLimitWorkerThreadCount);
end;

[Remy Lebeau 1397]

3 hours ago, clubreseau said:

I try with

And your point of showing that code is .... what, exactly?

Edited October 28, 2020 by Remy Lebeau

[Remy Lebeau 1397]

Did you post this same issue on StackOverflow?

https://stackoverflow.com/questions/64581935/

[clubreseau 0]

12 minutes ago, Remy Lebeau said:

Did you post this same issue on StackOverflow?

https://stackoverflow.com/questions/64581935/

Yes I use all platform to get help.

[Remy Lebeau 1397]

7 minutes ago, clubreseau said:

Yes I use all platform to get help.

It is generally considered bad practice to ask the same question across multiple forums at the same time. It is rude to the people who decide to help you. If you are getting help somewhere, STAY THERE.

 

I spent the last 3 days of my time helping you, just for you to now throw away everything I had given you and go somewhere else to start over. That is a slap in my face. Thanks for nothing.

[clubreseau 0]

13 minutes ago, Remy Lebeau said:

It is generally considered bad practice to ask the same question across multiple forums at the same time. It is rude to the people who decide to help you. If you are getting help somewhere, STAY THERE.

 

I spent the last 3 days of my time helping you, just for you to now throw away everything I had given you and go somewhere else to start over. That is a slap in my face. Thanks for nothing.

I have published elsewhere precisely to not ask you too much.

[clubreseau 0]

4 hours ago, clubreseau said:

I have published elsewhere precisely to not ask you too much.

the code freeze the windows sometime, when I have more then 10 url in listbox. that why need no more then 5 per time

Edited October 29, 2020 by clubreseau

[Kryvich 165]

Hi, @clubreseau

@Remy Lebeau gave you everything you needed to finish the job. But OK, in the attachment to this message is a ready test application based on Remy's code, with the addition of a thread pool. Tested on Delphi CE.

 

Downloader.zip

Edited October 29, 2020 by Kryvich

[clubreseau 0]

7 hours ago, Kryvich said:

Hi, @clubreseau

@Remy Lebeau gave you everything you needed to finish the job. But OK, in the attachment to this message is a ready test application based on Remy's code, with the addition of a thread pool. Tested on Delphi CE.

 

Downloader.zip

Your code is Perfect only 1 problem, the windows freeze when the app run, can we fix this ?

 

Someone respond me this... It freezes the main thread (where UI of your app is running) and thus you may feel it's frozen. Move your task into a worker thread

 

Edited October 29, 2020 by clubreseau

[Kryvich 165]

Yes, it's possible. And just in case an user needs to stop the download, I added the corresponding button. Now there are 3 states: dsIdle, dsExecuted and dsStopped.

Downloader.zip

[clubreseau 0]

37 minutes ago, Kryvich said:

Yes, it's possible. And just in case an user needs to stop the download, I added the corresponding button. Now there are 3 states: dsIdle, dsExecuted and dsStopped.

Downloader.zip

Thank You Kryvich everything are perfect.

The windows not feezing anymore, and the loop are going fast.

 

Thank you for your time !