IdThread and THttpCli

[APAPE 0]

I have build a thread (using TIdThread), which can be startet multiple times to get data from different places. Each thread processes  accesses one place / Url.

I'm using a thread for each place to get the data of each of them totally as quick as possible. So the longest time is just the one which needs the longest time to retrieve the data.

 

I set the property "MultiThreaded" to true in the component.

 

Calling the .Post method does not work. I always get a timeout.

 

If the same code is executed not as a self running thread, it works fine.

I did not find any help on the internet. 

Does anybody knows why this timeout appears and how it can be moved away (so the thread does it's job)?

 

I'm using Delphi 2007 and the main application also uses THttpCli components.

 

Please help. Thx.

[Angus Robertson 574]

TIdThread is not an ICS component, probably Indy, which may mean  you have mixed two libraries.  May or may not be relevant to your problem. 

 

The only benefit of using threads with ICS to download multiple URLs is where the host is unavailable and Windows waits up to 30 seconds attempting to connect, blocking other actions. Apart from that, ICS will do several hundreds downloads in parallel. 

 

The most important issue if you really need to use threads is to make sure the ICS message pump is called in a solid loop if using async methods. 

 

Angus

 

[Remy Lebeau 1394]

4 hours ago, Angus Robertson said:

TIdThread is not an ICS component, probably Indy, which may mean  you have mixed two libraries.  May or may not be relevant to your problem. 

I was thinking the same thing.  Yes, TIdThread is part of Indy.  So, why wouldn't you want to use Indy's TIdHTTP component if you are using Indy for other things?

[APAPE 0]

I'm using the TIdThread object for many years, because it can be started and stopped without needing to destroy and recreate it. It is very handy.

 

It does not make any difference, if I use a normal "TThread" or "TIdThread" for execution. The THttpCli object always throws an timeout exception and the requested URL is never been called.

 

I do not use Indy for other things. I just use the TIdThread object.

[Angus Robertson 574]

Sorry, no idea.  When used correctly, THttpCli just works.  You ned to add some logging to see what you are not sending for what responses you are receiving.

 

Angus

 

[APAPE 0]

I have debugged a little bit. In the unit "OverbyteIcsHttpProt" in class "THttpCli" in method "DoRequestSync", and in the while FState <> httpReady do loop the state (FState) is always "httpDnsLookupDone", but does not change any more.
I looked a little bit deeper into unit "OverbyteIcsWSocket" into procedure TCustomWSocket.Connect where the method "WSocket_Synchronized_Connect" is being called, and this method returns 10035.
 

I searched again in the internet and found the following site:  https://stackoverflow.com/questions/16488603/overbyteics-http-timeout-when-used-in-different-threads
 

And the  S O L U T I O N  is very simple:   DO NOT CREATE THE ICS OBJECTS AT THREAD CREATION TIME, create and destroy them every time you need them in the thread!
 

Hope, this helps to save others time in future

Edited May 5, 2020 by APAPE

[Angus Robertson 574]

From that extra information, I assume the first POST attempt succeeded, you then had a time out due to a bad connection and it died.,  This goes back to what I said yesterday, Windows keeps trying to open a TCP connection and you can not stop it early with a timeout, only by freeing the socket and starting again with a new socket.  

 

This is nothing to do with threads, just standard TCP connections.  And why we recommend using a array of several hundred dynamically created THttpCli components all using async methods running in parallel sharing the same events for testing lots of remote sites.  No threads necessary. 

 

Angus

 

[FPiette 383]

19 minutes ago, APAPE said:

And the  S O L U T I O N  is very simple:   DO NOT CREATE THE ICS OBJECTS AT THREAD CREATION TIME, create and destroy them every time you need them in the thread!

This is how Delphi and Windows works. Anything related to Windows messages and many other things need to be created/destroyed within a thread's context not the thread constructor. This is because a thread constructor run on the context of the calling thread, which obviously not the thread under construction. If you had given a look at ICS multihread samples, you'd known...

 

[Remy Lebeau 1394]

7 hours ago, APAPE said:

I searched again in the internet and found the following site:  https://stackoverflow.com/questions/16488603/overbyteics-http-timeout-when-used-in-different-threads

 

And the  S O L U T I O N  is very simple:   DO NOT CREATE THE ICS OBJECTS AT THREAD CREATION TIME, create and destroy them every time you need them in the thread!

I don't use ICS, but that would imply that THttpCli uses a non-blocking socket that relies on window messages to progress its state.  In which case, yes, the THttpCli would need to be created in the same thread context that the message window is created in, ie created in the thread's Execute method or Run event, not in its constructor.  Another argument in favor of using Indy's socket components, like TIdHTTP, is that they use only blocking sockets and no window messages, so they can be created and used across thread boundaries.

[FPiette 383]

1 hour ago, Remy Lebeau said:

THttpCli uses a non-blocking socket that relies on window messages to progress its state. 

That is correct. But this doesn't prevent ICS from being used from several threads.

ICS do not need multithreading to do simultaneous operations on hundreds of sockets. It works nicely on a single thread. You only need multithreading if you need thousands of ACTIVE communications. And you need thread to off-lone the communication thread from processing. For example, if you have a lengthy database request and use a synchronous (blocking component) then you need a worker thread for the blocking database request.

If you create a thread for each communication, you'll be quickly have poor performances. The best is to balance number of thread and number of connections. If well designed, you can develop an application using ICS having as much thread as the CPU core and each thread handling hundreds of connections.

Whatever component you use, it is sure that handling thousands of simultaneous active connections requires proper design and proper hardware.