Shutting down TidTCPServer (kbmMWTCPIPIndyServerTransport)causing Window Service timeout

[c0d3r 17]

During last whole week, I tried to find what could be possible to cause my Windows service timeout while calling TidTCPServer.Active := False,  I'm using Delphi 10.4.1,  codes were working fine in Delphi 2007 with Indy 9+ until we migrated to 10.4.1:

 

procedure TMyService.OnStop(...)

begin

    TransportServer.Close ;  <-- Error 1053 occurs.

end;

 

TransportServer is a type of TkbmMWTCPIPIndyServerTransport, the Close method internally was just calling TidTCPServer.Active := False;

 

I can't figure it out because if i don't stop the service,  it would serves the client requests/responses without any issue for weeks until at the point the users want to stop the service.

Edited February 9, 2021 by c0d3r

[Remy Lebeau 1421]

7 hours ago, c0d3r said:

During last whole week, I tried to find what could be possible to cause my Windows service timeout while calling TidTCPServer.Active := False,

That usually happens if you cause a thread deadlock inside your TIdTCPServer event handlers.  For instance, by syncing with the same thread that is trying to deactivate the server.  The Active setter closes all open client sockets, AND waits for the owning threads to terminate.  The server events are fired in the context of those threads.  So, either don't perform thread-blocking syncs during server shutdown, or else do the server deactivatation in a separate thread so the syncs can be processed normally.

7 hours ago, c0d3r said:

I'm using Delphi 10.4.1,  codes were working fine in Delphi 2007 with Indy 9+ until we migrated to 10.4.1:

TIdTCPServer shutdown hasn't really changed between Indy 9 and 10.  Sure, the underlying architecture is a bit different, but the basics are still the same.  Although, you might have been getting saved by the TIdTCPServer.TerminateWaitTime property in Indy 9, which defaulted to 5 seconds.  If the server shutdown exceeded that timeout, an EIdTerminateThreadTimeout exception would be raised, maybe your service was crashing  and you just didn't notice it before?  That property doesn't exist in Indy 10.

7 hours ago, c0d3r said:

procedure TMyService.OnStop(...)

begin

    TransportServer.Close ;  <-- Error 1053 occurs.

end;

 

TransportServer is a type of TkbmMWTCPIPIndyServerTransport, the Close method internally was just calling TidTCPServer.Active := False;

Can you show what your TIdTCPServer event handlers are doing?

[c0d3r 17]

54 minutes ago, Remy Lebeau said:

That usually happens if you cause a thread deadlock inside your TIdTCPServer event handlers.  For instance, by syncing with the same thread that is trying to deactivate the server.  The Active setter closes all open client sockets, AND waits for the owning threads to terminate.  The server events are fired in the context of those threads.  So, either don't perform thread-blocking syncs during server shutdown, or else do the server deactivatation in a separate thread so the syncs can be processed normally.

TIdTCPServer shutdown hasn't really changed between Indy 9 and 10.  Sure, the underlying architecture is a bit different, but the basics are still the same.  Although, you might have been getting saved by the TIdTCPServer.TerminateWaitTime property in Indy 9, which defaulted to 5 seconds.  If the server shutdown exceeded that timeout, an EIdTerminateThreadTimeout exception would be raised, maybe your service was crashing  and you just didn't notice it before?  That property doesn't exist in Indy 10.

Can you show what your TIdTCPServer event handlers are doing?

@Remy Lebeau Thanks for the answers.  Much appreciated.  We didn't write any codes for any TidTCPServer event handlers.  The TransportServer is a TkbmMWTCPIPIndyServerTransport object, it uses TIdTCPServer internally as its Socket instance. What we did was that in OnServiceStartEvent, calling TransportServer.Listen (which calls TidTCPServer.Active := True),  and in OnServiceStopEvent,  calling TransportServer.Close (which calls TidTCPServer.Active := False),  kbmMW handles everything else underneath (dealing with client side requests/server side responses).

 

As for TerminateWaitTime property default to 5 seconds,  we never set or change it by any chance,  I thought Windows service timeout was set way longer (at least 30 seconds, we tested 90 seconds in the past) than TerminateWaitTime.  The hard part was that we can't reproduce on our development machines.  Only some of our customers who were heavily using the servers were having the issue.  We don't know how to debug at this point and where we should look at from.

 

Edited February 9, 2021 by c0d3r

[c0d3r 17]

Here was what I found tonight on one of our customers server:   While stopping the service and timeout, there was a remote connection to the port that having FIN_WAIT_1 status (using netstat -a),   What was that?

[chmichael 12]

cod3r are you sure that it isn't a kbmMW bug that you should report to Kim ?

[Guest]

My thought on this, and i never used the library, so it is a guess work

4 hours ago, c0d3r said:

there was a remote connection to the port that having FIN_WAIT_1

Getting FIN_WAIT_1 should have triggered a flag somewhere, so it is one of the following

1) An exception was raised and you silently ignoring it, or the kbmMW did ignored it silently, and that is wrong, this is unlikely by the library.

2) kbmMW is capturing it and passing it to an event to be handled or observed, and you missed assigning such event (callback)

3) kbmMW is buggy, it is unlikely, i think such library is already mature enough after all these years to miss such basic functionality.

 

Also here from this

7 hours ago, c0d3r said:

calling TransportServer.Listen (which calls TidTCPServer.Active := True),  and in OnServiceStopEvent,  calling TransportServer.Close (which calls TidTCPServer.Active := False)

I think there is something missing, here because calling TransportServer.Close ; should not be that simple to handle exactly these cases where there is an active connection, so it either you should be calling something before Close like Dissconnect, abort, StopListen... or Close itself can have a parameter like Close(True), that is the right and normal design for any sockets handling library, unless there is some properties can be should be control this process of closing/disconnecting.

 

in any case, i think you should refer to the documentation, browse the demos and look for the right way to close/terminate.

[Lars Fosdal 1793]

Here is how our TIdTCPServer shutdown code looks.

 

Quote

      TCPServer.OnConnect := nil;
      TCPServer.OnExecute := nil;
      TCPServer.TerminateWaitTime := 0;
      TCPServer.StopListening;
      try
        TCPServer.Active := False;
      except
        on E:Exception
        do DebugOutException(E, 'TPSD_TCPServer.Destroy');
      end;

That exception doesn't really happen anymore, though. 🙂

[c0d3r 17]

8 hours ago, chmichael said:

cod3r are you sure that it isn't a kbmMW bug that you should report to Kim ?

As usual I sent email to Kim last week, no response.  Wish me luck if I get response.  Try registrating to its forum to ask questions, nope,  my work email was used and can't register,  try 'Forgot/Lost Password' link,  Oops,  your email doesn't exist.  I sent email to Kim about registration issue on Sunday,  lets see when I would get the response, so far no any response. 

 

EDIT:   Oh, Oh,  the only email I can always get from him was the email that asking me to pay the annual subscription fee on due date.

Edited February 9, 2021 by c0d3r

[c0d3r 17]

5 hours ago, Kas Ob. said:

My thought on this, and i never used the library, so it is a guess work

Getting FIN_WAIT_1 should have triggered a flag somewhere, so it is one of the following

1) An exception was raised and you silently ignoring it, or the kbmMW did ignored it silently, and that is wrong, this is unlikely by the library.

2) kbmMW is capturing it and passing it to an event to be handled or observed, and you missed assigning such event (callback)

3) kbmMW is buggy, it is unlikely, i think such library is already mature enough after all these years to miss such basic functionality.

 

Also here from this

I think there is something missing, here because calling TransportServer.Close ; should not be that simple to handle exactly these cases where there is an active connection, so it either you should be calling something before Close like Dissconnect, abort, StopListen... or Close itself can have a parameter like Close(True), that is the right and normal design for any sockets handling library, unless there is some properties can be should be control this process of closing/disconnecting.

 

in any case, i think you should refer to the documentation, browse the demos and look for the right way to close/terminate.

 

No Disconnect/StopListen method,  only Listen and Close,  and the codes were there for 10+ years since 2007,  people who use our old app (built with Delphi 2007, Indy 9) are still working great.  Its those people who upgraded to our new app (built with Delphi 10.4.1, Indy 10) are having the issue.

[Remy Lebeau 1421]

13 hours ago, c0d3r said:

We didn't write any codes for any TidTCPServer event handlers.

You can't use TIdTCPServer without at least an OnExecute event handler, otherwise setting the Active property to true will raise an EIdTCPNoOnExecute exception.  Are you suggesting that your OnExecute handler is blank, doing nothing at all?

13 hours ago, c0d3r said:

The TransportServer is a TkbmMWTCPIPIndyServerTransport object, it uses TIdTCPServer internally as its Socket instance. What we did was that in OnServiceStartEvent, calling TransportServer.Listen (which calls TidTCPServer.Active := True),  and in OnServiceStopEvent,  calling TransportServer.Close (which calls TidTCPServer.Active := False),  kbmMW handles everything else underneath (dealing with client side requests/server side responses).

I'm not familiar with kbmMW, why are you using TIdTCPServer at all if it does nothing?  I don't understand the rational here.

6 hours ago, Kas Ob. said:

I think there is something missing, here because calling TransportServer.Close ; should not be that simple to handle exactly these cases where there is an active connection, so it either you should be calling something before Close like Dissconnect, abort, StopListen... or Close itself can have a parameter like Close(True), that is the right and normal design for any sockets handling library, unless there is some properties can be should be control this process of closing/disconnecting.

As I said earlier, setting TIdTCPServer.Active to False will handle the disconnect of active client sockets, closing the listening sockets, and waiting for their threads to fully terminate.

[c0d3r 17]

17 minutes ago, Remy Lebeau said:

You can't use TIdTCPServer without at least an OnExecute event handler, otherwise setting the Active property to true will raise an EIdTCPNoOnExecute exception.  Are you suggesting that your OnExecute handler is blank, doing nothing at all?

I'm not familiar with kbmMW, why are you using TIdTCPServer at all if it does nothing?  I don't understand the rational here.

As I said earlier, setting TIdTCPServer.Active to False will handle the disconnect of active client sockets, closing the listening sockets, and waiting for their threads to fully terminate.

Sorry, I didn't make it clear.  its the kbmMWTCPIPIndyServerTransport that use its FSocket: TidTCPServer,  to handle everything using some private/protected/public methods, including Execute event. e.g.

 

TransportServer.Listen ->  FSocket.Active := True;

 

TransportServer.Close -> FSocket.Active := False;

 

[Remy Lebeau 1421]

9 minutes ago, c0d3r said:

Sorry, I didn't make it clear.  its the kbmMWTCPIPIndyServerTransport that use its FSocket: TidTCPServer,  to handle everything using some private/protected/public methods, including Execute event.

Well, then that goes back to my earlier question - what do those methods actually look like?  Because there is likely a deadlock happening inside one of them, preventing one or more socket threads from terminating correctly, thus causing the TIdTCPServer.Active property setter to hang waiting for it.

 

[c0d3r 17]

3 hours ago, Remy Lebeau said:

Well, then that goes back to my earlier question - what do those methods actually look like?  Because there is likely a deadlock happening inside one of them, preventing one or more socket threads from terminating correctly, thus causing the TIdTCPServer.Active property setter to hang waiting for it.

 

@Remy Lebeau  Thanks, I totally agree it.  just not sure how to find.  Any workarounds we can do?  Is there any properties/methods in TidTCPServer that can let all these socket threads being terminated gracefully before calling FSocket.Active := False; 

Edited February 9, 2021 by c0d3r

[Remy Lebeau 1421]

4 hours ago, c0d3r said:

@Remy Lebeau  Thanks, I totally agree it.  just not sure how to find.

That is why I keep asking you to SHOW YOUR CODE (or at least a small demo that reproduces the same issue).  We can't tell you what is wrong if we can't see what you are actually doing.

Quote

Is there any properties/methods in TidTCPServer that can let all these socket threads being terminated gracefully before calling FSocket.Active := False; 

No.  It is your responsibility to not write code that blocks the socket threads from terminating properly.

Edited February 9, 2021 by Remy Lebeau

[c0d3r 17]

1 hour ago, Remy Lebeau said:

That is why I keep asking you to SHOW YOUR CODE (or at least a small demo that reproduces the same issue).  We can't tell you what is wrong if we can't see what you are actually doing.

No.  It is your responsibility to not write code that blocks the socket threads from terminating properly.

@Remy Lebeau I would like to give the codes, but I'm not allow to, they are kbmMW codes, all socket threads related things are done by kbmMW components internally.  What we did was just wrote codes in one of its OnRequest event handler to deal with client request by the given request name,  and set the return Result.  For each request, We made 100% sure to have try--finally--end blocks to have objects created get freed properly, each LockList to has a matched UnlockList to access thread string lists, ..., etc.   Anyway,  Thanks for your time to anwser all my questions, really appreciated!

 

The codes in ProcessRequest event:   

@ServiceMethod := MethodAddress(AClientRequestName);
if Assigned(ServiceMethod) then
begin
  Result := ServiceMethod(Self, ClientIdent, Args);
  Handled := True;
end;

 

This one of the request codes we wrote (in published section):

 

function TMyService.GetSequenceValue(const ClientIdent: TkbmMWClientIdentity;
  const Args: array of Variant): Variant;
var
  ds: TkbmMWDataset;
  SeqName, Step: string;
  SeqList: TStringList;
  i: Integer;
begin
  Result := -1;
  if not PrepareQuery(SEQUENCE_GETVALUE, ds) then Exit;
  try
    SeqName := VarToStr(Args[0]);
    Step := VarToStr(Args[1]);
    SeqList := FSequences.LockList;
    try
      i := SeqList.IndexOfName(SeqName);
      if i < 0 then Exit;
      SeqName := SeqList.ValueFromIndex[i];
    finally
      FSequences.UnlockList;
    end;

    ds.Query.Text := Format(ds.Query.Text, [SeqName, Step]);
    ds.Open;

    if not ds.Eof then
      Result := ds.FieldByName('ID').AsInteger;
  finally
    ds.Free;
  end;
end;

 

Edited February 9, 2021 by c0d3r

[Remy Lebeau 1421]

1 hour ago, c0d3r said:

@Remy Lebeau I would like to give the codes, but I'm not allow to, they are kbmMW codes, all socket threads related things are done by kbmMW components internally.

Without seeing your TIdTCPServer code, there is no way to help you diagnose why your TIdTCPServer is freezing up.  What you describe is a classic deadlock scenario, but there are so many different ways a deadlock could happen.

[chmichael 12]

On 2/9/2021 at 5:47 PM, c0d3r said:

As usual I sent email to Kim last week, no response.  Wish me luck if I get response.  Try registrating to its forum to ask questions, nope,  my work email was used and can't register,  try 'Forgot/Lost Password' link,  Oops,  your email doesn't exist.  I sent email to Kim about registration issue on Sunday,  lets see when I would get the response, so far no any response. 

  

EDIT:   Oh, Oh,  the only email I can always get from him was the email that asking me to pay the annual subscription fee on due date.

 

Well i made him notice you! Post the report on his forums and if it's an Indy bug i'm sure that @Remy Lebeau will help!

Edited February 17, 2021 by chmichael

[c0d3r 17]

15 hours ago, chmichael said:

 

Well i made him notice you! Post the report on his forums and if it's an Indy bug i'm sure that @Remy Lebeau will help!

I got Kim's emails 3 days ago.  Thanks.

Edited February 18, 2021 by c0d3r

[c0d3r 17]

@Remy LebeauAlthough we still can't figure it out after weeks (we don't deal with any TCP/IP socket codes), we found a workaround: kbmMW's Server component has a property called 'DisconnectAfterResponse'.  By setting it to True, Most of our clinic customers don't have any TCP/IP socket close freezing issues, however few were still having the problem, they were the large and busiest clinics.  We are planning to use Indy Github version to see if it helps, I'm just wondering whats difference between GitHub version and the one comes with Sydeny?

Edited April 1, 2021 by c0d3r

[Remy Lebeau 1421]

6 hours ago, c0d3r said:

@Remy LebeauAlthough we still can't figure it out after weeks (we don't deal with any TCP/IP socket codes)

Again, there is really nothing I cn do to help you without seeing you actual TCP server code.  You clearly have a logic issue somewhere, but I can't see what you are doing with it.

Quote

we found a workaround: kbmMW's Server component has a property called 'DisconnectAfterResponse'.  By setting it to True, Most of our clinic customers don't have any TCP/IP socket close freezing issues, however few were still having the problem, they were the large and busiest clinics.

That workaround only minimizes the issue, but does not eliminate it.  There could still be clients connected when the service is being shut down, and it only takes 1 misbehaving client/thread to cause the problem.

Quote

We are planning to use Indy Github version to see if it helps, I'm just wondering whats difference between GitHub version and the one comes with Sydeny?

Very little, since Embarcadero had incorporated the latest GitHub version (at the time) shortly before Sydney's release.  There have been a number of checkins to GitHub since that time, but nothing that would address this particular issue.

Edited April 2, 2021 by Remy Lebeau