Exception Handling with a Middle Tier Application

[JIMSMITH 1]

I have an application  that connects a middle tier application that I created all using VCL windows.  The middle tier application is accessed VIA tcp using Indy.  The client application from many computers can access the middle tier at the same time.  The problem I have is how do I harden the middle tier so that if an exception happens the entire middle tier is not hung.  I do want the exception message to go back to the respective client, but I need the middle tier to be able to continue processing requests from other users.

[Lars Fosdal 1826]

Firstly, you need to know the nature of the exception.  Do you use EurekaLog or MadExcept? If not, you are blind to the actual cause of the exceptions.

Do you know what your code is doing when it stops processing requests or why it stops?

 

Once you know what goes wrong, add checks to avoid execution when you have invalid data.

 

It may be useful to have a try/except block to set return values that indicate a failure, instead of exiting by exception, but it is rarely useful to have "catch all" blocks simply to silence exceptions.

 

Secure each piece of code that runs in a thread and uses shared resources with try/finally so that any resource you allocate can be returned to the shared pool, and any locks you grab can be unlocked.

 

 

[PeaShooter_OMO 24]

Adding to Lars's advice above. A side note in respect to Indy...

 

I mostly use Indy's TidTCPServer and in it each connection (context) has its own thread where your code is run, the OnExecute event. If you have not caught an exception in those threads through your code then it will be caught by TidTCPServer in anyway and it will affect that specific connection. It should not affect other connections that were made to TidTCPServer. This is in relation to Indy itself. It is of course another ball game if you are talking about exceptions inside your server application but outside Indy's influence where your code determines how it affects the continious working of the server application in regards to the other clients. Extra care should be given as Lars said.

 

Exceptions related to Indy TCP socket operations are important to TidTCPServer itself and if you catch an exception when doing socket operations so you can act on them then it is always a good idea to also let that exception flow upwards so that TidTCPServer can also get it in order for it to take appropriate action like closing a socket.

 

Prevention is better than cure and anywhere in code where you might get an exception is a good place to handle each of them and respond appropriately.

Edited March 6 by PeaShooter_OMO

[A.M. Hoornweg 148]

I use Remobjects Remoting SDK as a middleware and that is perfectly capable of passing an exception that happens in a remote procedure call back to the client without hanging. The exception is then raised in the client.

[JIMSMITH 1]

6 hours ago, A.M. Hoornweg said:

I use Remobjects Remoting SDK as a middleware and that is perfectly capable of passing an exception that happens in a remote procedure call back to the client without hanging. The exception is then raised in the client.

I tried to use Remobjects many years ago.  Wasn't much documenrtation or many examples.  Any questions took several days to get answers.  Anyway I wonder how they handle exceptions so that other users continue to process without disruption..

[JIMSMITH 1]

7 hours ago, PeaShooter_OMO said:

Adding to Lars's advice above. A side note in respect to Indy...

 

I mostly use Indy's TidTCPServer and in it each connection (context) has its own thread where your code is run, the OnExecute event. If you have not caught an exception in those threads through your code then it will be caught by TidTCPServer in anyway and it will affect that specific connection. It should not affect other connections that were made to TidTCPServer. This is in relation to Indy itself. It is of course another ball game if you are talking about exceptions inside your server application but outside Indy's influence where your code determines how it affects the continious working of the server application in regards to the other clients. Extra care should be given as Lars said.

 

Exceptions related to Indy TCP socket operations are important to TidTCPServer itself and if you catch an exception when doing socket operations so you can act on them then it is always a good idea to also let that exception flow upwards so that TidTCPServer can also get it in order for it to take appropriate action like closing a socket.

 

Prevention is better than cure and anywhere in code where you might get an exception is a good place to handle each of them and respond appropriately.

I use indy tidtcpserver and client.  The other day I had a computer that did not have ssl .dll files which resulted in exception in the middle tier.  All processing was disrupted.  I just need to send exception back to that client connection and allow the other connections continue to process.

[Anders Melander 1934]

9 minutes ago, JIMSMITH said:

Anyway I wonder how they handle exceptions so that other users continue to process without disruption..

Supposedly each request is handled by a thread - Like in every other middleware.

 

If your experience is that an exception in the request handling of the middle-tier blocks the whole server then there's either something seriously wrong with the middleware or the architecture of the middle-tier. The problem is probably that the main thread is somehow involved in the request handling.

 

Not that it's related to your problem, but if you want middle-tier exceptions passed back to the client, as exceptions, you can look in the DataSnap source code to see how it's done.

[Remy Lebeau 1529]

4 hours ago, JIMSMITH said:

I use indy tidtcpserver and client.  The other day I had a computer that did not have ssl .dll files which resulted in exception in the middle tier.  All processing was disrupted.

Can you be more specific?  What exactly was disrupted?  It makes sense that lack of OpenSSL DLLs would cause errors for clients that try to connect using TLS, but that is handled on a per-client basis, it should have disrupted the entire server as a whole.

4 hours ago, JIMSMITH said:

I just need to send exception back to that client connection and allow the other connections continue to process.

Once a TLS handshake begins, if TLS fails for any reason, you can't continue communicating with that failed client, the connection must be dropped.  OpenSSL MIGHT be able to send an alert to the client, depending on what stage of TLS the error occurs at, but that is the extent that can be sent.  You certainly cannot send the original exception back to the client.

 

On the other hand, if an exception happens that is not related to TLS, then you can certainly catch it and send whatever you want back to the client, but that depends on the design of your application protocol.  As long as you do that in a thread-safe manner, it shouldn't affect other clients.

[JIMSMITH 1]

9 hours ago, Remy Lebeau said:

Can you be more specific?  What exactly was disrupted?  It makes sense that lack of OpenSSL DLLs would cause errors for clients that try to connect using TLS, but that is handled on a per-client basis, it should have disrupted the entire server as a whole.

Once a TLS handshake begins, if TLS fails for any reason, you can't continue communicating with that failed client, the connection must be dropped.  OpenSSL MIGHT be able to send an alert to the client, depending on what stage of TLS the error occurs at, but that is the extent that can be sent.  You certainly cannot send the original exception back to the client.

 

On the other hand, if an exception happens that is not related to TLS, then you can certainly catch it and send whatever you want back to the client, but that depends on the design of your application protocol.  As long as you do that in a thread-safe manner, it shouldn't affect other clients.

This is a simple case where the 2 .dll files were not with the executable.  The problem was not with the indy components either.  The key information that I am looking for is how to manage the middle tier so that when any type of exception happens that the middle tier sends the exception to the client and continues to process other connections.  Yes I can do this in the application using try finally and try except blocks.  Just not sure how to implement this in a middle tier where I want the exception to be passed back to the client.

[PeaShooter_OMO 24]

2 hours ago, JIMSMITH said:

Yes I can do this in the application using try finally and try except blocks.  Just not sure how to implement this in a middle tier where I want the exception to be passed back to the client.

Exceptions are application bound and as far as I know they cannot be passed between applications (would be interested to know if they could be). 

 

I believe you may be approaching this incorrectly. What happens on the server is not an extension of what happens on the client. The TCP connection is merely a channel to exchange data and each side has its own municipality and exceptions should stay in their own municipalities. 

 

For instance; Let's say the client is currently executing from inside a procedure and it sends a request or some other data to the server. The server then does something with it and maybe an exception happens. It means the exception happened on the server side, not the client side. The server code is supposed to handle that in an acceptable Delphi standard way and then inform the client that there was an issue and as such appropriate action should take place, like resending the request, etc. It does not mean that an exception happened inside that client procedure thus the client does not need an exception, it only needs a headsup that an issue occured which could be remedied. Also, if the exception happened because of a Non-TCP-related issue then there should be no reason for the TCP communications to be affected.

But I think the most important for you would be to handle all your exceptions as a default way of coding and then decide what to do when an exception happens. Exceptions should not halt an application if you coded in a preventative manner. After you coded exception-handling for all possible exceptions you could think of then you design a TCP protocol around what you want to achieve. For instance the client sends a request to the server for data then the server will try to get the data from its source and send it back to the client. If the server fails to get the data then the server will inform the client it failed just by saying "Hey I could not get that data for you" and not "Here's an exception for you". The client can decide what it must do next. There should be no halts, no freezing, no sudden stoppages.

 

A side note; Many exceptions can be avoided by looking at state beforehand and acting upon that instead of waiting for an exception to happen. If you don't want a situation of StrToInt('hello') to cause an exception then don't allow 'hello' to reach that point. In a way it can be proactive instead of reactive.

Edited March 7 by PeaShooter_OMO

[Lars Fosdal 1826]

3 hours ago, JIMSMITH said:

Just not sure how to implement this in a middle tier where I want the exception to be passed back to the client.

It isn't worse than your response having result codes that indicate success + data, or failure + type of failure, and your client responding to these the way you want it to.

 

F.x
Client -> Server, send me all the details on <some reference>

<- Server: dude, that was a bad reference - I almost crashed!

Client: Well, server didn't like that, so I better tell the user that the server couldn't find his shit and that the request upset the server.

[A.M. Hoornweg 148]

21 minutes ago, Lars Fosdal said:

It isn't worse than your response having result codes that indicate success + data, or failure + type of failure, and your client responding to these the way you want it to.

 

F.x
Client -> Server, send me all the details on <some reference>

<- Server: dude, that was a bad reference - I almost crashed!

Client: Well, server didn't like that, so I better tell the user that the server couldn't find his shit and that the request upset the server.

 

The disadvantage of that is that it doesn't tell you in detail what went wrong on the server.  It is also tedious, the server needs to CATCH exceptions and convert them into something meaningful (an error code?) to pass back to the client, hopefully yielding sufficient information.  

 

 

If the framework itself is designed in such a way that it can pass exceptions back from server to client, the client will re-raise the same exception type (eOverflow, eConvertError, eInOutError, eListError...) with (hopefully) the server's error address in the error message so you can easily find the root cause.   It also simplifies server-side development enormously; if something goes wrong on the server, it can simply raise an exception and it will be passed back to the client. 

 

 

 

 

[PeaShooter_OMO 24]

52 minutes ago, A.M. Hoornweg said:

The disadvantage of that is that it doesn't tell you in detail what went wrong on the server

Whether you pass a specific exception to the client or a specific failure code to the client comes down to the same thing. I do not see the need for the client to be handling an exception which happened on the server. The client literally only needs a proper response from the sever with enough information for the client to know exactly what happened. A specific exception (eOverflow, eConvertError, eInOutError, eListError...) does not have more information than a well coded failure code with additional accompanying data (if required). This all comes down to proper coding practices and a well thought out application layer protocol.

 

I also think this might be moot because sending an exception to another application is as far as I know impossible. What might look like its is being done in other frameworks might just be the framework getting information via TCP from the other side and then just reraising the relevant exception on this side.

Edited March 7 by PeaShooter_OMO

[Lars Fosdal 1826]

@A.M. Hoornweg Passing the exception type and callstack info from the server would be useful for making an educated handling in the client, but I don't see why raising an exception in the client would be helpful. The callstack from the server side exception won't be the same as the callstack from an exception raised in the client,. It all depends on the client-side context, of course, so sometimes an exception may be acceptable, but frankly; I prefer to not raise an exception if I can handle a problem flow in a better way.

 

[A.M. Hoornweg 148]

19 minutes ago, Lars Fosdal said:

@A.M. Hoornweg Passing the exception type and callstack info from the server would be useful for making an educated handling in the client, but I don't see why raising an exception in the client would be helpful. The callstack from the server side exception won't be the same as the callstack from an exception raised in the client,. It all depends on the client-side context, of course, so sometimes an exception may be acceptable, but frankly; I prefer to not raise an exception if I can handle a problem flow in a better way.

 

One advantage is that it's way less work server-side.  If anything goes wrong, you need not worry too much about how to inform the client because that information is already contained in the exception that was thrown by whatever action failed.  The stack is unrolled automatically just like in regular Delphi apps and the exception information is automatically propagated to the client.

 

 

Client-side, a remote procedure call is used like any other Delphi call.  In the case of an error, you use structured exception handling and use the exception type as a criterium for how the client should proceed.

 

try
  server.executequery(query)
except
  on e:exception do
  begin
     if e is eCommsError then  //did the internet connection fail?  If so, client should try again later.
           ...
      else
      if e is eDatabaseError then 
        showmessage(format('The server really didn''t like this query, this is wrong with it: (%s) "%s"',[e.classname, e.message]))
      else 
       ...
   end;
end;

 

 

 

 

 

 

[Anders Melander 1934]

4 hours ago, A.M. Hoornweg said:

If the framework itself is designed in such a way that it can pass exceptions back from server to client, the client will re-raise the same exception type (eOverflow, eConvertError, eInOutError, eListError...)

The exception type is context that only has meaning in the server and the client might not even have code that implement the exception type. Of course the client middleware code can perform mapping between the server exception type and a client exception type, based on the string name, an error code or something like that, but why should it? The client just needs to know that a request failed so a generic exception should be adequate.

 

4 hours ago, PeaShooter_OMO said:

I do not see the need for the client to be handling an exception which happened on the server.

It entirely depends on how the client is calling the server and what the client code is doing. Getting an exception back from a server call might be preferred to having to make a call, check an error code, decide on what to do. I.e. traditional error handling vs. exception handling.

The client isn't expected to handle the server problem but it needs to know that something went wrong. What went wrong is just context that can be used to decide flow or fix the problem after the fact.

 

4 hours ago, PeaShooter_OMO said:

I also think this might be moot because sending an exception to another application is as far as I know impossible. What might look like its is being done in other frameworks might just be the framework getting information via TCP from the other side and then just reraising the relevant exception on this side.

Yes, of course that's what they are doing. And they do it because exception handling is often preferred to checking status codes after each call.

 

3 hours ago, Lars Fosdal said:

Passing the exception type and callstack info from the server would be useful for making an educated handling in the client, but I don't see why raising an exception in the client would be helpful.

If you have ever written code against COM servers using the safecall calling convention you would know. Safecall converts the regular COM error codes into exceptions which means that you can treat a COM object like you would any other Delphi object. You don't call GetLastError after each and every call to the object; You expect errors to be raised as exceptions.

Now extend this to DCOM. The COM server might reside on another system but your error handling stays exactly the same. Errors in the COM objects are still raised as exceptions in the client.

 

Any call stack that the server might pass on to the client is just context for a bug report and not meant to be consumed by the client.

 

 

2 hours ago, A.M. Hoornweg said:

One advantage is that it's way less work server-side.

And client-side.

 

2 hours ago, A.M. Hoornweg said:

If anything goes wrong, you need not worry too much about how to inform the client because that information is already contained in the exception that was thrown by whatever action failed.  The stack is unrolled automatically just like in regular Delphi apps and the exception information is automatically propagated to the client.

Client-side, a remote procedure call is used like any other Delphi call.  In the case of an error, you use structured exception handling and use the exception type as a criterium for how the client should proceed.

Exactly. I should have read the whole thread before responding 🙂

[JIMSMITH 1]

12 hours ago, A.M. Hoornweg said:

 

The disadvantage of that is that it doesn't tell you in detail what went wrong on the server.  It is also tedious, the server needs to CATCH exceptions and convert them into something meaningful (an error code?) to pass back to the client, hopefully yielding sufficient information.  

 

 

If the framework itself is designed in such a way that it can pass exceptions back from server to client, the client will re-raise the same exception type (eOverflow, eConvertError, eInOutError, eListError...) with (hopefully) the server's error address in the error message so you can easily find the root cause.   It also simplifies server-side development enormously; if something goes wrong on the server, it can simply raise an exception and it will be passed back to the client. 

 

 

 

 

You are correct.  This is what I am about to correct.  I am getting some good clues from these responses.  A response could be sent to the client in xml with Success or fail.  Success would have the data and fail would have the extended error details.  In the case of the ssl failure trapping the message and logging it to a file on the server and then sending the XML with failure would normally work, but this was a failure on connection so there might not be a client to respond to.  Just log it on the server when capturing via the exception handler and just move on so the next client communication happens.  This might be what I am looking for.