Jump to content
Joe Sansalone

Indy HTTP server

Recommended Posts

(Currently trying with updated openssl DLL files to see if it fixes the problem .. although I need to wait a while to know for sure)

 

In the meantime ...

 

Hi,

 

When testing a Indy Server application everything was fine.

Now in production, it stops responding to requests after a while.

 

Is it ok to bind multiple ports like this?  (should bind to the same default IP, right?)

 

if not FServer.Active then
  begin
    FServer.Bindings.Clear;
    FServer.Bindings.Add.Port := 443;
    FServer.DefaultPort := StrToInt(EditPort.Text);
    FServer.Bindings.Add.Port := 80;
    FServer.Bindings.Add.Port := 8080;  // for testing
    FServer.Active := True;
  end;

 

 

Another question:

 

If an exception occurs in the OnCommandGet of the server and is not handled, would this screw things up for Indy HTTP server?

 

Am I fine as long as I set this before end of OnCommandGet? 

    AResponseInfo.ResponseNo := 200;
    AResponseInfo.ContentText := 'OK';  // or whatever appropriate response here
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"'; 

 

 

What else could I be doing to get IdHTTPServer to stop responding to requests?

 

Thanks,

Joe

 

 

 

Edited by Joe Sansalone

Share this post


Link to post
2 hours ago, Joe Sansalone said:

When testing a Indy Server application everything was fine.

Now in production, it stops responding to requests after a while.

That has nothing to do with the ports you are using, unless you have a firewall that is blocking them.  Which version of Indy and OpenSSL DLLs are you using?

Quote

Is it ok to bind multiple ports like this?  (should bind to the same default IP, right?)

Yes.  Though, you don't need to set the DefaultPort at all if you are just going to set each Binding.Port explicitly.  If you are going to bind to multiple ports, then the DefaultPort should be 0.  Just make sure you are handling the OnQuerySSLPort event correctly, especially for non-standard ports, like 8080.  Also, make sure you are using an up-to-date version of Indy, since the behavior of default HTTPS handling did change a few years ago.

Quote

If an exception occurs in the OnCommandGet of the server and is not handled, would this screw things up for Indy HTTP server?

No, that is perfectly fine, even expected and intentionally designed for.  You SHOULD be throwing exceptions on unexpected failures. The server will catch and process uncaught exceptions internally.  For socket I/O errors, it will close the socket and stop the socket's owning thread, triggering OnDisconnect and OnException events.  For most other exceptions, it will instead trigger the OnCommandError event, then send an error response to the client if a response has not already been sent, and then carry on with normal HTTP KeepAlive handling - closing the socket or waiting for the next HTTP request, as needed.

Quote

Am I fine as long as I set this before end of OnCommandGet?

If an exception is raised and not caught, the server will catch it, and in most cases will overwrite those values with its own error values.

Quote

What else could I be doing to get IdHTTPServer to stop responding to requests?

There is no way to answer that with the limited information you have provided.  Do you have KeepAlive enabled or disabled on the server?  Once the server stops responding, are you getting OnConnect events for new connections, and getting OnHeadersAvailable events for new requests, at least?  What do your OnCommand... events actually look like?  Have you verified with a packet sniffer like Wireshark that your server machine is still receiving new HTTP requests from the network?

Edited by Remy Lebeau

Share this post


Link to post

I'm using Indy 10.6.2.0.

The OpenSSL DLLs were more than a year old, so I just updated the DLLs today with 1.0.2u version (Jun 25, 2021).

I'm hoping these new DLLs might solve the problem - I later learned that Twilio server error logs indicated that a problem occurred 

during the TLS handshaking.   Twilio servers send requests to my HTTP(S) application server.

 

I am using the OnQuerySSLPort event.  I only return True (use SSL) for the 443 port.

 

I didn't get a chance to use Wireshark when the problem happened.

If it happens again, I will.

 

My OnCommand events do not create any object.  They use pre-created objects

with pre-connected database connections to handle requests.  I was careful to make

sure the code is thread-safe.  In general, all requests are handled by separate threads 

anyway (pooled).

 

Thanks for pointing out OnConnect, OnHeadersAvailable events ... 

I'll put some code there to log info so that the next time it happens, I'll see more.

Also, OnException/OnCommandError ... I'll put some logging code.

 

Thanks for your help.

 

I will write again - either saying it's fixed OR if it happens again 

I'll update this thread with more logging information.

 

Joe

 

 

Share this post


Link to post
1 hour ago, Joe Sansalone said:

I'm using Indy 10.6.2.0.

Which specific release of 10.6.2 exactly?  Are you using a stock version that shipped pre-installed in a specific IDE version?  Or have you updated to the latest trunk version from Indy's GitHub repo?  Indy's versioning has been broken since Indy switched from SVN to GitHub, so 10.6.2.0 does not accurately reflect the real version.

1 hour ago, Joe Sansalone said:

The OpenSSL DLLs were more than a year old, so I just updated the DLLs today with 1.0.2u version (Jun 25, 2021).

That should be OK.

1 hour ago, Joe Sansalone said:

I'm hoping these new DLLs might solve the problem - I later learned that Twilio server error logs indicated that a problem occurred 

during the TLS handshaking.

Any specific details about the errors?

1 hour ago, Joe Sansalone said:

I am using the OnQuerySSLPort event.  I only return True (use SSL) for the 443 port.

OK.

1 hour ago, Joe Sansalone said:

My OnCommand events do not create any object.  They use pre-created objects

with pre-connected database connections to handle requests.  I was careful to make

sure the code is thread-safe.  In general, all requests are handled by separate threads 

anyway (pooled).

Any synchronization between threads being done?

Share this post


Link to post

I'm using Indy 10.6.2.0 that came with Delphi 10.4.2.

 

Details on error from Twilio:

SSL/TLS Handshake Error

An attempt to retrieve content from https://live.projectone.ca/Phone returned the HTTP status code 502

During SSL/TLS negotiation, Twilio experienced a connection reset.

 

12 hours ago, Remy Lebeau said:

Any synchronization between threads being done?

 

Yes, there's some synchronization.  I'm pretty sure I put some timeouts to make sure 

that it returns.   You bring up a good point.  I'm assuming you are suggesting that

synchronization can cause the OnCommand events to timeout, correct?  Which could

be a problem.  

Share this post


Link to post
1 hour ago, Joe Sansalone said:

I'm using Indy 10.6.2.0 that came with Delphi 10.4.2.

OK.

Quote

Details on error from Twilio:

SSL/TLS Handshake Error

An attempt to retrieve content from https://live.projectone.ca/Phone returned the HTTP status code 502

Makes sense, if Twilio sends a request to ProjectOne, which sends a request to your server, which fails, then ProjectOne would send an error back to Twilio.

Quote

During SSL/TLS negotiation, Twilio experienced a connection reset.

Are you sure it is Twilio and not ProjectOne? In any case, a connection reset during a TLS handshake usually means the server did not like something in the client's (Twilio/ProjectOne) handshake data, so it simply chose to abort the connection rather than send a TLS alert back to the client explaining the reason.

Quote

Yes, there's some synchronization.  I'm pretty sure I put some timeouts to make sure 

that it returns.   You bring up a good point.  I'm assuming you are suggesting that

synchronization can cause the OnCommand events to timeout, correct?

Hang/deadlock rather than timeout, but yes, if you are not careful with it.

Edited by Remy Lebeau

Share this post


Link to post
40 minutes ago, Remy Lebeau said:

Hang/deadlock rather than timeout, but yes, if you are not careful with it.

I'll review my code looking for possible problems with synchronization (although I remember being careful when writing it).

 

The production application now has logging on OnException, OnCommendError, OnHeadersAvailable, OnConnect.

And using the updated SSL DLLs.

 

So I'll wait and see if it happens again.  

 

Share this post


Link to post

Now that logging is enabled on the OnException event handler of TIdHTTPServer,

I'm seeing all sorts of different exceptions.  

 

However, the application is responding correctly.

These are other "bad" requests hitting the server.

 

I guess this is normal?  

 

Below is part of the log:

 

 

[HTTP          : Exception 08/10 07:01:51.935] OnException: Error accepting connection with SSL.
EOF was observed that violates the protocol

[HTTP          : Exception 08/10 07:45:07.863] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:08.035] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:08.113] OnException: Error accepting connection with SSL.
error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher
[HTTP          : Exception 08/10 07:45:08.379] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:08.582] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:08.738] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:08.926] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:09.113] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:09.348] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:09.567] OnException: Socket Error # 10054
Connection reset by peer.
[HTTP          : Exception 08/10 07:45:29.097] OnException: Connection Closed Gracefully.
[HTTP          : Exception 08/10 08:05:17.495] OnException: Connection Closed Gracefully.
[HTTP          : Exception 08/10 08:33:29.720] OnException: Error accepting connection with SSL.
error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
[HTTP          : Exception 08/10 09:01:26.542] OnException: Error accepting connection with SSL.
EOF was observed that violates the protocol
[HTTP          : Exception 08/10 09:41:58.993] OnException: Error accepting connection with SSL.
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
[HTTP          : Exception 08/10 09:57:41.154] OnException: Connection Closed Gracefully.

Share this post


Link to post

The application (HTTP Server) stopped responding to requests again.

 

This time, I simply "Stopped" and "Started" the TIdHTTPServer without restarting the application and it worked - application started 

responding to requests.

 

 

// Stopped

procedure TForm1.ButtonStopClick(Sender: TObject);
begin
  FServer.Active := False;
  FServer.Bindings.Clear;
end;

 

// Started

procedure TForm1.StartServer;
begin
  if not FServer.Active then
  begin
    FServer.Bindings.Clear;
    FServer.Bindings.Add.Port := 443;
    FServer.DefaultPort := StrToInt(EditPort.Text);
    FServer.Bindings.Add.Port := 80;
    FServer.Bindings.Add.Port := 8080;  // for testing
    FServer.Active := True;
  end;
end;

 

 

I'm either doing something strange in the TIdHTTPServer events or Indy HttpServer has a bug.

 

BELOW is my code.

It's just a bunch of "If then begin end" to handle each Pathinfo differently.

And all of it is wrapped in a Try/Except

 

 

 

 

procedure TForm1.FServerCommandGet(AContext: TIdContext;
  ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
 // engine: TSMSEngineIn;
  DB: TDynamicDBSession;
  task, numMedia, numSegments, profile_id: integer;
  PathInfo, id, temp, DID, NoResponse, callerID, main, mapped, PathInfoUser: string;
  ms: TMemoryStream;
  flowEngine: TFlowEngine;
  sms: TTwilioSMS;
  nexmo: TNexmoSMS;
  ibm: TIBMTranslator;
  from, toDN, text, forwardedFrom: string;
  FLog: TLogger;
  MsgStatus, InParams, tag, ip: string;

  CallSid, TwiMLResponse, engineID, CallStatus, SpeechResult, StableSpeechResult, UnstableSpeechResult,
  seqNumber, DigitsResult, Confidence: string;

   DialCallSid, DialCallStatus, TwilioPayResult, TwilioPayError, QueueResult, QueueTime, DequeingCallSid: string;

  RecordingSid, RecordingUrl, SMSSid: string;
  num, address: string;  // form input from web
  isMobile, privatecall, isOptedOut: boolean;
begin
  PathInfo := ARequestInfo.URI;
  PathInfoUser := UpperCase(PathInfo);  // when it's a user typing url in a browser, we want to make sure they can spell it
                                        // upper, lower etc

try
  NoResponse := '<?xml version="1.0" encoding="UTF-8"?> <Response> </Response>';
  AResponseInfo.ContentType := 'text/html;charset="UTF-8"';


  // if incoming params are empty, it's a "fake" request
  InParams := ARequestInfo.Params.DelimitedText;


  // default Path
  if ((PathInfo = '/') and (InParams <> '')) then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: path /');
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams);

    From := RemovePlus1(aRequestInfo.Params.Values['From']);
    ToDN := RemovePlus1(aRequestInfo.Params.Values['To']);

    // avoid hackers


    if ((From = '') and (ToDN='')) then
    begin
      AResponseInfo.ResponseNo := 404;
      AResponseInfo.ResponseText := 'Bug off';
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'IP: ' + AContext.Binding.PeerIP);
      Exit;
    end;

 

    Text := IndyTextEncoding_UTF8.GetString(ToBytes(aRequestInfo.Params.Values['Body'], IndyTextEncoding_8Bit));

    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'From: ' + From + ' To: ' + ToDN);
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Text: ' +StringReplace(Text, #13#10, ' ', [rfReplaceAll]));


    FDisplayThread.Display(TimeToStr(Now) +   ': SMS From: ' + From + '   To: ' + ToDN);
    FDisplayThread.Display(TimeToStr(Now) +   ': Text: ' + Text);


    try
      NumMedia := StrToInt(aRequestInfo.Params.Values['NumMedia']);
      numSegments := StrToInt(aRequestInfo.Params.Values['NumSegments']);
    except on E:EConvertError do
      begin
        NumMedia := 0;
        numSegments := 1;
      end;
    end;
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'MMS: ' + IntToStr(numMedia));

    // TODO: 1st check to see if session is already active


    if EngineSessions.LookupSession(From + toDN, task) then
    begin
      EnginePool[task].Engine.SMSEvent(From, toDN, Text, NumMedia, numSegments);
      DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'SMSEvent');
    end
    else // new
      if EnginePool.GetEngineFromPool(task) then
      begin
        EngineSessions.AddSession(From + toDN, task);
        EnginePool[task].IncomingSMS(toDN, From, Text, NumMedia, numSegments,
                                      aRequestInfo.Params.Values['MediaUrl0'],   // 1st Media URL for now
                                      aRequestInfo.Params.Values['MediaContentType0']);
        AResponseInfo.ContentText := NoResponse;
      end
      else
      begin
        DefaultLog.WriteTimeStamp(lctSMS, lptImportant, 'Unable to get an Engine from Pool');
        AResponseInfo.ContentText := TTwilioML.NoCapacitySMS;  // for get this
//        AResponseInfo.ContentText := 'Overloaded server';
//        AResponseInfo.ResponseNo := 503; // busy, overloaded, not available for now
      end;

    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;

 

 

  // Nexmo SMS
  if ((PathInfo = '/Nexmo/SMS') and (InParams <> '')) then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '**** Begining webmodule default action: path /Nexmo/SMS');
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams);

    From := Remove1(aRequestInfo.Params.Values['msisdn']);
    ToDN := Remove1(aRequestInfo.Params.Values['to']);
    Text := IndyTextEncoding_UTF8.GetString(ToBytes(aRequestInfo.Params.Values['text'], IndyTextEncoding_8Bit));

    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'From: ' + From + ' To: ' + ToDN);
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Text: ' +StringReplace(Text, #13#10, ' ', [rfReplaceAll]));

    // 1st check to see if session is already active
    // not needed for SMS at this time

    if EnginePool.GetEngineFromPool(task) then
    begin
      EngineSessions.AddSession(From + toDN, task);
      EnginePool[task].IncomingSMS(toDN, From, Text, 0, 1, '', ''); // no media stuff for nexmo

    end
    else
      DefaultLog.WriteTimeStamp(lctSMS, lptImportant, 'Unable to get an Engine from Pool');

    AResponseInfo.ResponseNo := 200;
    AResponseInfo.ContentText := 'OK';
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType

  end;

 

 

  // Nexmo Delivery Receipt
  if ((PathInfo = '/Nexmo/DLR') and (InParams <> '')) then
  begin
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Begining webmodule default action: path /Nexmo/DLR');
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams + #13#10);

    if (aRequestInfo.Params.Values['status'] <> 'delivered') then
      begin
        DefaultLog.WriteTimeStamp(lctSMS, lptImportant, 'Nexmo msg not delivered');
      end;

    AResponseInfo.ResponseNo := 200;
    AResponseInfo.ContentText := 'OK';
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;

 

 

  // Twilio Delivery Receipt
  if ((PathInfo = '/Status') and (InParams <> '')) then
  begin
   // DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams + #13#10);
    MsgStatus := ARequestInfo.Params.Values['MessageStatus'];
  //if ((MsgStatus <> 'sent') and (MsgStatus <> 'delivered')) then
    if (MsgStatus = 'delivered') then
    begin
      DefaultLog.WriteTimeStamp(lctSMS, lptNormal, '***** /Status    delivered'   );
      SMSSid := ARequestInfo.Params.Values['SMSSid'];
      if EngineSessions.LookupSession(SMSSid, task) then
      begin
        EnginePool[task].Engine.SignalSMSDelivery;
      end

    end;

    AResponseInfo.ContentText := NoResponse;
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;

 

 

  // Twilio CallStatus events
  if ((PathInfo = '/CallStatus') and (InParams <> '')) then
  begin

    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** /CallStatus ');
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams + #13#10);

    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid + '  sessions cnt:' + IntToStr(EngineSessions.Count) );
    CallStatus := aRequestInfo.Params.Values['CallStatus'];
    if (CallStatus = 'completed') or (CallStatus = 'canceled') then
    begin
      if EngineSessions.LookupSession(CallSid, task) then
      begin
        EngineSessions.RemoveSession(CallSid);
        EnginePool[task].Engine.SignalHangup;
        //send hangup event
      end
      else
         DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** /CallStatus: ' + 'did not find a session ');
    end
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '***** /CallStatus: ' + CallStatus + '   something other than completed/canceled??');

    AResponseInfo.ContentText := NoResponse;
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;


  // Twilio MakeCallStatus events
  // we send the engine# via the url, so no need to lookup session
  // however, in future, we may need to lookup session to get OTHER engines that are interested in this status
  if ((PathInfo = '/MakeCallStatus') and (InParams <> '')) then
  begin

    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, '***** /MakeCallStatus ');
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams + #13#10);

    CallSid := aRequestInfo.Params.Values['CallSid'];
    engineID := aRequestInfo.Params.Values['engineid'];
    task := StrToInt(engineID);
    CallStatus := aRequestInfo.Params.Values['CallStatus'];

    if (CallStatus = 'completed') then  // i.e. hungup after connected
    begin
      if EngineSessions.LookupSession(CallSid, task) then
      begin
        EngineSessions.RemoveSession(CallSid);
        EnginePool[task].Engine.SignalHangup;
        // send a hangup event
      end;

    end
    else   // busy, no-answer, canceled,  (possible that hangs up before connected, Red button on mobile) etc
      begin
        EnginePool[task].Engine.SignalMakeCallStatus(CallStatus);
      end;


    AResponseInfo.ContentText := NoResponse;
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;

 

 

  // Twilio calls this to get MakeCall TwiML, at this point we know MakeCall is connected
  if ((PathInfo = '/MakeCall') and (InParams <> '')) then
  begin

    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, '***** /MakeCall ');
    DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams + #13#10);

    CallSid := aRequestInfo.Params.Values['CallSid'];
    engineID := aRequestInfo.Params.Values['engineid']; // value that we sent Twilio via url, so that we know engine#
    task := StrToInt(engineID);                        // TODO: make sure engineid param is there
    EngineSessions.AddSession(CallSid, task);
    TwiMLResponse := NoResponse;

    if not(EnginePool[task].Engine.WaitingOnMakeCallTwiML(TwiMLResponse)) then
    begin
       DefaultLog.WriteTimeStamp(lctSMS, lptNormal, '***** /MakeCall,  Timed out waiting for TwiML, sending Pause ');
       AResponseInfo.ContentText := TTwilioML.PauseResponse;
    end
    else
      AResponseInfo.ContentText := TwiMLResponse;

    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType
  end;

 

 

  // Media Path
  if ((PathInfo = '/Media') or (PathInfo = '/MEDIA')) then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '****Begining MEDIA action: ');
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams);
    // get file or FILE value
    // file=balls.jpg  or file=sample.gif,  extension determines what ContentType to return
    id := ARequestInfo.Params.Values['file'];
    if id = '' then id := ARequestInfo.Params.Values['FILE'];

    ms := TMemoryStream.Create;

    if (id <> '') then
      ms.LoadFromFile(id);

    ms.Position := 0;

    AResponseInfo.ContentType := FileToContentType(id); // based on filename extension
    AResponseInfo.ContentStream := ms;

    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'File sent.' + #13#10);

    ms.Free;  // TODO: try/finally
  end;  // "/Media"


  // email Path
  if ((PathInfo = '/Email') or (PathInfo = '/EMAIL')) then
  begin
    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, '****Begining EMAIL action: ');
    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams);
    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, ARequestInfo.RemoteIP);
    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, ARequestInfo.UserAgent);
    ip := ARequestInfo.RemoteIP;

    SMS := TTwilioSMS.Create(cAccountSID, cTwilioPassword);
    sms.Log := DataModule1.Log;

    // get file or FILE value
    // file=balls.jpg  or file=sample.gif,  extension determines what ContentType to return
    id := ARequestInfo.Params.Values['info'];
    if id = '' then id := ARequestInfo.Params.Values['INFO'];
    tag := ARequestInfo.Params.Values['tag'];

    ms := TMemoryStream.Create;

    if (id <> '') then
      ms.LoadFromFile('balls.jpg');

    ms.Position := 0;

    AResponseInfo.CacheControl := 'no-cache';  // don't have browser cache this image
    AResponseInfo.ContentType := FileToContentType(id); // based on filename extension
    AResponseInfo.ContentStream := ms;

    text := tag + #13#10 + ip;
    sms.SendSMS('5146295764', '4385001040', text, '', false, numSegments, isMobile, isOptedOut, SMSSid);
    sms.Free;

    ms.Free;

    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, 'Picture sent.' + #13#10);
  end;


  // Phone Path
  if ((PathInfo = '/Phone') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /Phone');
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams);

    From := aRequestInfo.Params.Values['From'];
    privatecall := (From = '+266696687');  // Twilio sends this number when private


    From := RemovePlus1(From);
    if privatecall then From := 'PRIVATE';

    ToDN := RemovePlus1(aRequestInfo.Params.Values['To']);
    forwardedFrom := RemovePlus1(aRequestInfo.Params.Values['ForwardedFrom']);
    CallSid := aRequestInfo.Params.Values['CallSid'];

    FDisplayThread.Display(TimeToStr(Now) + ': Call  From: ' + From + '   To: ' + ToDN);
    if (forwardedFrom <> '') then FDisplayThread.Display('Forwarded from: ' + forwardedFrom);


    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Incoming Call From: ' + From + ' To: ' + ToDN
                         + '  forwarded from: ' + forwardedFrom);

    TwiMLResponse := NoResponse;  // in case we can't have an engine

    // ONLY new calls here
    if EnginePool.GetEngineFromPool(task) then
    begin
      if EngineSessions.AddSession(CallSid, task) then
        DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Added session: ' + CallSid + '  id: ' + IntToStr(task));
      TwiMLResponse := EnginePool[task].IncomingCall(ToDN, From, CallSid, forwardedFrom, privatecall);
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'TwiML(sent back to http server):  ' + TwiMLResponse);
      AResponseInfo.ContentText := TwiMLResponse;
    end
    else
    begin
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, 'Unable to get an Engine from Pool');
      AResponseInfo.ContentText := TTwilioML.NoCapacityCall;  // forget this
//      AResponseInfo.ContentText := 'Overloaded server';
//      AResponseInfo.ResponseNo := 503; // busy, overloaded, not available for now
    end;

    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';


  end; // "/Phone"


  // Redirect after a <Say> comes here
  if ((PathInfo = '/SayEnd') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /SayEnd');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalSayComplete
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/SayEnd:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;


  // Redirect after a <Play> comes here
  if ((PathInfo = '/PlayEnd') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /PlayEnd');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalPlayComplete
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/PlayEnd:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // Redirect after a <Call> comes here (non-action Dial version)
  // After coversation ends, busy, no answer or call failed
  if ((PathInfo = '/CallEnd') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /CallEnd');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalCallComplete
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/CallEnd:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // <Call> using action, provides DialCallStatus if we need OnNoAnswer, OnBusy
  // getting completed means Called Party hung up
  if ((PathInfo = '/DialCall') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /DialCall');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    DialCallSid := aRequestInfo.Params.Values['DialCallSid'];
    DialCallStatus := aRequestInfo.Params.Values['DialCallStatus'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'DialCallSid: ' + DialCallSid + '   DialCallStatus: ' + DialCallStatus);


    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
        EnginePool[task].Engine.SignalCall(DialCallStatus)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/DialCall:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;


  // Redirect after a <Gather> comes here
  if ((PathInfo = '/GatherTimeout') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /GatherTimeout');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalSpeech('xTimeoutx', '')  // we use string "xTimeoutx" to signal user timeout
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/GatherTimeout:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;


  // Redirect after a DTMF comes here
  if ((PathInfo = '/DTMFTimeout') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /DTMFTimeout');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalDTMF('xTimeoutx')  // we use string "xTimeoutx" to signal user timeout
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/DTMFTimeout:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // Received DTMF,  comes here
  if ((PathInfo = '/DTMF') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /DTMF');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    DigitsResult := aRequestInfo.Params.Values['Digits'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Digits: ' + DigitsResult);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalDTMF(DigitsResult)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/DTMF:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // Received speech, <Gather> comes here
  if ((PathInfo = '/Gather') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /Gather');
  //  DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'Params: ' + InParams);

    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);
    Confidence := aRequestInfo.Params.Values['Confidence'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'score: ' + Confidence);

    SpeechResult := aRequestInfo.Params.Values['speechResult'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'speechResult: ' + SpeechResult);


    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalSpeech(SpeechResult, Confidence)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/Gather:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;

  end;

 

 

  // Received speech Partial, <Gather> comes here
  if ((PathInfo = '/GatherPartial') and (InParams <> ''))  then
  begin
    CallSid := aRequestInfo.Params.Values['CallSid'];
    StableSpeechResult :=  ARequestInfo.Params.Values['StablespeechResult'];
    seqNumber :=  ARequestInfo.Params.Values['SequenceNumber'];
    UnStableSpeechResult := aRequestInfo.Params.Values['UnstableSpeechResult'];

    if (cbLogPartialSpeech.IsChecked) then
    begin
      DefaultLog.LineFeed;
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /GatherPartial');
   //   DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams);
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Stable speechResult: ' + StablespeechResult);
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Sequence Number: ' + seqNumber);
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Unstable speechResult: ' + UnstableSpeechResult);
    end;

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.maybeSignalSpeechEnd(UnstableSpeechResult, StableSpeechResult)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/GatherPartial:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.NoResponse;
  end;


  // action for <Pay> TwiML for <TwilioPay>
  if ((PathInfo = '/TwilioPay') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /TwilioPay');
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Params: ' + InParams);
    CallSid := aRequestInfo.Params.Values['CallSid'];
    TwilioPayResult := aRequestInfo.Params.Values['result'];
    TwilioPayError := aRequestInfo.Params.Values['PaymentError'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    //
    if Pos('insufficient funds', TwilioPayError) > 0 then
      TwilioPayResult := 'not-enough-funds';

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalTwilioPay(TwilioPayResult)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/TwilioPay:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // Redirect after a <Conference> comes here
  if ((PathInfo = '/Conference') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /Conference');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalConferenceEnd
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/Conference:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;


  // Received RecordingStatus,
  if ((PathInfo = '/RecordingStatus') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /RecordingStatus');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    RecordingSid := aRequestInfo.Params.Values['RecordingSid'];
    RecordingUrl := aRequestInfo.Params.Values['RecordingUrl'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'Recording: ' + RecordingSid + '     ' + RecordingUrl);

    //TODO:  maybe assign Sid and URL somewhere in Engine for this task, so that
    // we may in real-time play it back, store it etc.

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.RecordingComplete(RecordingSid)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '/RecordingStatus:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // url for ModifyCall API (if we are using the url instead of TwiML parameter)
  if ((PathInfo = '/ModifyCall') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /ModifyCall');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      TwiMLResponse := EnginePool[task].Engine.GetModifyCallTwiML
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/ModifyCall:  unable to find Call Session');

    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'TwiML: ' + TwiMLResponse);

    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TwiMLResponse;
  end;

 

 

  // NOTE: for a user to type into a browser
  if (PathInfoUser = '/WEB')   then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /Web');

    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := cWebformHTML;
  end;

  if ((PathInfo = '/webinput') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /webinput');
    AResponseInfo.ContentText := 'Unsuccessful. Try again.';

    num := aRequestInfo.Params.Values['num'];
    address := aRequestInfo.Params.Values['address'];
    num := Trim(num); address := Trim(address);
    if EngineSessions.LookupSession(num, task) then
    begin
      EnginePool[task].Engine.HTTPEvent(num,address);
      DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'HTTPEvent');
      AResponseInfo.ContentText := 'Successful. Thank You.';
    end;


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
  end;


  // Twilio's <Enqueue> action="" comes here
  if ((PathInfo = '/EnqueueAction') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /EnqueueAction');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    QueueResult := aRequestInfo.Params.Values['QueueResult'];
    QueueTime := aRequestInfo.Params.Values['QueueTime'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid + ' QueueResult: ' + QueueResult);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalEnqueue(QueueResult, QueueTime)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/EnqueueAction:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

 

  // Twilio's <Dial> ... <Queue url=""  comes here
  // This is where we send back TwiML to caller waiting before connected
  if ((PathInfo = '/QueueAction') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /QueueAction');
    DequeingCallSid := aRequestInfo.Params.Values['DequeingCallSid']; // <== this caller
    CallSid := aRequestInfo.Params.Values['CallSid'];      // <=== caller that is waiting on hold, this caller will connect to
    QueueTime := aRequestInfo.Params.Values['QueueTime'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'this CallSid: ' + DequeingCallSid + ' other CallSid: + ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(DequeingCallSid, task) then
      EnginePool[task].Engine.SignalDequeueConnected(QueueTime, CallSid)
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/QueueAction:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.EnqueueStdResponse;
  end;


  // Redirect for our <Queue>
  if ((PathInfo = '/Queue') and (InParams <> ''))  then
  begin
    DefaultLog.LineFeed;
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, '***** Begining webmodule default action: /Queue');
    CallSid := aRequestInfo.Params.Values['CallSid'];
    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'CallSid: ' + CallSid);

    // lookup session and deliver currentEvent
    if EngineSessions.LookupSession(CallSid, task) then
      EnginePool[task].Engine.SignalDequeueTimeout
    else
      DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, '/Queue:  unable to find Call Session');


    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.PauseResponse;
  end;

 

except on E:Exception do
  begin
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.NoResponse;
    DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, 'Exception occurred in the HTTP server GetCommand');
  end;
end;

end;
 

 

 

Share this post


Link to post
28 minutes ago, Joe Sansalone said:

The application (HTTP Server) stopped responding to requests again.

In what way?  Did you verify that ALL of the server events stopped (OnConnect, OnHeadersAvailable, etc)?  Or is it just the OnCommandGet event?  Do you have a capture of the requests that don't respond?

28 minutes ago, Joe Sansalone said:

This time, I simply "Stopped" and "Started" the TIdHTTPServer without restarting the application and it worked - application started 

responding to requests.

Then it can't be a deadlock issue, since the server is still responsive to shutdown and restart, and thus the socket threads are not blocking.  So there has to be an issue in the request handling.  But there is WAY too much code provided with no explanation or context, so it is very difficult to diagnose your issue.

 

Your OnCommandGet handler is pretty massive (also, there are a lot of places where Exit statements are missing that would avoid unnecessary processing).  Have you considered breaking up the code and then unit-testing the pieces individually?

28 minutes ago, Joe Sansalone said:

It's just a bunch of "If then begin end" to handle each Pathinfo differently.

 

And all of it is wrapped in a Try/Except

That MIGHT be part of the problem.  You really should not be catching unhandled exceptions, but if you do, make sure to RE-RAISE any Indy exceptions you happen to catch (they all derive from EIdException).  The server needs to handle those internally in order to close the socket and stop its owning thread (unless you close the socket manually in your except block).

28 minutes ago, Joe Sansalone said:

Text := IndyTextEncoding_UTF8.GetString(ToBytes(aRequestInfo.Params.Values['Body'], IndyTextEncoding_8Bit));

Why are you doing that?  The Params have already been decoded before OnCommandGet is called (assuming TIdHTTPServer.ParseParams is true, which it is by default), and UTF-8 is the default charset used for that decode (unless the request specifies a charset in its Content-Type header).  Why can't you use the Body text as-is?

Text := aRequestInfo.Params.Values['Body'];

Since the text was already pre-decoded, re-encoding it and decoding it as a potentially different charset is just asking for trouble.  This is a good way to corrupt the text.

 

If you absolutely need custom decoding, then you should instead parse the ARequestInfo.UnparsedParams, ARequestInfo.QueryParams, and/or ARequestInfo.FormParams properties as needed.  That is the raw data where TIdHTTPServer decodes the ARequestInfo.Params data from.

28 minutes ago, Joe Sansalone said:

    if EngineSessions.LookupSession(From + toDN, task) then

    begin
      EnginePool[task].Engine.SMSEvent(From, toDN, Text, NumMedia, numSegments);
      DefaultLog.WriteTimeStamp(lctSMS, lptNormal, 'SMSEvent');
    end
    else // new
      if EnginePool.GetEngineFromPool(task) then
      begin
        EngineSessions.AddSession(From + toDN, task);
        EnginePool[task].IncomingSMS(toDN, From, Text, NumMedia, numSegments,
                                      aRequestInfo.Params.Values['MediaUrl0'],   // 1st Media URL for now
                                      aRequestInfo.Params.Values['MediaContentType0']);
        AResponseInfo.ContentText := NoResponse;
      end
      else
      begin
        DefaultLog.WriteTimeStamp(lctSMS, lptImportant, 'Unable to get an Engine from Pool');
        AResponseInfo.ContentText := TTwilioML.NoCapacitySMS;  // for get this
//        AResponseInfo.ContentText := 'Overloaded server';
//        AResponseInfo.ResponseNo := 503; // busy, overloaded, not available for now
      end;

Are you sure all of your Engine code is thread-safe and non-blocking?

28 minutes ago, Joe Sansalone said:

 AResponseInfo.ContentType := 'text/html;charset="UTF-8"';  // IMPORTANT to have no spaces in ContentType

What is that supposed to mean?  You can absolutely have a space after the ';' character, that will work just fine:

AResponseInfo.ContentType := 'text/html; charset="UTF-8"';

Alternatively, simply use the separate AResponseInfo.CharSet property instead:

AResponseInfo.ContentType := 'text/html';
AResponseInfo.CharSet := 'UTF-8';
28 minutes ago, Joe Sansalone said:

    ms := TMemoryStream.Create;

    if (id <> '') then
      ms.LoadFromFile(id);

    ms.Position := 0;

    AResponseInfo.ContentType := FileToContentType(id); // based on filename extension
    AResponseInfo.ContentStream := ms;

    DefaultLog.WriteTimeStamp(lctHTTP, lptNormal, 'File sent.' + #13#10);

    ms.Free;  // TODO: try/finally

This is wrong.  One, because the LoadFromFile() is a potential exception waiting to happen, since you appear to be loading files using relative paths. ALWAYS use absolute paths instead!  But more importantly, you are freeing the TMemoryStream that is assigned to the AResponseInfo.ContentStream property.  Indy DOES NOT make a copy of that stream, so you are leaving behind a dangling pointer that will crash when the server tries to write the stream data to the client after the OnCommandGet handler exits.  The AResponseInfo object will free the ContentStream for you when it is no longer needed, so DO NOT free it manually.  Unless you really need to free the stream manually (which you don't in this cae), in which case you would have to call AResponseInfo.WriteHeader() and AResponseInfo.WriteContent() manually before freeing the stream.

28 minutes ago, Joe Sansalone said:

  // email Path
  if ((PathInfo = '/Email') or (PathInfo = '/EMAIL')) then

You should consider using Indy's TextIsSame() function for case-insensitive string comparisons.

28 minutes ago, Joe Sansalone said:

    ms := TMemoryStream.Create;

    if (id <> '') then
      ms.LoadFromFile('balls.jpg');

    ms.Position := 0;

    AResponseInfo.CacheControl := 'no-cache';  // don't have browser cache this image
    AResponseInfo.ContentType := FileToContentType(id); // based on filename extension
    AResponseInfo.ContentStream := ms;

    text := tag + #13#10 + ip;
    sms.SendSMS('5146295764', '4385001040', text, '', false, numSegments, isMobile, isOptedOut, SMSSid);
    sms.Free;

    ms.Free;

    DataModule1.Log.WriteTimeStamp(lctHTTP, lptNormal, 'Picture sent.' + #13#10);

Same here.  Also, is the SMS and Twilio code thread-safe and non-blocking?

28 minutes ago, Joe Sansalone said:

    if (cbLogPartialSpeech.IsChecked) then

This is NOT thread-safe!  You CANNOT access UI controls from outside of the context of the main UI thread.  The OnCommandGet event is fired in the context of a worker thread, so you MUST synchronize with the main UI thread here.  Otherwise, I would suggest storing the CheckBox value in a global Boolean variable at server startup, and then you can read that variable without synchronizing it (as long as you don't alter the vaiable's value while the server is running, otherwise you do need to synchronize it).

28 minutes ago, Joe Sansalone said:

except on E:Exception do

  begin
    AResponseInfo.ContentType := 'text/html;charset="UTF-8"';
    AResponseInfo.ContentText := TTwilioML.NoResponse;
    DefaultLog.WriteTimeStamp(lctHTTP, lptImportant, 'Exception occurred in the HTTP server GetCommand');
  end;
end;

If your OnCommandGet code encounters an unhandled exception, you are catching it to send a response to the client, but you are not setting an appropriate ResponseCode, like 500.  So the default 200 will likely be sent instead.  Is that what you really want?  By default, TIdHTTPServer already sends a 500 response to the client for unhandled exceptions, so this code is largely unnecessary.  But since you are customizing the ContextText, then at the very least I would suggest getting rid of the outer try/except altogether and move this code into the OnCommandError event instead.

Share this post


Link to post
36 minutes ago, Remy Lebeau said:

In what way?  Did you verify that ALL of the server events stopped (OnConnect, OnHeadersAvailable, etc)?  Or is it just the OnCommandGet event?  Do you have a capture of the requests that don't respond?

Unfortunately, HTTP logging was off .. I'll capture on the next time.

 

37 minutes ago, Remy Lebeau said:

Your OnCommandGet handler is pretty massive (also, there are a lot of places where Exit statements are missing that would avoid unnecessary processing).  Have you considered breaking up the code and then unit-testing the pieces individually?

That MIGHT be part of the problem.  You really should not be catching unhandled exceptions, but if you do, make sure to RE-RAISE any Indy exceptions you happen to catch (they all derive from EIdException).  The server needs to handle those internally in order to close the socket and stop its owning thread (unless you close the socket manually in your except block).

 

Ok, I won't catch unhandled exceptions.  So far, there has never been a log to indicate any.  And logging for this is always on.

 

 

47 minutes ago, Remy Lebeau said:

 Why can't you use the Body text as-is?


Text := aRequestInfo.Params.Values['Body'];

 

Twilio sends the data in SMS texts with emoticons, emojis etc.  Using aRequestInfo.Params.Values['Body'] doesn't work.

 

49 minutes ago, Remy Lebeau said:

 

Are you sure all of your Engine code is thread-safe and non-blocking?

 

 

It is thread-safe.  However, it does block for some calls with a maximum timeout so it returns for sure.

 

Is there a timeout where it's too much for TIdHTTPServer in terms of waiting for CommandGet event?

I can change some of the blocking timeouts to be lower.

 

54 minutes ago, Remy Lebeau said:

 But more importantly, you are freeing the TMemoryStream that is assigned to the AResponseInfo.ContentStream property.  Indy DOES NOT make a copy of that stream, so you are leaving behind a dangling pointer that will crash when the server tries to write the stream data to the client after the OnCommandGet handler exits.  The AResponseInfo object will free the ContentStream for you when it is no longer needed, so DO NOT free it manually.  

Oops!  I will fix that.

 

 

56 minutes ago, Remy Lebeau said:

 if (cbLogPartialSpeech.IsChecked) then

 

This is NOT thread-safe!  You CANNOT access UI controls from outside of the context of the main UI thread.  The OnCommandGet event is fired in the context of a worker thread, so you MUST synchronize with the main UI thread here.  Otherwise, I would suggest storing the CheckBox value in a global Boolean variable at server startup, and then you can read that variable without synchronizing it (as long as you don't alter the vaiable's value while the server is running, otherwise you do need to synchronize it).

I'm aware that UI controls need to be accessed within the main UI thread.

In this specific case, I verified that cbLogPartialSpeech.IsChecked really only reads a boolean (nothing else)

and I need to change it sometimes while the server is running .. so I took the liberty to do it this way.

 

1 hour ago, Remy Lebeau said:

If your OnCommandGet code encounters an unhandled exception, you are catching it to send a response to the client, but you are not setting an appropriate ResponseCode, like 500.  So the default 200 will likely be sent instead.  Is that what you really want?  By default, TIdHTTPServer already sends a 500 response to the client for unhandled exceptions, so this code is largely unnecessary.  But since you are customizing the ContextText, then at the very least I would suggest getting rid of the outer try/except altogether and move this code into the OnCommandError event instead.

In the case of using Twilio, I need to send something to tell Twilio what to do with the "Call" or "SMS" - something that won't abruptly end the call.

Thus, I send something in their XML-like language to tell them not to drop the call etc.  Sending a 500 response will evoke their default behavior to thrash the

existing call.  We prefer otherwise.  But we may change this ... I understand your concern.

 

 

Thanks for the help - much appreciated!

 

Sorry for the massive code.  So much happens in the engine that the 

HTTP event code was left to grow without possibly making it more readable.

It's the reason I simply made each section an IF then for each PathInfo, 

without using else's or it would really be hard to read.

 

 

 

Share this post


Link to post

I forgot to mention that each section in the CommendGet event has

been tested quite a bit with no known problems.

 

I'll have more HTTP logs the next time.  

 

Just realized that even though Checkbox.IsChecked simply accesses 

a boolean and does nothing else, it's possible that Windows OS may 

require that only the UI thread does this?? Although I thought it 

was because of thread-safety and avoiding weird collisions of updating the UI.

 

In any case I have changed that code.   

Edited by Joe Sansalone
forgot something

Share this post


Link to post
1 hour ago, Joe Sansalone said:

Twilio sends the data in SMS texts with emoticons, emojis etc.  Using aRequestInfo.Params.Values['Body'] doesn't work.

Can you provide an example?  And is the data coming from the URL querystring, or the request body? I assume the latter.  It makes a difference to how HTTP behaves.

 

The code you showed using UTF8.GetString() only makes sense if TIdHTTPServer were storing UTF-8 bytes as Chars in a String, which it should not be doing unless the request specifies a charset (in the Content-Type header) which Indy does not recognize/support.  Otherwise, it should be using that charset properly, or if there is no charset specified then it uses UTF-8 (assuming you are using an up-to-date version of Indy).  So, it is a little hard to imagine that if Twilio is sending UTF-8 data for the 'Body' text that TIdHTTPServer would not be decoding it properly by default.  But, that goes back to my earlier comment about needing to see the raw HTTP requests.

1 hour ago, Joe Sansalone said:

Is there a timeout where it's too much for TIdHTTPServer in terms of waiting for CommandGet event?

No.  It will happily wait as long as it takes to read in the request data and send out the response data.

Share this post


Link to post

NOTE:  Twilio server receives SMSs sent to a phone number that resides in their network.  They then send an HTTP request to my

            server indicating that they received an SMS and they put the contents of the SMS in the Body parameter.  

            

 

Here is an example of receiving an SMS (i.e. text on a cellular phone) Hello followed by 2 different smiley face emojis, received by Twilio and sent

as a request to my server:

 

Unparsed Param:   Body=Hello+%F0%9F%98%85%F0%9F%A4%A3

Parsed Param:  Body=Hello ð???ð?¤£

 

This is perfectly fine.  

 

But, I found that then I need to do this when receiving:

 

Text := IndyTextEncoding_UTF8.GetString(ToBytes(aRequestInfo.Params.Values['Body'], IndyTextEncoding_8Bit));

 

 in order to SEND the same SMS with the same Hello and 2 smiley faces using the REST components:

 

FRestRequest.Params.AddItem('Body',TIDURI.ParamsEncode(aText, IndyTextEncoding_UTF8) , TRestRequestParameterKind.pkGETorPOST,
                              [poDoNotEncode]);

 

 

 

If I code it the regular way, it doesn't work. 

It sends the outgoing SMS with Hello .. but the 2 smiley faces are weird characters instead of 2 emojis.

 

 

 

 

 

 

 

 

Share this post


Link to post
On 8/17/2021 at 3:47 PM, Joe Sansalone said:

Unparsed Param:   Body=Hello+%F0%9F%98%85%F0%9F%A4%A3

Parsed Param:  Body=Hello ð???ð?¤£

It SHOULD NOT be decoding the unparsed param like that.  The encoded hex shown is valid UTF-8, and I can decode it manually into the two Emojis (U+1F605 and U+1F923).  But the parsed param you have shown is the result of Indy storing the raw UTF-8 bytes into the String rather than decoding the UTF-8 bytes into Unicode text.

 

Like I said earlier, TIdHTTPServer decodes the params using UTF-8 by default, unless overridden by an explicit charset in the request.

 

Oh, wait... (checks Indy's change history...) that was actually a recent fix made earlier this year.  Prior to that fix, Indy would indeed decode the Param data using 8bit instead of UTF-8, unless the request explicitly stated UTF-8.  So that would explain the behavior you are seeing, and why the IndyTextEncoding_UTF8.GetString() workaround was needed.

 

You should update to the latest version of Indy.  Then the param data should be decoded properly as UTF-8 by default and you won't need the IndyTextEncoding_UTF8.GetString() workaround anymore.

Quote

This is perfectly fine. 

No, it is not fine.  It is broken, actually.  But it was fix several months ago.

Quote

But, I found that then I need to do this when receiving

Yes, but you shouldn't have had to resort to that if Indy were decoding the param data correctly to begin with.  Which clearly your version is not.

Edited by Remy Lebeau

Share this post


Link to post

Ok got it.   

 

When I update to latest Indy version (either with next Delphi 11 or manually),

I will make sure to change my code.

 

Thanks!

Share this post


Link to post

I got the PROBLEM again - HTTP Server stops responding.

 

This time I logged OnConnect, OnHeadersAvailable, OnException, OnCommandError events as well as

the usual OnCommandGet, OnCommandOther.

 

There was NO activity in the above events when the problem occurred.

 

Below is the log from Twilio's server which sends a request to my server (live.projectone.ca) :

 

========================================================================================

 

MESSAGE
An attempt to retrieve content from https://live.projectone.ca/Phone returned the HTTP status code 502
SSL/TLS Handshake Error

Error - 11220

SSL/TLS Handshake Error

During SSL/TLS negotiation, Twilio experienced a connection reset.

Possible Causes

  • Incompatible cipher suites in use by the client and the server. This would require the client to use (or enable) a cipher suite that is supported by the server.

Possible Solutions

  • Verify cipher suites in-use are up to date. Twilio-supported ciphers can be found here
  • Use compatible version of TLS, Twilio supports TLS 1.0,1.1,1.2.

============================================================================================

 

 

After closing/starting the application again, everything was fine.

 

Is it possible that there's a newer version of Indy that fixes this problem?

 

I'm thinking of running without SSL to see if this problem occurs.

 

Joe

 

 

 

 

 

 

Share this post


Link to post
6 hours ago, Joe Sansalone said:

I got the PROBLEM again - HTTP Server stops responding.

Did you do what Twilio said to do? Did you verify the TLS version and ciphers being used by your server during a failed connection?  Have you tried sniffing the TLS handshake with Wireshark to see what is actually going on inside of it?

Quote

During SSL/TLS negotiation, Twilio experienced a connection reset.

The only time I have ever seen that happen is when OpenSSL decides to reject a handshake and just closes the TCP connection abruptly without first sending a TLS alert to explain why it is being rejected. That usually implies that something in the handshake is corrupted or incorrect, and OpenSSL is just being cautious about sending anything, but that is not always the case.

Quote

Is it possible that there's a newer version of Indy that fixes this problem?

I doubt it, since this is not really an Indy issue, it is more of an OpenSSL TLS issue, but whether it is on the client side or the server side is still TBD.

Edited by Remy Lebeau

Share this post


Link to post

Why does it work for days and then it doesn't?

Strange.  It's the same instance.  I don't change anything.

 

I'll install Wireshark on the server.

 

Thanks for your help.

 

 

 

 

Share this post


Link to post
1 hour ago, Joe Sansalone said:

Why does it work for days and then it doesn't?

I can't answer that.  There simply has not been enough information provided yet to diagnose the root cause.  You are just going to have to continue debugging deeper.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×