How to log raw data on Android?

[ertank 28]

Hello,

 

I am asked for some raw data including header and body between a rest server and android mobile app.

 

My search on internet lead me to use TIdLogEvent (I am not sure that is the right component for job. Since I do communication in a thread, I select to create everything using code at run-time.

 

I have following procedures to be assigned at run-time

procedure TfrmMain.IdLogEvent1Received(ASender: TComponent; const AText,
  AData: string);
var
  Filename: string;
begin
  Filename := TPath.Combine(TPath.GetSharedDocumentsPath(), 'response_');
  Filename := Filename + FormatDateTime('yyyy-mm-dd_hh.nn.ss', Now()) + '.log';
  TFile.WriteAllText(Filename, AData);
end;

procedure TfrmMain.IdLogEvent1Sent(ASender: TComponent; const AText,
  AData: string);
var
  Filename: string;
begin
  Filename := TPath.Combine(TPath.GetSharedDocumentsPath(), 'request_');
  Filename := Filename + FormatDateTime('yyyy-mm-dd_hh.nn.ss', Now()) + '.log';
  TFile.WriteAllText(Filename, AData);
end;

 

I am trying to create TIdHTTP and others in a thread as following:

  var
    NewsRec: TGetNewsRequest;
    Json: string;
    SS: TStringStream;
    Http: TIdHTTP;
    IOHandler: TIdIOHandlerStack;
    LogEvent: TIdLogEvent;
    Response: string;
    ResponseRec: TGetNewsResponse;
  begin
    // some non-relevant code for json preperation, etc. deleted
    
    Http := nil;
    IOHandler := nil;
    SS := nil;
    LogEvent := nil;
    try
      SS := TStringStream.Create(Json);
      LogEvent := TIdLogEvent.Create(nil);
      LogEvent.OnSent := frmMain.IdLogEvent1Sent;
      LogEvent.OnReceived := frmMain.IdLogEvent1Received;
      Http := TIdHTTP.Create(nil);
      IOHandler := TIdIOHandlerStack.Create(Http);
      Http.IOHandler := IOHandler;
      Http.IOHandler.Intercept := LogEvent;
      Http.Request.ContentType := 'application/x-www-form-urlencoded';
      Response := Http.Post(GetNewsURL, SS); // try..except removed to simplify code
    finally
      LogEvent.Free();
      Http.Free();
      SS.Free();
    end;

  // some non-relevant code to deal with result deleted
  end;

Communication is not secured. It is plain "http". I could not figure what I am doing wrong as I always get Abstract error. That stands true no matter I assign to TIdHTTP.IOHandler be it TIdIOHandler, TIdIOHandlerSocket, TIdIOHandlerStack. I cannot get past to that error and save raw communication dumped in log files.

 

Any help is appreciated.

 

Thank & regards,

Ertan

[mjustin 23]

There is a specific implementation for writing Indy raw data to a log file, TIdInterceptSimLog in unit IdInterceptSimLog.

 

Intercept := TIdInterceptSimLog.Create;
Intercept.Filename := (create the file name);

Your implementation uses the abstract class TIdLogEent, which is an abstract class, because it does not implement the abstract methods of its parent class TIdLogBase,
 

 

[Remy Lebeau 1436]

2 hours ago, ertank said:

I am asked for some raw data including header and body between a rest server and android mobile app.

 

My search on internet lead me to use TIdLogEvent

Why not TIdLogFile instead?  Especially since you are saving data to files.  For debugging purposes, you DON'T want individual reads/writes to be logged to separate files.  It makes more sense to keep them in a single file.

 

Even if you use TIdLogEvent with separate files for input and output, you are creating filenames based on timestamps with seconds precision, so you are likely to end up with multiple files that contain smaller pieces of the overall data.  At the very least, I would suggest either dropping the seconds from the filename, or use the timestamp at the very beginning of the initial request rather than the timestamps of the individual reads/writes.

 

Also FYI, TFile.WriteAllText() overwrites an existing file.  You need TFile.AppendAllText() instead.

Quote

Communication is not secured. It is plain "http".

Even if it were secured with HTTPS, the TIdLog... components would be able to log the unencrypted HTTP data, as they log outgoing data before it is encrypted, and log incoming data after it is decrypted.

Quote

I could not figure what I am doing wrong as I always get Abstract error.

I see nothing in your code that would cause an Abstract error, as there are no abstract classes being used.

 

That being said, you don't need to instantiate an IOHandler manually in this case, since you are not using HTTPS.  TIdHTTP has its own Intercept property, and will create an IOHandler internally for you:

 

var
  NewsRec: TGetNewsRequest;
  Json: string;
  SS: TStringStream;
  Http: TIdHTTP;
  LogEvent: TIdLogEvent;
  Response: string;
  ResponseRec: TGetNewsResponse;
begin
  // some non-relevant code for json preperation, etc. deleted
    
  SS := TStringStream.Create(Json);
  try
    LogEvent := TIdLogEvent.Create(nil);
    try
      LogEvent.OnSent := frmMain.IdLogEvent1Sent;
      LogEvent.OnReceived := frmMain.IdLogEvent1Received;
      Http := TIdHTTP.Create(nil);
      try
        Http.Intercept := LogEvent;
        Http.Request.ContentType := 'application/x-www-form-urlencoded';
        Response := Http.Post(GetNewsURL, SS); // try..except removed to simplify code
      finally
        Http.Free;
      end;
    finally
      LogEvent.Free;
    end;
  finally
    SS.Free;
  end;

  // some non-relevant code to deal with result deleted
end;

On a side note, 'application/x-www-form-urlencoded' is the wrong ContentType to use for JSON data, use 'application/json' instead.  If you really want to send an 'application/x-www-form-urlencoded' post, you need to use the overloaded version of TIdHTTP.Post() that takes a TStrings as input, not a TStream.

Edited June 5, 2019 by Remy Lebeau

[Remy Lebeau 1436]

2 hours ago, mjustin said:

There is a specific implementation for writing Indy raw data to a log file, TIdInterceptSimLog in unit IdInterceptSimLog.

Intercept := TIdInterceptSimLog.Create;
Intercept.Filename := (create the file name);

Your implementation uses the abstract class TIdLogEent, which is an abstract class, because it does not implement the abstract methods of its parent class TIdLogBase,

TIdLogEvent is not an abstract class, as it does implement all of the abstract methods of TIdLogBase.

 

There is also TIdLogFile, which is not abstract, either.

 

TIdInterceptSimLog and TIdLogFile are completely different file loggers.  They write data in different formats.  TIdLogFile is more "raw", logging data as-is, with optional timestamp and text replacements of CR and LF bytes.  While TIdInterceptSimLog prefixes each read/write with a summary line describes whether the data is textual with a EOL present, or is raw bytes and what the byte size is, and appends a line break after each data.

Edited June 5, 2019 by Remy Lebeau

[ertank 28]

12 minutes ago, Remy Lebeau said:

Why not TIdLogFile instead?  Especially since you are saving data to files.  For debugging purposes, you DON'T want individual reads/writes to be logged to separate files.  It makes more sense to keep them in a single file.

That is the first time I am dealing with raw logging using Indy. I will have a look at it if they are to stay in the app for long. All this I am trying to log is only for checking some headers for web service developers.

14 minutes ago, Remy Lebeau said:

Even if you use TIdLogEvent with separate files for input and output, you are creating filenames based on timestamps with seconds precision, so you are likely to end up with multiple files that contain smaller pieces of the overall data.  At the very least, I would suggest either dropping the seconds from the filename, or use the timestamp at the very beginning of the initial request rather than the timestamps of the individual reads/writes.

Thanks for the warning. There are not many calls to web service done. Logs needed after something like button click or similar.

25 minutes ago, Remy Lebeau said:

Even if it were secured with HTTPS, the TIdLog... components would be able to log the unencrypted HTTP data, as they log outgoing data before it is encrypted, and log incoming data after it is decrypted.

I see nothing in your code that would cause an Abstract error, as there are no abstract classes being used.

 

That being said, you don't need to instantiate an IOHandler manually in this case, since you are not using HTTPS.  TIdHTTP has its own Intercept property, and will create an IOHandler internally for you:

 

On a side note, 'application/x-www-form-urlencoded' is the wrong ContentType to use for JSON data, use 'application/json' instead.  If you really want to send an 'application/x-www-form-urlencoded' post, you need to use the overloaded version of TIdHTTP.Post() that takes a TStrings as input, not a TStream.

I received that abstract error maybe because I tried to use TIdHTTP.IOHandler.Intercept and not TIdHTTP.Intercept? This is something I read in one of the Indy 10.0.0.50 version or similar post that I found. I did not understand why abstract error received and so I wrote in here thinking there maybe changes from the time of that posts to Indy 10.6.2.xxxx

 

Now, project compile and run fine using your sample code. There is no abstract error. However, unseen part of the code is that I am always receiving "Connection closed gracefully" exception on debug run in thread. There are no files saved, too. When debugging execution does not come in assigned TIdLogEvent.OnSent() or TIdLogEvent.Received() functions.

 

I could not understand why there is such an exception. Web service is built with PHP. Laravel v5.8 or something used as I am told.

 

I do not know PHP and web service developers cannot answer most of my questions, unfortunately. That ContentType is also enforced by that PHP developers to me. I started using that TStrings overloaded function at the beginning. That did not work with that web services. Framework exception or no communication happens. Above code could communicate. But, it does not log raw communication data.

[Remy Lebeau 1436]

3 minutes ago, ertank said:

Thanks for the warning. There are not many calls to web service done. Logs needed after something like button click or similar.

That is not what I was referring to.  It takes multiple writes to the socket to send a web request, and multiple reads to receive a complete response.  Those writes and reads get logged individually by Indy's Intercept components.  They are not going to log a complete request in one go, or a complete response in one go.

3 minutes ago, ertank said:

I received that abstract error maybe because I tried to use TIdHTTP.IOHandler.Intercept and not TIdHTTP.Intercept?

No, that is not the cause of the error.  And besides, the TIdHTTP.Intercept gets assigned to the IOHandler.Intercept internally.  So assigning the TIdHTTP.Intercept is just a shortcut.

3 minutes ago, ertank said:

This is something I read in one of the Indy 10.0.0.50 version or similar post that I found. I did not understand why abstract error received and so I wrote in here thinking there maybe changes from the time of that posts to Indy 10.6.2.xxxx

Make sure you DO NOT have multiple conflicting versions of Indy on your machine.  If you do, then it is likely that the compiler is finding an old file that does not match the latest interface.

3 minutes ago, ertank said:

Now, project compile and run fine using your sample code. There is no abstract error. However, unseen part of the code is that I am always receiving "Connection closed gracefully" exception on debug run in thread.

That is perfectly fine, especially if you are only seeing it in the debugger.  It just means the web server closed the connection on its end while TIdHTTP was still reading from the socket.  Which is one possible way (not the preferred way, but still possible) for a web server to signal the end of a response.  TIdHTTP should be handled such an error internally.  You can configure your debugger to ignore that particular exception.

3 minutes ago, ertank said:

There are no files saved, too. When debugging execution does not come in assigned TIdLogEvent.OnSent() or TIdLogEvent.Received() functions.

Make sure you set the Active property of the TIdLog... component to True.

3 minutes ago, ertank said:

I do not know PHP and web service developers cannot answer most of my questions, unfortunately. That ContentType is also enforced by that PHP developers to me.

Then those PHP devs don't really understand what they are doing. You can't post JSON using 'application/x-www.form-urlencoded'. If they are enforcing that, they are doing something wrong on their end.

3 minutes ago, ertank said:

I started using that TStrings overloaded function at the beginning. That did not work with that web services. Framework exception or no communication happens.

Makes sense, since you can't post JSON using 'application/x-www-form-urlencoded'.

 

[ertank 28]

7 minutes ago, Remy Lebeau said:

That is not what I was referring to.  It takes multiple writes to the socket to send a web request, and multiple reads to receive a complete response.  Those writes and reads get logged individually by Indy's Intercept components.  They are not going to log a complete request in one go, or a complete response in one go.

Yes. It is clear now. Converted to TFile.AppendText and changed filename not to include seconds.

 

10 minutes ago, Remy Lebeau said:

Make sure you DO NOT have multiple conflicting versions of Indy on your machine.  If you do, then it is likely that the compiler is finding an old file that does not match the latest interface.

I most of the time do not change stock Indy version if there is a reason to. I am using Delphi 10.3.1 installed Indy on my system and there is no version mixing or similar.

 

8 minutes ago, Remy Lebeau said:

Make sure you set the Active property of the TIdLog... component to True.

That was missing. Now logging is working fine.

 

Thanks.