TIDMessage Terminator Sequence

[Mark Williams 14]

Quote

If you are trying to import .EML message files from MS Outlook Express, or another email client, make sure that the text file contains the message terminator sequence.

The above is a quote from the Indy help file re TIDMessage LoadFromFile. I have tested loading .eml files with TIDMessage none of which contain the specified terminator sequence "CRLF.CRLF". They load without problem. I can see online that there are quite a few posts dealing with this, but I am wondering if it is still an issue for the latest versions of INDY. If so, how do you deal with it. I have tried adding the terminator sequence to emails before reading them and IDMessage fails to read the data.

 

I have tried with the following code:

var
    strStream : TStringStream;
begin
  strStream:=TStringStream.Create;
  try
    strStream.LoadFromFile(FileName);
    strStream.seek(0, soEnd);
    strStream.writeString(sLineBreak+'.'+sLineBreak);   
    strStream.position := 0;
    IDMessage1.LoadFromStream(strStream);
 finally
	strStream.Free;
 end;

The email loads as expected, but my questions are: (1) is this is actually necessary and, if so,  (2) is this the right way to go about it?

 

I am dubious because if this was needed in every case to ensure that the email opens then I would have thought it would have been incorporated in TIDmessage's Load methods.

[Remy Lebeau 1421]

7 hours ago, Mark Williams said:

The above is a quote from the Indy help file re TIDMessage LoadFromFile.

The help file (and the documentation in general) is old.  It hasn't been updated in a very long time.

Quote

I can see online that there are quite a few posts dealing with this, but I am wondering if it is still an issue for the latest versions of INDY. If so, how do you deal with it.

It depends on how the EML file is formatted.  EML files created outside of Indy typically DO NOT escape leading periods on lines in the message body, which TIdMessage.LoadFromFile() requires, because it expects the EML file to have been created by TIdMessage.SaveToFile(), which does escape leading periods.  To address this, you can add the IdMessageHelper.pas unit to your 'uses' clause.  It introduces new SaveTo...() and LoadFrom...() methods into TIdMessage (in versions of Delphi that support class helpers), as well as standalone TIdMessageHelper_SaveTo...() and TIdMessageHelper_LoadFrom...() procedures, which have an additional AUsesDotTransparency parameter added that can be set to False when needed.  For example:

uses
  ..., IdMessageHelper;

//IDMessage1.LoadFromFile(FileName, False);
IDMessage1.LoadFromFile(FileName, False, False);

// or: TIdMessageHelper_LoadFromFile(IDMessage1, FileName, False, False);

Quote

I have tried adding the terminator sequence to emails before reading them and IDMessage fails to read the data.

In what way does it fail?

Quote

is this is actually necessary

No.

Quote

is this the right way to go about it?

No.

Quote

I am dubious because if this was needed in every case to ensure that the email opens then I would have thought it would have been incorporated in TIDmessage's Load methods.

It is a bit more complicated than that, because TIdMessage currently uses TIdMessageClient internally to parse the EML, but TIdMessageClient is not designed for parsing emails from files, it is designed for parsing emails over protocols like SMTP, which escape the email data during transmission.  See https://github.com/IndySockets/Indy/issues/135

Edited December 4, 2020 by Remy Lebeau

[Mark Williams 14]

 

13 hours ago, Remy Lebeau said:

It introduces new SaveTo...() and LoadFrom...() methods into TIdMessage (in versions of Delphi that support class helpers), as well as standalone TIdMessageHelper_SaveTo...() and TIdMessageHelper_LoadFrom...() procedures, which have an additional AUsesDotTransparency parameter added that can be set to False when needed. 

I've had a look at the TIDMessageHelper unit. I've been testing with EML files saved to file from Gmail. Reading your comments in the Github issue you have mentioned, I would have thought these files should not be using dot transparency. However, I can read them fine from file with TIDMessage using LoadFromFile and without the helper unit. But if I am dealing with email files coming from possibly any email client presumably there is no way of knowing whether the email uses dot transparency before loading it. 

 

If that is correct then is my best option:

 

1. Load the file into stream
2. Try IDMessage.LoadFromStream - if it doesn't load...
3.  Try the helper unit

Although I recall reading somewhere that if you try and read an email without dot transparency via IDMessage it takes a while and eventually triggers a time out error. Is that correct and is there any way around it?

 

13 hours ago, Remy Lebeau said:

In what way does it fail?

That statement should have been deleted. I had tested the sample code included in my original post without positioning the stream back to 0 so IDMessage was processing from the end of the stream. I realised my mistake and corrected it and I thought I had deleted this statement.

[Remy Lebeau 1421]

On 12/5/2020 at 1:01 AM, Mark Williams said:

I've been testing with EML files saved to file from Gmail. Reading your comments in the Github issue you have mentioned, I would have thought these files should not be using dot transparency.

I have never exported EML files from Gmail before, but it is not common for email to contain lines starting with periods, so it may be difficult to see at first whether dot transparency is actually being used or not.  So, as a test, I just sent an email containing such lines to my Gmail, and sure enough the resulting EML file DOES NOT use dot transparency.  Which makes sense, it shouldn't be.

On 12/5/2020 at 1:01 AM, Mark Williams said:

However, I can read them fine from file with TIDMessage using LoadFromFile and without the helper unit.

That will be true for any emails that do not contain lines starting with periods.  But try loading such EMLs using just TIdMessage by itself and you will see the periods get chopped off.  Hence the need for the TIdMessageHelper unit.

On 12/5/2020 at 1:01 AM, Mark Williams said:

But if I am dealing with email files coming from possibly any email client presumably there is no way of knowing whether the email uses dot transparency before loading it. 

99.999999% of EML files will likely NOT be using dot transparency.  Just the ones created by TIdMessage, or any other similarly broken email clients (which hopefully is rare).

On 12/5/2020 at 1:01 AM, Mark Williams said:

If that is correct then is my best option:

 

1. Load the file into stream
2. Try IDMessage.LoadFromStream - if it doesn't load...
3.  Try the helper unit

The problem with that approach is that you won't know when the load fails in step #2.  There is no error reported, lines containing leading periods will simply have those periods chopped off.

 

I would say instead to use TIdMessageHelper always with AUsesDotTransparency=False, unless you KNOW the EML file came from TIdMessage.  And even then, you can still use TIdMessageHelper, just set AUsesDotTransparency=True instead in that case.

On 12/5/2020 at 1:01 AM, Mark Williams said:

Although I recall reading somewhere that if you try and read an email without dot transparency via IDMessage it takes a while and eventually triggers a time out error.

I don't know where you read that, but it is not true.