Batch Reading Emails from Windows Explorer

[Mark Williams 14]

I am trying to work out the best way to batch read emails dropped into a windows explorer folder. I can think of three possibilities, but I can't get any of them to work!

 

MAPI & IMessage

This would probably do the job save for the fact that I cannot figure out how to open a file from explorer to use with the IMessage interface.

 

Outlook

An outlook MailItem would also probably do the job, but it has no LoadFromFile method and no other obvious method that I can find to load a file into a MailItem object.

 

INDY TIDMessage

I have tried with the following code:

 IDMessage1.LoadFromFile(FileName);

  with IDMessage1 do
        begin
          label6.caption := From.Text;
          label7.Caption := Recipients.EmailAddresses;
          label9.Caption := CCList.EmailAddresses;
          label10.Caption := BCCList.EmailAddresses;
          Label12.Caption := Subject;
          Label14.Caption := MsgId;
          Memo1.Text := Body.Text;
        end;

 

Filename is a msg file dropped from outlook into explorer.

 

It loads without complaint, but the output is wrong. Eg

From.Text returns

Quote

"W

W is the correct initial for the sender.

 

Recipients.EmailAddresses returns

Quote

"M

Again M is the correct initial.

 

Subject returns

Quote

R

This time with no leading double quote.

 

I assume I am missing something, but I do not know what and I can find no helpful examples.

 

I am looking for something that can reliably read emails in most formats. Don't know if TIDMessage is designed for this purpose. 

 

Hopefully someone can help re any of the above or even suggest any other possible solutions.

 

[FPiette 385]

12 minutes ago, Mark Williams said:

the best way to batch read emails dropped into a windows explorer folder.

Not sure I understand what you want to do! Please clarify the context.

By the way, the email file format depend on the source (mail client) you use to drop the file. Outlook will drop a .msg file.

 

 

[Mark Williams 14]

Sorry if I didn't make my requirements clear.

 

My app needs to be able to read emails in (hopefully) almost any format that have been dropped into an explorer folder. In my case I dropped them from Outlook so they were saved as .msg. Could be .eml format etc. I don't have control over the email source from which my users will dropping emails into Explorer.

 

Basically, I wish to read the various properties of the email (From, To, CC, BCC, Subject, body, MessageID etc) and add them to a database.

 

But to do that I need to be able to load the emails from a file rather than outlook. And that's where I am getting stuck.

 

By way of update I have just saved an email from GMail to disk (so eml format) and that was read ok by TIDMessage save for the body part, which I assume requires me to read the messageParts.

 

So with TIDMessage the issue seems to be with msg files.

 

As I originally said, I am not particularly bothered how I do this as long as whichever method I adopt it can reliably open and read most email formats.

[Anders Melander 1815]

Outlook msg files are COM Structured Storage files in the CFB3 (Compound File Binary) format.

Here's file file format specification: http://msdn.microsoft.com/en-us/library/cc463912.aspx

 

You can use the IStorage interface to load the file (via IStream) but you will need to locate and extract the desired properties manually. You can find tons of code on the net that shows how to do that and I think the above specs contains the names of the properties. I also think the last version of the Drag and Drop Component Suite I released even contained some code that did it.

 

AFAIR there's also an OpenIMsgOnIStg API function that can do some of the work for you but I think it might require that Outlook is installed.

[Mark Williams 14]

1 hour ago, Anders Melander said:

You can use the IStorage interface to load the file (via IStream)

I was hoping to find a one-size fits all solution. Off hand do you happen to know if most other email types are also structured storage files?

 

1 hour ago, Anders Melander said:

AFAIR there's also an OpenIMsgOnIStg API function that can do some of the work for you but I think it might require that Outlook is installed.

Requiring Outlook wouldn't be an issue, but I think it's MAPI that's needed. I've had a look at the Microsoft docs this returns me an IMessage on top of the storage doc. I have also had a look for some examples of how to deploy OpenIMsgonIStg. Haven't found much on it yet and suspect it will take me some time (form me) to master.  Do you (or anyone) happen to know know if the IMessage interface will work with other email types or just Outlook? 

 

Before I go down that route, I would really appreciate any feedback on whether TIDMessage can handle MSG files and, if so, how. @Remy Lebeau

[Anders Melander 1815]

16 minutes ago, Mark Williams said:

Off hand do you happen to know if most other email types are also structured storage files?

I would guess that no other types use it. It's not a very friendly format.

 

18 minutes ago, Mark Williams said:

Do you (or anyone) happen to know know if the IMessage interface will work with other email types or just Outlook? 

I can't speak authoritatively on the subject but if IMessage is a MAPI interface then in theory it should work with all MAPI providers. Unfortunately I don't think there's many applications left that support MAPI.

 

23 minutes ago, Mark Williams said:

I would really appreciate any feedback on whether TIDMessage can handle MSG files

I doubt it but it should be easy to determine since you have the source...

[Remy Lebeau 1436]

57 minutes ago, Mark Williams said:

Before I go down that route, I would really appreciate any feedback on whether TIDMessage can handle MSG files and, if so, how. @Remy Lebeau

No, it does not, only emails in RFC822 format, like .EML files.

Edited November 6, 2020 by Remy Lebeau

[Guest]

7 hours ago, Mark Williams said:

Off hand do you happen to know if most other email types are also structured storage files?

I think "MiTeC E-mail History Browser" http://mitec.cz/ehb.html is capable of doing this.

 

So it worth to try "MiTeC Outlook Express Reader" http://mitec.cz/oe.html , or you can drop the author an email and discuss buying one or more of his components, @mitzi invited to elaborate about this subject here.

[Mark Williams 14]

4 hours ago, Kas Ob. said:

http://mitec.cz/ehb.html

My Malware bytes is preventing access to this site as it believes there is a trojan. Just thought @mitzi would want to know!

 

[Mark Williams 14]

10 hours ago, Remy Lebeau said:

No, it does not, only emails in RFC822 format, like .EML files.

OK. Thanks for the info.

 

It didn't throw up an exception when trying to open the .msg file. 

 

I have decided that possibly the best way to proceed is to use TIDMessage to open and read any emails in the right format and Outlook for any emails TIDMessage can't handle by adding the latter to a temp folder in Outlook, processing and then removing the temp folder (subject to me working out exactly how I do that!).

 

If you or anyone else has any alternative suggestions before I start coding this, they would be really appreciated.

[Anders Melander 1815]

37 minutes ago, Mark Williams said:

... and Outlook for any emails TIDMessage can't handle by adding the latter to a temp folder in Outlook, processing and then removing the temp folder (subject to me working out exactly how I do that!).

If I'm understanding what you're saying you are going drop the msg files into Outlook and handle them there somehow. If that's an option then why not also handle the eml files this way?

[Mark Williams 14]

2 hours ago, Anders Melander said:

If I'm understanding what you're saying you are going drop the msg files into Outlook and handle them there somehow. If that's an option then why not also handle the eml files this way?

I am processing the email files for upload to a server. I don't need to add them to Outlook to do this. If I can open and handle them via TIDMessage it will save a significant amount of time.

[Anders Melander 1815]

1 minute ago, Mark Williams said:

I am processing the email files for upload to a server. I don't need to add them to Outlook to do this. If I can open and handle them via TIDMessage it will save a significant amount of time.

Yes I understood that. I was talking about your statement : "drop the msg files into Outlook and handle them there somehow".

[Mark Williams 14]

Just now, Anders Melander said:

Yes I understood that. I was talking about your statement : "drop the msg files into Outlook and handle them there somehow".

I am looking to handle email messages saved to disk in whatever format. I need to open the emails, read certain properties from them and then post the file and the data to a database. If I can do this with just the file (as I can with EML using TIDMessage) then there is no need to fire up Outlook and copy the files into Outlook and then process them.

[Mark Williams 14]

In case anyone is interested, it is fairly straightforward to open a MSG file in an Outlook MailItem (if you know how). Code below (thanks to Dmitry Streblechenko posted on StackOverflow):

var
    App, NS, Msg:Variant;
begin
      App := CreateOLEObject('Outlook.Application');

      NS := App.GetNamespace('MAPI');
      NS.Logon;
      Msg := NS.OpenSharedItem(fileName);

I've also found a VB example of how to import EML files into Outlook: https://www.howto-outlook.com/howto/import-eml-files.htm. This shows how to import them into an Outlook folder.

[Anders Melander 1815]

So you are doing it with Outlook which again prompts me to ask why you're not also handling the eml files the same way (after importing them into Outlook) instead of handling them in two different ways?

 

Personally I would probably:

1. Load the msg file into an IStorage with StgOpenStorage
2. Get an IMessage from the IStorage with OpenIMsgOnIStg
3. Extract the IMessage properties with HrGetOneProp or IMessage.OpenProperty

[Mark Williams 14]

2 hours ago, Anders Melander said:

So you are doing it with Outlook which again prompts me to ask why you're not also handling the eml files the same way (after importing them into Outlook) instead of handling them in two different ways?

I don't want to import the emails into Outlook as it is an unnecessary and added overhead (it's also not as straightforward to do so for EML files as for MSG files). 

 

My app needs to deal with EML files on a folder in Explorer. I can read the properties I need quickly using TIDMessage. I then post that data and the file to the server/database. I don't need the overhead of loading them into Outlook before doing this.

 

If TIDMessage could handle MSG files I'd do it the same way. However, it doesn't so I am considering using Outlook to do it for me. But that comes at a cost. So it may be better for me to do as you suggest and use StgOpenStorage just for MSG files. Haven't quite made my mind up yet.

[Angus Robertson 577]

Probably not the answer you want, but the ICS library has a component TIcsMailQueue which sends emails from a directory of EML files (not MSG), using multiple SSL SMTP servers and retrying until sent.  But it only parses the EML files sufficiently for the SMTP protocol.  There are other components for reading mail headers and bodies.

 

Angus

 

[Remy Lebeau 1436]

1 hour ago, Mark Williams said:

If TIDMessage could handle MSG files I'd do it the same way. However, it doesn't

Since the MSG format is publicly documented, it would be feasible to add support for it in Indy, in a manner that does not depend on Outlook APIs.  Similar to how Indy already does for TNEF (winmail.dat) attachments via the TIdCoderTNEF class.  But having the TIME to make such an implementation is an issue, for me at least.

 

Edited November 6, 2020 by Remy Lebeau

[Mark Williams 14]

@Remy Lebeau I have found reference to TIDCoderTNef, but I can't this component nor can I find an IDCoderTNef unit. I can't find anything in the help files either. Has this been replaced? If not, where can I find it please and is there any help on or any examples of how to use it that you are aware of?

[Remy Lebeau 1436]

On 11/8/2020 at 6:48 AM, Mark Williams said:

@Remy Lebeau I have found reference to TIDCoderTNef, but I can't this component nor can I find an IDCoderTNef unit.

Which version of Indy are you looking at?  TIdCoderTNEF was introduced in Indy 10, are you looking at Indy 9, perhaps?

Quote

I can't find anything in the help files either.

TIdCoderTNEF is not documented in the help files.

Quote

Has this been replaced?

No, TIdCoderTNEF still exists in Indy 10.

Quote

If not, where can I find it please

TIdCoderTNEF has been in Indy 10 for over a decade.  It is in the IndyProtocols package.

Quote

is there any help on or any examples of how to use it that you are aware of?

There is no documentation available for TIdCoderTNEF, other than comments in the unit itself.  I think it was created after the last time the documentation was updated, which was sadly a very long time ago.

 

In a nutshell, you would simply extract a TNEF attachment from a normal email, and pass that attachment to TIdCoderTNEF.Parse(), which will then populate a TIdMessage with the relevant message data (headers, body, attachments, etc) from the TNEF.

Edited November 9, 2020 by Remy Lebeau

[Mark Williams 14]

4 hours ago, Remy Lebeau said:

TIdCoderTNEF was introduced in Indy 10, are you looking at Indy 9, perhaps?

Quote

No. Definitely version 10. I have a host of IDCoder... units in the protocols folder, but no TIDCoderTNEF.

 

Indy was installed same time as I installed Delphi 10.4. Just been back over the last couple of installs 18.0, 19.0, 20.0 and latest is 21.0. IDCoderTNef is not in the protocols folder of any of these installs.

[Remy Lebeau 1436]

5 hours ago, Mark Williams said:

No. Definitely version 10. I have a host of IDCoder... units in the protocols folder, but no TIDCoderTNEF.

 

Indy was installed same time as I installed Delphi 10.4. Just been back over the last couple of installs 18.0, 19.0, 20.0 and latest is 21.0. IDCoderTNef is not in the protocols folder of any of these installs.

Like I said, the IdCoderTNEF unit has been in the Protocols folder for years.  But if you are using the version of Indy that shipped pre-installed with each IDE, and they don't have the unit present, then Embarcadero must have chosen not to distribute it.  The unit is not actually referenced by the IndyProtocols package, as it is not part of the main Indy packages, just a standalone utility unit, so Embarcadero may have just skipped it as a non-dependency.  In which case, you will have to download it from Indy's GitHub repo instead (https://github.com/IndySockets/Indy/blob/master/Lib/Protocols/IdCoderTNEF.pas).

Edited November 10, 2020 by Remy Lebeau

[Mark Williams 14]

Adding TIDCoderNef.pas to the protocols folder installed with Delphi doesn't work. Compiling fails to locate the unit.

 

Saving it in a different location causes compile to fail whilst trying to find IDCompilerDefines.inc. I have copied the latter from protocols folder. to my new location, which compiles fine. However, is this the right approach or should I be dealing with this in a different way?

[Remy Lebeau 1436]

3 hours ago, Mark Williams said:

Adding TIDCoderNef.pas to the protocols folder installed with Delphi doesn't work. Compiling fails to locate the unit.

Then you don't have your project setup to include that folder in its search paths.  Simply adding the unit to the Protocols folder won't magically add it to Indy, since it is already compile.  That would require adding the unit to Indy's Protocols package, and then recompiling and reinstalling Indy.

Edited November 11, 2020 by Remy Lebeau