JohnLM 27 Posted October 30 Specs 1: Delphi XE7, VCL, Win7 laptop - main work station laptop computer Specs 2: Delphi 12.2, VCL, Win10 tablet - optional I started to create an "email client" IDE shell for my Optimum.net internet provider. Optimum provides an online email client, https://myemail.optimum.net/webmail/, although it is very, very slow. I want to see if I can just create a very basic email client IDE of my own. No fancy bells or whistles, but just something simple enough to read whatever email subject titles are in my feed and then be able to click on any of them and open/read/send the actual email inside the client. For this starting project, I will be using XE7 because that is what I mainly use. As of current, I am clueless how to connect to my provider's service account in order to do this. I will figure out how to get the info in the respective panes, clicking/selecting/etc. -- see image below. Can I please have some how to's from anyone knowledgeable in this area with some solution(s) ? TIA Share this post Link to post
Remy Lebeau 1694 Posted October 30 (edited) Don't try to go through the web interface. Use your provider's IMAP (preferred) or POP3 server for receiving mail, and its SMTP server for sending mail. In your case, those servers are as follows: IMAP Host: mail.optimum.net Port 993, requires SSL Port 143, non-secure POP3 Host: mail.optimum.net Port 110, requires SSL Port 995, non-secure SMTP Host: mail.optimum.net Port: 465, requires SSL Port 587, non-secure There are plenty of IMAP/POP3/SMTP libraries available for Delphi. The IDE comes with Indy pre-installed, which has such components. Or, you can install other 3rd party libraries with such capabilities, such as ICS, Synapse, etc. Edited October 30 by Remy Lebeau Share this post Link to post
JohnLM 27 Posted October 31 Progress update. . . (re -- retrieving header parts into a list to view (using a tmemo for now) ---) Hi, thanks for the pointers. Okay. question. Is there a way I can retrieve the message via date order? Something I'm missing in the code as part of the call, like [msSortByDate] for instance? IdMessage1.Date Or, do I have to: 1. create a list (via tstringlist) 2. retrieve all the message headers into the list 3. sort the list 4. then, re-display the list (via tmemo, again, for now) Share this post Link to post
Remy Lebeau 1694 Posted October 31 27 minutes ago, JohnLM said: Okay. question. Is there a way I can retrieve the message via date order? Something I'm missing in the code as part of the call, like [msSortByDate] for instance? Neither POP3 nor IMAP guarantee any kind of ordering when you retrieve a list of messages from a mailbox (however, IMAP does provide an extension to search for items in a sorted manner, but that is not standard behavior). So, if you want to sort the mails, you typically have to sort them yourself after you have downloaded them. 27 minutes ago, JohnLM said: Or, do I have to: 1. create a list (via tstringlist) 2. retrieve all the message headers into the list 3. sort the list 4. then, re-display the list (via tmemo, again, for now) Basically, yes. Share this post Link to post
JohnLM 27 Posted October 31 (edited) Okay, so I was correct. Thanks. I don't know which component to correctly use in order to pull certain things, like the headers for instance. But, I am using POP3 for that, not IMAP--as far as I understand things so far--it's all new to me. But at least the project is working this far into it. However, reteaving the first 150 headers is slow. It takes 31 seconds. And I have over 7400 counts, per the following code snippet I created below: (note, not sorting yet in this code. I quickly threw this together in order to make it work, so far so good. sorting will be later, maybe) function tform1.sortList(const pop3: TIdPOP3): tstringlist; var i: integer; MsgCount: Integer; Msg: TIdMessage; s1,s2,s3,s4, buildStr: string; begin buildstr:=''; try try result:=tstringlist.Create; msg:=tidmessage.Create(self); MsgCount := pop3.CheckMessages; // idx starts at 1, no 0 for i := 1 to 150 do {MsgCount do} begin // <-- this section takes 31 seconds to pull 150 pop3.RetrieveHeader(i, msg); s1:=i.ToString(i); s2:=FormatDateTime('yyyy mm dd hh:mm:ss', msg.Date); s3:=msg.From.Text; s4:=msg.Subject; buildStr:= s1+' | ' +s2+' | ' +s3+' | ' +s4; result.Add(buildstr); end; except on e: Exception do ShowMessage('Error: ' + e.Message); end finally if pop3.Connected then pop3.Disconnect end; end; And now I am wondering how many hours it will take to download all the headers just to get to the most current date of emails. I must be missing something, in order to, let's say, download the most current email(s), like the first five current ones, for instance. Edited October 31 by JohnLM Share this post Link to post
Remy Lebeau 1694 Posted November 1 6 hours ago, JohnLM said: I don't know which component to correctly use in order to pull certain things, like the headers for instance. But, I am using POP3 for that, not IMAP--as far as I understand things so far--it's all new to me. But at least the project is working this far into it. POP3 is basically meant to just download and delete messages, it is not designed for mailbox management. Use IMAP for that instead. 6 hours ago, JohnLM said: However, reteaving the first 150 headers is slow. It takes 31 seconds. And I have over 7400 counts. You should profile your code to see where it's really spending it's time. But, since you are not extracting any structured data from the emails, try setting TIdMessage.NoDecode=True, see if that helps speed it up a little. 6 hours ago, JohnLM said: function tform1.sortList(const pop3: TIdPOP3): tstringlist; var i: integer; MsgCount: Integer; Msg: TIdMessage; s1,s2,s3,s4, buildStr: string; begin buildstr:=''; try try result:=tstringlist.Create; msg:=tidmessage.Create(self); MsgCount := pop3.CheckMessages; // idx starts at 1, no 0 for i := 1 to 150 do {MsgCount do} begin // <-- this section takes 31 seconds to pull 150 pop3.RetrieveHeader(i, msg); s1:=i.ToString(i); s2:=FormatDateTime('yyyy mm dd hh:mm:ss', msg.Date); s3:=msg.From.Text; s4:=msg.Subject; buildStr:= s1+' | ' +s2+' | ' +s3+' | ' +s4; result.Add(buildstr); end; except on e: Exception do ShowMessage('Error: ' + e.Message); end finally if pop3.Connected then pop3.Disconnect end; end; 6 hours ago, JohnLM said: And now I am wondering how many hours it will take to download all the headers just to get to the most current date of emails. I must be missing something, in order to, let's say, download the most current email(s), like the first five current ones, for instance. Try IMAP instead, which has some search capabilities. However, TIdIMAP4 does not currently implement the SORT extension, but you should be able to send that command manually. Searching and sorting gives you a list of message numbers/IDs, then you can download just the corresponding mails you want. Share this post Link to post
JohnLM 27 Posted November 1 progress update. . . not going well with IMAP. FYI: in case it was not clear, I am using XE7 and Indy. Quote POP3 is basically meant to just download and delete messages, it is not designed for mailbox management. Use IMAP for that instead. I would like to use IMAP but it appears to be pretty complicated for this novice. I found a link to a resource link for an Indy help document, It's the closest to my version for XE7. here: https://www.indyproject.org/documentation/#:~:text=IndyDocs_10.1.1.822_Pdf.zip. So it should be good enough for this endeavor. But it is not helping me to get IMAP working. Also, I have been searching around on google. And, no, I don't use chatGPT, no have. And I don't even know how, nor where it is, to use it. Anyway. I will continue to search until I find something with some code snippits. I did try this snippet: IMAP4.Connect(true); -- which fails. Actually, I get a response back, "Authenticated" via this code: IMAP4.ConnectionState, and followed by: IMAP4.MailBox.TotalMsgs, gives me an error message: "Unable to execute command, wrong connection state;Current connection state: Authenticated.". And, i'm not sure how to loop through all the headers, etc. I have no found. I'm sure I'll figure it out, eventually, and post about it if it works. Until then. d Share this post Link to post
JohnLM 27 Posted November 3 (edited) Progress update. . . - showing mailboxes in my apps left pane area of my app (similar to like the one shown in my first post). I've made some progress today. I managed to learn how to display mailboxes and show them in my "email client" app, using the following codesnippet, below. procedure TForm1.btn2Click(Sender: TObject); var ts: tstringlist; i: integer; msgListCol: TIdMessageCollection; // testing..? itmCol: TCollectionItem; // testing..? begin m1.Lines.Clear; ts:=tstringlist.Create; msgListCol := TIdMessageCollection.Create; // testing.. i think i need a collection, not working w/ this routine. //itmCol := TCollectionItem ate(msgListCol); // testing...? borked.. does not work IMAP4.connect; m3.Lines.Add( conStateRead(imap4.ConnectionState) ); if not IMAP4.ListMailBoxes(ts) then m3.Lines.add( 'no mailboxes.'); // testing.. retrieve mailboxes = success m3.Lines.Add(ts.Count.ToString()+ 'entries'); // just shows that I have 6 mailboxes. for i := 0 to ts.Count-1 do m1.Lines.Add('>'+ts.Strings[i]); // show the mailboxes = success IMAP4.Disconnect(); ts.Free; end; The mailboxes that my provider uses are the following mailbox's (or folders?): { Bulk Mail Deleted Drafts INBOX Junk Sent } I am not sure what "Bulk Mail" is. I searched google and it said one possible use is for SPAM. But I would assume that the "JUNK" would be used for that. IDK. I created a new test email with Optimum in order to see how things go with finding it, and retrieving it via IMAP. But so far, no success. Edited November 3 by JohnLM edited source codesnippet Share this post Link to post
JohnLM 27 Posted November 3 What I believe I need to do is find the routine responsible for indexing? into a collation that holds those mailboxes below and then call the retriever routine to pull the header and/or messages stored in the mailbox via an index number, I'm guessing. So, something like: IMAP4.MailBoxes[3].RetrieveHeaders(aList) or something like that. IDK. { 0 Bulk Mail 1 Deleted 2 Drafts 3 INBOX 4 Junk 5 Sent } Share this post Link to post
JohnLM 27 Posted November 3 (edited) I managed to get IMAP working in order to pull Headers in my app. But on some mailboxes I am getting the following pop-up error message: ERROR: "Unexpected: Non-last response line (i.e. a data line) did not start with a *, offending line: . . . " NOTE: The ". . ." is the part of the email subject title that displays. I am just leaving that out in this post. The error message appears to be saying that the data it is trying to retrieve did not start with an asteric, "*" character. I did some google searches on this error message and it gave me this responses--and a few others: "The issue usually points to a problem with the data received from the server that does not match what the Indy component expects at that specific point in the protocol flow. " I am not sure how to resolve. I thought that maybe some flag or switch might be missing or should be added into the routine or component's properties. But I don't know. Can someone provide some answers? Thanks. Edited November 3 by JohnLM typo Share this post Link to post
JohnLM 27 Posted November 3 Forgot to add the following for more clarity. . . In this routine, the error stems from the line I indicated in the // <-- comment section below: procedure TForm1.btn3Click(Sender: TObject); var MsgList: TIdMessageCollection; begin try if imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} )=true then MsgList := TIdMessageCollection.Create; try IMAP4.UIDRetrieveAllEnvelopes(MsgList); // <--- this is the line where the error occurs. . . end; Share this post Link to post
Lajos Juhász 336 Posted November 4 try if imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} )=true then MsgList := TIdMessageCollection.Create; try IMAP4.UIDRetrieveAllEnvelopes(MsgList); // <--- this is the line where the error occurs. . If SelectMailbox returns false, the variable MsgList remains uninitialized (undefined). Continuing execution will then cause data to be written to an unpredictable memory location, leading to potential data corruption or an Access Violation. You should try: if not imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} ) then begin // maybe add an error message here to inform the user that selectmailbox failed. exit; end; MsgList := TIdMessageCollection.Create; try IMAP4.UIDRetrieveAllEnvelopes(MsgList); .... Share this post Link to post
JohnLM 27 Posted November 4 When using your method, I am getting access violation errors instead. But when I go back to my method, everything works as I asusme them should. No errors. Let me explain. . . In the method I was using, any mailboxes that were empty (had no mail in them) went through the routine fine, and with no errors The folders were empty for "Deleted", "Bulk Mail". So the routine succeeded with no errors. The folders that were not empty are "Sent", "INBOX", and "Draft". These give me the issues I stated previously. However, the only folder that does work, and returns data (emails) is "Junk". This succeeds with no issues and gives me all email headers. Share this post Link to post
Remy Lebeau 1694 Posted November 4 18 hours ago, JohnLM said: I managed to get IMAP working in order to pull Headers in my app. But on some mailboxes I am getting the following pop-up error message: ERROR: "Unexpected: Non-last response line (i.e. a data line) did not start with a *, offending line: . . . " That means either the server's response is malformed in some way, or TIdIMAP4 is not able to parse it properly. Hard to say either way without seeing the raw response data. You can attach a TIdLog... component, such as TIdLogFile, to the TIdIMAP4.Intercept property to capture that data for debugging. 4 hours ago, JohnLM said: When using your method, I am getting access violation errors instead. What Lajos said is correct and you should follow his advice, but the fact that you get errors in such code means you are likely doing something wrong elsewhere in your code and this code is just a victim of circumstance. You need to debug your code. 4 hours ago, JohnLM said: But when I go back to my method, everything works as I asusme them should. No errors. Even worse, because the code you did show has a failure waiting to happen, as Lajos explained. But even so, without seeing your whole code, it's hard to say what else you may be doing wrong. 4 hours ago, JohnLM said: In the method I was using, any mailboxes that were empty (had no mail in them) went through the routine fine, and with no errors The folders were empty for "Deleted", "Bulk Mail". So the routine succeeded with no errors. The folders that were not empty are "Sent", "INBOX", and "Draft". These give me the issues I stated previously. However, the only folder that does work, and returns data (emails) is "Junk". This succeeds with no issues and gives me all email headers. None of that is related to what Lajos said. The issue he described is strictly about memory management, it has nothing to do with the IMAP protocol or your mail data. Share this post Link to post
JohnLM 27 Posted November 5 (edited) Remy, thank you for the tip on the TIdLogFile. I did not know I could utilize that in situations like this. I don't know anything about the log file and how to read it but I did play around with trying to parse through it (in a separate app project) to view it slightly better (without touching the file itself). But as it turns out there are too much private info in it for me to post it here. However, if there are sections you would like to see that I can extract out of it and post here, let me know what sections and I will have a look to see what is/is not private to post here. . . . The code snippet below is the latest routine I am testing out to read the email headers. It only works for the "Junk" mailbox folder. But the "INBOX" folder fails, as well as the rest The following snippet takes the MsgList contents and formats it (date, subject, text) into one line of detail (like in typical email client IDE's) for a cleaner view function FormatOutputHeaders(msgList: TIdMessageCollection): tstringlist; var idx: integer; s1,s2,s3,s4: string; cnt: integer; // temporary: limit amt of MsgList lines in tmemo for now. begin result:=tstringlist.Create; cnt:=0; for idx := 0 to msgList.Count-1 do begin if cnt>50 then exit; // temporary cut-off for now until issue(s) are resolved. s1:=idx.ToString(idx); s2:=FormatDateTime('yyyy mm dd hh:mm:ss', msgList[idx].Date); s3:=msgList[idx].From.Text; s4:=msgList[idx].Subject; result.Add(s1+' | ' +s2+' | ' +leftstr(s3,30)+' | ' +s4); // add to stringlist and return to tmemo (2nd output pane) end; end; Next comes the main routine for retrieving the email headers. procedure TForm1.btn5LajosClick(Sender: TObject); var MsgList: TIdMessageCollection; begin m2.Lines.Clear; IdLogFile1.Active:=true; try imap4.Connect; if not imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} ) then begin // <-- this never gets "false" m3.Lines.Add('mailbox: '+ebMailBoxName.text+' failed to open'); exit; end; MsgList := TIdMessageCollection.Create; IMAP4.UIDRetrieveAllEnvelopes(MsgList); m2.Lines.assign(FormatOutputHeaders(MsgList)); // display the email headers in 2nd pane window (see first post of screenshot) finally imap4.Disconnect; msgList.Free; IdLogFile1.Active:=false; end; end; When I select "INBOX", there is a pause, and then there is this pop-up error message, as seen below. . . . Please note, all the info I found through Google searches. And mainly with code snippets that had issues in them-selves. They were all that were available as a how-to since there are no official resource(s) or free book(s), with working code snippet examples. So, you use what can find and do the best you can get them to work or ask questions. Edited November 5 by JohnLM typos at the bottom Share this post Link to post
JohnLM 27 Posted November 5 Analyzing and investigating the log file shows a lot of incomplete final "FETCH"'s. That is, the final FETCH in each "Recv " group of Fetches are not complete. It is always the last Fetch that is incomplete. And it appears to me that the line is too long to process, maybe?. For example. In TMEMO, the maximum line length is 1024 characters. Any chars after that length get's bumped to the next line as junk. That is, it is no longer a part of the previous line. That may be causing the corruption, I guess. Each group of Recv has between 4 and 6 Fetches. It is always the last Fetch that is incomplete. There is definitely no <EOL> code at the end. But that sequence of the last Fetch is not complete, so the process does not complete? and messes up. I don't know. I am just guessing now. Share this post Link to post
Remy Lebeau 1694 Posted November 5 (edited) 5 hours ago, JohnLM said: Analyzing and investigating the log file shows a lot of incomplete final "FETCH"'s. Can you be more specific? Show an example. 5 hours ago, JohnLM said: That is, the final FETCH in each "Recv " group of Fetches are not complete. It is always the last Fetch that is incomplete. And it appears to me that the line is too long to process, maybe?. For example. In TMEMO, the maximum line length is 1024 characters. Any chars after that length get's bumped to the next line as junk. That is, it is no longer a part of the previous line. That may be causing the corruption, I guess. Hard to say without seeing the raw data, but don't just rely on what's going on with line breaks in the log file. Remember that TCP is a byte stream. It may take multiple reads/writes to transmit a complete message, and each one gets logged separately. So make sure you are paying attention to what the actual data is doing, not just what the log file is doing. Quote Each group of Recv has between 4 and 6 Fetches. It is always the last Fetch that is incomplete. There is definitely no <EOL> code at the end. That may not be a problem, depending on the actual data, as not all IMAP data is terminated by line breaks. String data may be prefixed with a byte length and terminated when the count is reached. It really depends on the context of the data. IMAP is a complex protocol. Edited November 5 by Remy Lebeau Share this post Link to post
Kas Ob. 177 Posted November 5 While Remy is helping you i want to point to this 6 hours ago, JohnLM said: In TMEMO, the maximum line length is 1024 characters. Any chars after that length get's bumped to the next line as junk. That is, it is no longer a part of the previous line. That may be causing the corruption, I guess. This, did hit a nerve on me, as i fall in this trap few times when testing few things, like downloading an SVG saving it to memo then try to load it, just to see it fail to render right. And yes, in Windows Memo and Text the line is limited to 1024 char, thus feeding more will wrapped and introduce a CRLF in case of multi lines are supported, so if are reading the data and saving it to TMemo then extracting it from TMemo to parse then it will be invalid and its integrity can't be guaranteed, there is also a bug in the older VCL (and may be the newer) that expect the lines to be 4096 char and allocate a buffer on the stack for that, can't remember where right now, Anyways, here what you should check while continuing with Remy 1) Are you concatenating the received data using visual component ? this is unreliable way to parse raw data or data designed as protocol in whole. 2) Your received data (packets ) somehow parsed individually or appended to each other until no further receive then parsed, in other words like (1) are parsing the responses partially ? the way to do it is to make sure you are receiving and concatenating/appending the packets or data then parse after all received, otherwise the protocol will and might be failing exactly as you described. Share this post Link to post
JohnLM 27 Posted November 5 \OT\ -- In my view, you guys are Pros. I'm just not fast enough at this programming. I'm a very slow learner. It takes a while for things to sink in and, . . . "Ah, I get it now...!" -- \OT end\ @Kas Ob. 1. no. I don't believe so. 2. I parse or work with the complete data as received. \FWIW\ I am using XE7 for this project. I also have D11.2 and D12.2, but those are on a tablet and another laptop that I rarely use. XE7 is on my main daily use laptop. \FWIW end\ Share this post Link to post
JohnLM 27 Posted November 5 @Remy Lebeau - As you may recall earlier in this discussion, POP3 works when using XE7, and pulls whatever headers are in any of the folders, successfully. But it was realized that it is far too slow and not applicable to my needs. So, we know POP3 works, just not IMAP4 when using XE7. This is assuming that the code snippet I posted in my previous post are correct. Share this post Link to post
Lajos Juhász 336 Posted November 6 In the code you last posted you have made at least 2 errors. First, you are not increasing cnt variable thus you are displaying all the data, not the first 50. The new memory related bug you have introduced is a memory leak of the TStringList. The result of the FormatOutputHeaders is never freed. Share this post Link to post
JohnLM 27 Posted November 13 Progress update. . . - resolving the Leak that Lajos pointed out. Finally!! After a long week or two, I have managed to figure out how to resolve the memory leak. I know pretty much nothing about this issue with leaks. But I learnt something new today. And without hast, below is my newly updated routine and no leaks found (via the Delphi built-in method to detect general leaks at app closure). procedure TForm1.btn5LajosClick(Sender: TObject); var MsgList : TIdMessageCollection; MsgHeaders: TStringList; begin m2.Lines.Clear; IdLogFile1.Active:=true; imap4.Connect; try if not imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} ) then begin m3.Lines.Add('mailbox: '+ebMailBoxName.text+' failed to open'); exit; end; MsgList := TIdMessageCollection.Create; IMAP4.UIDRetrieveAllEnvelopes(MsgList); MsgHeaders:= FormatOutputHeaders(MsgList); m2.Lines.Assign(msgheaders); finally imap4.Disconnect; msgList.Free; msgHeaders.Free; IdLogFile1.Active:=false; end; end; If there is no objection to the above code snippet, I can now move into the next routine, which is to obtain the body text (the email) and display it in the 3rd pane area of the app. Share this post Link to post
Remy Lebeau 1694 Posted November 14 (edited) 7 hours ago, JohnLM said: And without hast, below is my newly updated routine and no leaks found (via the Delphi built-in method to detect general leaks at app closure). Your code still has a fatal flaw in it. If SelectMailBox() happens to fail, your finally block will try to Free() 2 invalid objects, leading to possible crashes or subsequent undefined behavior for the remaining lifetime of your app. You can initialize the pointers to nil before entering the try block (as calling Free on nil pointers is safe), eg: procedure TForm1.btn5LajosClick(Sender: TObject); var MsgList : TIdMessageCollection; MsgHeaders: TStringList; begin MsgList := nil; MsgHeaders := nil; ... try ... finally ... msgList.Free; msgHeaders.Free; ... end; end; However, there is also the problem that the LogFile will remain active if Connect() fails. So, you should use nested try..finally blocks instead to cleanup each resource you acquire, eg: procedure TForm1.btn5LajosClick(Sender: TObject); var MsgList : TIdMessageCollection; MsgHeaders: TStringList; begin m2.Lines.Clear; IdLogFile1.Active := true; try imap4.Connect; try if not imap4.SelectMailBox(ebMailBoxName.text {'INBOX'} ) then begin m3.Lines.Add('mailbox: '+ebMailBoxName.text+' failed to open'); exit; end; MsgList := TIdMessageCollection.Create; try IMAP4.UIDRetrieveAllEnvelopes(MsgList); MsgHeaders := FormatOutputHeaders(MsgList); try m2.Lines.Assign(msgheaders); finally MsgHeaders.Free; end; finally MsgList.Free; end; finally imap4.Disconnect; end; finally IdLogFile1.Active := false; end; end; Edited November 14 by Remy Lebeau Share this post Link to post
JohnLM 27 Posted November 15 Remy, thank you for taking the time to explain the issues and the need for try/finally in the code I was using. I will keep this in mind for other situations if I run into them and understand where to put the code snippets in the try/finally sections, as they are confusing to work with at times and is why I don't use them most of the time. Also also to Lajos, for pointing out the leaks, as it helped me to realize how to detect at least basic leak issues. Share this post Link to post