Sending Email via GMail Using OAuth 2.0 via Indy

[Remy Lebeau 1396]

3 minutes ago, rvk said:

I don't think TIdMessage creates a much different text but I'm not sure.

TIdMessage creates RFC822-style emails.  Do note, however, that the TIdMessage.SaveTo...() methods are designed for SMTP usage, and so will escape emails per SMTP dot-transparency guidelines.  So, to use TIdMessage emails in other protocols, be sure to use the overloaded SaveTo...() methods provided by the TIdMessageHelper unit, which adds an AUseDotTransparency parameter that you can set to False to disable that escaping.

 

https://www.indyproject.org/2015/07/30/new-tidmessage-helper/

[Geoffrey Smith 45]

On 12/14/2021 at 1:11 AM, JLG said:

the following is based on your existing code for getting token

 

function TgmailFrm.Sendemail(FromName, emailFrom, emailRecip, emailSubject,
  emailBody, emailAttach: string; Quietly: Boolean): Boolean;

I believe the problem with your code is you are not base64url encoding the message.  If you have Delphi 11, you could add the code fragment below to your code and I think it should work.

 

 

uses System.NetEncoding;

 

var

  MSEncoded : TStringStream;

begin

        MSEncoded := TStringStream.Create;

        IdMessage.SaveToStream(MS);

        TNetEncoding.Base64URL.Encode(MS, MSEncoded);

 

end;

 

[JLG 0]

in order to use the drafts folder of gmail, you need permission. gmail does not allow apps permission to drafts if all they need is to send emails. they insist to use https://gmail.googleapis.com/gmail/v1/users/' + emailFrom +/messages/send  instead.

if i had access to drafts, then I could use the smtp component. this works well, but the google team complained to me that they don't allow this as I only require to send email, so no access to smtp.

the code using synapse above is using smtp. when I experimented with smtp, I did not need JSON as you say correctly. problem is now that I am attempting to email directly, the code above wont work.

I tried adding the header accept json and mmime type as suggested, but to no avail...

gmail staff are incredibly unhelpful. they refuse to support delphi questions. they have sample code for other platforms(see https://developers.google.com/gmail/api/quickstart/js). I just cant get code working for delphi.

does anyone there have a simple send with attachment code using delphi for gmail without smtp but using REST?

[rvk 33]

1 minute ago, JLG said:

I tried adding the header accept json and mmime type as suggested, but to no avail...

Can you post the raw rfc822 data in a txt file here with a simple testmail? (Did you try the overloaded saveto helper to disable escaping email adresses as suggested??)

 

I can try it here to see what goes wrong.

The rfc822 mimetype should work with send-api too (i thought).

 

I also found that the error messages you get are indeed often unhelpfull (just a 4xx rejected without reason).

 

Another option is just plain smtp with a specific app-password. But that's something each user should set in their own account (and provide your app with that password). Using the authetication method is much easier. Downside is that your app needs to go through the approval process of Google for the specified access.

 

[JLG 0]

On 12/16/2021 at 10:06 PM, Geoffrey Smith said:

I believe the problem with your code is you are not base64url encoding the message.  If you have Delphi 11, you could add the code fragment below to your code and I think it should work.

 

 

uses System.NetEncoding;

 

var

  MSEncoded : TStringStream;

begin

        MSEncoded := TStringStream.Create;

        IdMessage.SaveToStream(MS);

        TNetEncoding.Base64URL.Encode(MS, MSEncoded);

 

end;

 

I am afraid I still get a "bad request" response

[rvk 33]

Btw. It's best you first try a very simple rfc822 style message without attachment. If that works you can debug why adding attachments doesn't work.

 

You can post the simple message here if it doesn't work for you so we can debug it. (i.e. wrong headers due to not disabling escaping mail adresses)

 

 

[JLG 0]

On 12/16/2021 at 4:35 PM, Remy Lebeau said:

TIdMessage creates RFC822-style emails.  Do note, however, that the TIdMessage.SaveTo...() methods are designed for SMTP usage, and so will escape emails per SMTP dot-transparency guidelines.  So, to use TIdMessage emails in other protocols, be sure to use the overloaded SaveTo...() methods provided by the TIdMessageHelper unit, which adds an AUseDotTransparency parameter that you can set to False to disable that escaping.

 

https://www.indyproject.org/2015/07/30/new-tidmessage-helper/

I did try what you said by doing this code IdMessage.SaveToStream(MS,False,false); this resolves the dottransparency. but still not working. I enclose the txt file of the MS saved as file (I just changed the email address to xxxx to protect privacy)

I do not know what is wrong, other than perhaps not the right json format for gmail ? I tried encoding base64... not resolved..

messsage.txt

[rvk 33]

I just tested your messsage.txt and it works for me. I can create a draft with that message but I can also directly send it (because I have both draft and send access in my app approval).

This is my code. (Although I would remove the headers From, Reply-To and Date because those would be added automatically. It doesn't hurt when you add them but they are NOT used. From is always the address of the default sender in gmail and Reply-To is always the gmail account and Date is always now.)

 

Quote

 

      Url := 'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/messages/send';
      Response := TStringList.Create;
      HTTP := THTTPSend.Create;
      try
        // WorkStr := CreatePlainEML;
        WorkStr := TFile.ReadAllText('C:\Temp\messsage.txt');

        WriteStrToStream(HTTP.Document, ansistring(WorkStr));

        json := nil;

        HTTP.MimeType := 'message/rfc822';
        HTTP.Headers.Clear;
        HTTP.Headers.Add('Authorization: Bearer ' + gOAuth2.access_token);
        HTTP.Headers.Add('Accept: application/json');
        if HTTP.HTTPMethod('POST', Url) then
        begin
          Response.LoadFromStream(HTTP.Document);
          if HTTP.ResultCode <> 200 then
              ShowMessage(Response.Text)
          else
          begin
            Result := SUCCESS_SUCCESS;
          end;
        end;

      finally
        HTTP.Free;
        Response.Free;
      end;

 

 

Looking at your code I now do see a difference with my code.

You are using Url := 'https://gmail.googleapis.com/gmail/v1/users/' + emailFrom + '/messages/send';

I'm using Url := 'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/messages/send';

 

You need to use the Upload URI, for media upload requests to post this as raw message.

See here https://developers.google.com/gmail/api/reference/rest/v1/users.messages/send

There is a difference with /gmail/v1 and /upload/gmail/v1.

 

If I use your Url I also get an invalid JSON error message 400.

So use the other Url and it should be fine.

 

Edited December 18, 2021 by rvk

[JLG 0]

On 12/18/2021 at 8:26 PM, rvk said:

I just tested your messsage.txt and it works for me. I can create a draft with that message but I can also directly send it (because I have both draft and send access in my app approval).

This is my code. (Although I would remove the headers From, Reply-To and Date because those would be added automatically. It doesn't hurt when you add them but they are NOT used. From is always the address of the default sender in gmail and Reply-To is always the gmail account and Date is always now.)

 

 

Looking at your code I now do see a difference with my code.

You are using Url := 'https://gmail.googleapis.com/gmail/v1/users/' + emailFrom + '/messages/send';

I'm using Url := 'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/messages/send';

 

You need to use the Upload URI, for media upload requests to post this as raw message.

See here https://developers.google.com/gmail/api/reference/rest/v1/users.messages/send

There is a difference with /gmail/v1 and /upload/gmail/v1.

 

If I use your Url I also get an invalid JSON error message 400.

So use the other Url and it should be fine.

 

thanks a million.

I now got it to work. the main points seem to be  'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/messages/send'; and  HTTP.Headers.Add('Accept: application/json');

it now works.

however when i try with an attachment - it does send, but the receiver does not get it as an attachment, but a load of base64 text within the email.

any further ideas?

I really appreciate your help.

[JLG 0]

4 minutes ago, JLG said:

thanks a million.

I now got it to work. the main points seem to be  'https://www.googleapis.com/upload/gmail/v1/users/' + gOAuth2.email + '/messages/send'; and  HTTP.Headers.Add('Accept: application/json');

it now works.

however when i try with an attachment - it does send, but the receiver does not get it as an attachment, but a load of base64 text within the email.

any further ideas?

I really appreciate your help.

I really sorted it now.

I needed the following line

IdMessage.ContentType := 'multipart/mixed';

 

phew.

thanks to all of you out there!!

 

[rvk 33]

Glad it works now 👍

[Remy Lebeau 1396]

21 hours ago, JLG said:

I really sorted it now.

I needed the following line

IdMessage.ContentType := 'multipart/mixed';

If you use the TIdMessageBuilder... classes, such as TIdMessageBuilderPlain or TIdMessageBuilderHtml (https://www.indyproject.org/2008/01/16/new-html-message-builder-class/), they will populate the TIdMessage for you.  If you populate the TIdMessage manually, then yes, you are responsible for populating it correctly (ie https://www.indyproject.org/2005/08/17/html-messages/).

[John Kouraklis 94]

Guys, the official way to interact with GMAIL is via OAuth authentication and JSON. That's what the documentation says.

 

The app password, 2FA or the IMAP settings are there but they depend on the user to allow and set up the parameters. 

 

 

 

Edited December 21, 2021 by John Kouraklis

[imperyal 2]

I ended up creating a TCustomAuthenticator descendent class for this. 

It implements a OAuth2 Authenticator:

- Allows authorization via externel browser

- Uses PKCE flow for added security

- Generates new tokens when they expire (using the refresh token)

 

In the repository there's a Demo app too.

 

Feel free to use it if you like:

delphi-google-oauth2 repo

Edited May 31, 2022 by imperyal

[Remy Lebeau 1396]

On 12/21/2021 at 9:25 AM, John Kouraklis said:

Guys, the official way to interact with GMAIL is via OAuth authentication and JSON. That's what the documentation says.

FYI, this morning I checked in a new 'sasl-oauth' branch in Indy's repo, which includes a new 'IdSASLOAuth.pas' unit for SASL classes for OAUTH10A, OAUTHBEARER, and XOAUTH2 for TIdDICT, TIdPOP3, TIdSMTP, and TIdIMAP4.  They are still a work in progress (ie, no parsing of response JSON yet), and you are responsible for obtaining the OAuth tokens externally (ie, over HTTP), but once you have the tokens then you can use these SASLs to login to the DICT/POP3/SMTP/IMAP servers.

Edited June 1, 2022 by Remy Lebeau

[mvanrijnen 123]

 

Ok, i have a question, in this case i adapted our sending servers to use oauth.

We are using office365/exchange online.

But the clientsecrets generated in the Azure AD (AppRegistrations), what use are they? 

* I only have to use the application-id, and an account+password to generate the token. 

This generated token i put in a config for later use, but in the method i use now, the clientsecrets are never used? 

 

[Brian Evans 105]

1 hour ago, mvanrijnen said:

 

Ok, i have a question, in this case i adapted our sending servers to use oauth.

We are using office365/exchange online.

But the clientsecrets generated in the Azure AD (AppRegistrations), what use are they? 

* I only have to use the application-id, and an account+password to generate the token. 

This generated token i put in a config for later use, but in the method i use now, the clientsecrets are never used? 

 

One place they are used in OAuth 2.0 is when exchanging a refresh token for a new access token. How long refresh tokens and access tokens last depend on the web service being accessed. 
One example would be  : Refresh Tokens with OAuth 2.0 - LinkedIn | Microsoft Docs

 

Edited August 25, 2022 by Brian Evans

[mvanrijnen 123]

Yes, so i mixed up some things. 

I was using OAuth2 as user identification, so with the login username/password and MFA authentication, then i got the token and logged in with that token.

 

As i'm adapting some service applications,  we have to use client_secret

So i'm using the code from @Geoffrey Smith as starting point.

I'v adapted it and i get a access token now, with the client_secret (grantype client_credentials), only thing is that the idSMTP.Authenticate does not authenticate with the returned access token. 

(Authentication unsuccessfull).

 

some hints would be great ! 🙂

 

maybe i gonna try the sasl-oauth branch from @Remy Lebeau, but i'm a little scared that i then get all kinds of bumps to get another indy branche working. 

(but i will try this out).

 

btw, is i use the access token generated by method with user/passw + 2fa, then i can send with smtp,

 

found out why i have no success:

on the doc page, where they explain smtp  oauth2 with client_credentials;

Authenticate an IMAP, POP or SMTP connection using OAuth | Microsoft Docs

Quote

Note As per the current test with SMTP Oauth 2.0 client credential flow with non-interactive sign in is not supported.

 

 

 

Edited August 26, 2022 by mvanrijnen

[Remy Lebeau 1396]

Quote

maybe i gonna try the sasl-oauth branch from @Remy Lebeau, but i'm a little scared that i then get all kinds of bumps to get another indy branche working.

That branch is still a work in progress, there has been some hiccups in it this week related to Microsoft's use of OAuth2 (see the discussion on https://github.com/IndySockets/Indy/issues/192).

 

Edited August 26, 2022 by Remy Lebeau

[mvanrijnen 123]

ah, ok. did not had a change to try it.

but we decided to go over to use the MS Graph API for outgoing emails.

(as we can configure which email sending mechanism our sending services use, so could also be reverted back to smtp when needed).

 

[mvanrijnen 123]

On 8/28/2022 at 2:04 PM, mvanrijnen said:

ah, ok. did not had a change to try it.

but we decided to go over to use the MS Graph API for outgoing emails.

(as we can configure which email sending mechanism our sending services use, so could also be reverted back to smtp when needed).

 

got it working, just a reminder for people struggling with the large attachments and getting an invalid Content-Range header error:

 

instead:

hdr := 'bytes 0..199/200'; // example value

MyRequest.AddParameter('Content-Range',  hdr, TRESTRequestParameterKind.pkHTTPHEADER);

 

use:

MyRequest.AddParameter('Content-Range',  hdr, TRESTRequestParameterKind.pkHTTPHEADER, [TRESTRequestParameterOption.poDoNotEncode]);

 

(took only a few hours struggling 🙂, putting a proxy behind it, and see it in 1 minute )/

 

 

 

 

Edited August 30, 2022 by mvanrijnen

[J23 0]

Hello,
i tried to run @Geoffrey Smith gmailauthsmtp in my Delphi 2010 but failed.
I know that D2010 is a very outdated version. I don't have a REST library.
I tried to install "MARS-master" 
https://github.com/andrea-magni/MARS
but it doesn't support versions older than Delphi XE.
Maybe there is some way to enable OAuth2 in Delphi 2010?
Thanks in advance for your advice.

J23
 

Edited November 22, 2022 by J23

[Remy Lebeau 1396]

3 hours ago, J23 said:

i tried to run @Geoffrey Smith gmailauthsmtp in my Delphi 2010 but failed.

Failed how, exactly?

3 hours ago, J23 said:

Maybe there is some way to enable OAuth2 in Delphi 2010?

Since you are posting this in an Indy forum, I'll give you an Indy answer -

 

You can either turn on 2-Factor Authentication in your Google account and then create an App-Specific password for Indy to use, thus not requiring OAuth2 and no code changes or upgrades needed to use TIdSMTP/TIdPOP3/TIdIMAP4.

 

Or, if you must use OAuth2, then install the latest Indy OAuth2 branch code (https://github.com/IndySockets/Indy/tree/sasl-oauth) into your Delphi 2010 (which Indy still supports), replacing any Indy version you may already have installed (https://github.com/IndySockets/Indy/wiki/Updating-Indy), and then you can use TIdHTTP to retrieve an access token from Google, and use TIdSASLXOAuth2 to use that token with TIdSMTP/TIdPOP3/TIdIMAP4.

Edited November 22, 2022 by Remy Lebeau

[imperyal 2]

12 hours ago, J23 said:

Hello,
i tried to run @Geoffrey Smith gmailauthsmtp in my Delphi 2010 but failed.

 

Hello, 

 

You can try this: delphi-google-oauth2

 

Not tested on Delphi 2010, but you can give it a try. There's a Demo in the repo, just open it and set your ClientID and ClientSecret.

 

 

 

[J23 0]

1 hour ago, imperyal said:

Not tested on Delphi 2010, but you can give it a try.

Thank you very much for your quick reply. Unfortunately, there are many problems:

 

1. {$HPPEMIT LINKUNIT}    - [DCC Error] U_DCS_OAuth2.pas(23): E1030 Invalid compiler directive: 'HPPEMIT'

and uses:

2.   Data.Bind.ObjectScope, - [DCC Fatal Error] U_DCS_OAuth2.pas(31): F1026 File not found: 'Data.Bind.ObjectScope.dcu'

...

3. REST.Client,                     - [DCC Fatal Error] U_DCS_OAuth2.pas(33): F1026 File not found: 'REST.Client.dcu'

and so on.

 

J23