New to x509 - how to mimic Postman config using ICS

[KMarb 5]

I have a project to access a corporate API for some integration. I have a client certificate, and using Postman I'm able to retrieve a token for subsequent API calls. I'm having trouble mapping my Postman config over to the ICS demo and was hoping for some guidance. Please see questions below.

 

First - in Postman, they separate parameters into separate lists (params, headers, body). I'm showing the Body page below, but note there are 9 entries in the Headers key/value table as well. Where do I enter those in the ICS demo?

 

 

Here's what I have currently entered in the ICS demo... I'm not sure where to specify if param is for body, header, etc:

 

Next, where do I enter or link to my client cert file? I've done this and it seems to work, but would like confirmation:

 

 

I say I think it works because I click "Start REST Request" and I get this feedback:

 

2/23/2023 1:41:50 PM Starting REST request for URL: https://<myURL>
POST https://<myURL>
{"grant_type":"client_credentials","client_id":"<clientID","client_secret":"client_secret"}
Connected OK to: uat-accounts.adp.com (170.146.92.231)
Client SSL Certificate Sent
<URL> SSL Connected OK with TLSv1.2, cipher DHE-RSA-AES256-SHA, key auth RSA, key exchange DH, encryption AES(256), message auth SHA1
<URL>SSL Chain Verification Succeeded
Request completed: 500 Internal Server Error
Request done, StatusCode 500
{
                "error":"invalid_request",
                "error_description":"Missing or invalid grant_type"
                }
Json main content type: stObject

Connection closed
Sync REST request completed, Status 500
 

With Postman I get this type of response:

 

{

    "access_token": "<access_token>",

    "token_type": "Bearer",

    "expires_in": 3600,

    "scope": "api"

}

[Lars Fosdal 1732]

https://www.charlesproxy.com/ is a great tool for pin-pointing issues with authentication on https connections.

[Angus Robertson 526]

Because you have not shown most of the posted parameters, it's impossible to say exactly what is wrong. 

 

The postman page says there are nine headers necessary, some may be sent already by ICS, others may need adding using 'Extra Headers', where you have typed ExtraHeaders for some reason. it should be headername:headervalue. 

 

Also Json parameters need to be correctly encoded, if you look at the TRestOAuth.GrantAuthToken method in OverbyteIcsSslHttpOAuth.pas, you'll see that when adding the client Id and secret and other parameters they are added with the Raw option so they are not escaped as Json strings, in the GUI you should replace RPTypeStr with RPTypeObj so that the values are not quoted. 

 

Angus

 

[KMarb 5]

Understood and thank you. Here are my (8 actually) header parameters in Postman. Do I need to add any of these to Extra Headers?

 

 

And here are the settings in Postman for the request:

 

I'm not sure how those settings and other parameters need to be mapped to ICS to get my request to work. Your feedback is really appreciated.

 

 

[Angus Robertson 526]

I would fix the Json parameters first, that was the "Missing or invalid grant_type" error you received, then worry about headers, most of which are standard.

 

Angus

 

[KMarb 5]

I've tried many variations for the parameters but continue to get the error. I also tried listing the full parameters in the raw parameters field but that also did not work.

 

 

Connected OK to: <URL>
Request completed: 500 Internal Server Error
Request done, StatusCode 500
{
                "error":"invalid_request",
                "error_description":"Missing or invalid grant_type"
                }
Json main content type: stObject
 

If tried with all 3 params as RPTypeObj. Same error.

 

What options affect this? Do I need to consider any pages other than the first two?

 

 

If it helps, here is the code generated for HTTP from Postman:

POST /auth/oauth/v2/token HTTP/1.1
Host: <URL>
Content-Type: application/x-www-form-urlencoded
Content-Length: 127

grant_type=client_credentials&client_id=<clientID>&client_secret=<clientSecret>

[KMarb 5]

Getting a different error now (changed REST Content)... not sure if this is progress or not. Please review and advise and thank you (a lot, really):

 

 

 

Here is the response - it is no longer complaining about the grant_type param, so maybe this is moving the right direction?:

 

Found Cached OCSP Status
OCSP Status Response: successful
<URL> Certificate OCSP Status Valid, Not Revoked
Request completed: 401 Unauthorized
Request done, StatusCode 401
{
  "error": "invalid_request",
  "error_description": "proper client ssl certificate was not presented"
}

 

 

[Angus Robertson 526]

Okay, so the REST server does not want Json but POST with REST Content 'URL, Encoded Body'. 

 

Angus

 

[KMarb 5]

I continue to get same error:

 

{
  "error": "invalid_request",
  "error_description": "proper client ssl certificate was not presented"
}

 

I'm guessing my pem file is lacking. In Postman I have this:

 

 

Given the error, proper client ssl cert was not presented, I think I need to combine the postman .pem and .key files into a single .pem for ICS.

 

All this is new to me, just learning what I need to get a job done. I will read up on how to create the pem needed for ICS and see if that gets things working.

 

I will post back later with status and maybe more questions. Thank you very much!

I'm wondering if my pem file is incorrect. You had suggested that I could create a pem file from my CRT and

[KMarb 5]

Why does the ICS Authentication Login form show sometimes? I cannot see the pattern of conditions that causes that dialog to show or not.

 

Everything I try now returns the error, "proper client ssl certificate was not presented"
 

I took my .key file and my .pem file (with 2 certs) and concatenated, in that order, to create a single pem file that I thought would be what ICS expects.

 

My situation is, this is a one off for me, the need to integrate with a corporate API. I don't live in the world of SSL certificates and REST. I say this because, if I could hire someone for an hour to review my specific situation and help me get over the hump (which is to make a successful request to get the bearer token, using ICS), I can handle the rest of it. Since I have Postman working, it should be straightforward to translate to ICS. My problem - I don't know where to hit the machine or how hard to hit it.

 

You are volunteering your time, and I am very appreciative of that. Not knocking this approach at all. But, since I'm so uninformed, and don't have the time to become fully informed about this world right now, do you know if there is someone I can hire to look at my Postman config and translate what I need to ICS?

[FPiette 367]

Since you made it working with Postman, I suggest you use WireShark or other network analyser software to dump the Postman request and then dump the request you send with ICS. Comparing the two will give you a good idea about what is wrong.

[Angus Robertson 526]

You get different errors as you correct each different problem.

 

ICS will raise an authentication window for a 401 error, at that point it has no idea what authentication scheme is needed.

 

You should build the ICS OverbyteIcsPemtool sample and try and open your PEM bundle with the View Bundle File button on the first tab, it should show the certificate details and whether there is a matching private key. 

 

Angus

 

[Lars Fosdal 1732]

Again, Charlesproxy handles http and https and will give full insight in what Postman and your app sends - url, body and headers. 

I used it myself to figure out what a C# app did with an OAuth2, that my Delphi app didn't do.

 

The trial version is functional, but annoying (no copying, pauses at odd times) - but a full license is reasonably priced

[KMarb 5]

Thank you. I will try Charlesproxy. the problem for me is that I don't know this world, I will need to learn charlesproxy and wireshark. So visiting more forums, more faqs, etc... Was hoping there was a shorter route from Postman to ICS.

[Lars Fosdal 1732]

Charlesproxy is easy to grasp. 

[KMarb 5]

Mr. Angus, I used the sample pem tool bundle file viewer. You said I could find out if I have a matching key. How do I determine that.

 

I don't know what info is sensitive and should remain private, so I've removed anything that might be so. Here is part of the results when I view the pem file:

GENERAL
Serial Number: <sn>
Issued on (UTC): 2/4/2023
Expires on (UTC): 2/3/2025 11:59:59 PM
Basic Constraints: CA=FALSE
Key Usage: Digital Signature
Extended Key Usage: TLS Web Client Authentication, Microsoft Smartcard Login
Authority Info, OCSP: <url>
Authority Info, Issuer Cert: <url>.crt
Certificate Policies: Policy: 1.3.6.1.4.1.500.12.1,   CPS: <url>
CRL Distribution Points: Full Name:,   URI:<url>
Authority Key Identifier: AE:...
Subject Key Identifier: 9C:...
Signature Algorithm: sha256WithRSAEncryption
Fingerprint (sha1): d7...
Fingerprint (sha56): e0...
Key Info: RSA Key Encryption 2048 bits, 112 security bits

 

How do I determine if I have a matching key?

 

[Angus Robertson 526]

OverbyteIcsPemTool 'View Bundle File' button will list each certificate and key in the bundle file, so one of mine starts:

 

Certificate file C:\certificates\x_magsys_co_uk-bundle.pem
ISSUED TO (Subject)
Common Name (CN): *.magsys.co.uk
Alt Name (DNS): *.magsys.co.uk, *.magsys.uk, magsys.co.uk, magsys.uk

xxx

ISSUED BY
Common Name (CN): R3
Organisation (O): Let's Encrypt

xxx

 

!! Private key available for certificate: ECDSA Key Encryption prime256v1 256 bits, 128 security bits

 

!! Intermediate certificates: Total 1
#1 Issued to (CN): R3, (O): Let's Encrypt

 

If it does not say !! Private key available, the file can not be used for SSL/TLS client or server, the intermediate is required for most server applications, but probably not for clients. You may need to specify a password to read the private key. 

 

Angus

 

 

 

 

[KMarb 5]

The PemTool does not show a private key. Here is the format of my Pem file, does this look correct?

 

-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
 

[Angus Robertson 526]

Usually the private key is placed after the certificate in the bundle file, otherwise tools would have to look ahead at all certificates in the file. 

 

Angus

 

[KMarb 5]

I've tried multiple variations but the PemTool does not "see" the private key, makes no mention of it. I've tried:

 

-----BEGIN CERTIFICATE----- (cert1)
-----BEGIN PRIVATE KEY-----
-----BEGIN CERTIFICATE----- (cert2)

 

and
 

-----BEGIN CERTIFICATE----- (cert1)
-----BEGIN CERTIFICATE----- (cert2)

-----BEGIN PRIVATE KEY-----
 

There is no mention of the private key in the PemTool demo. Do you have any suggestions?

[Angus Robertson 526]

If the private key contains a public key that matches the public key first certificate, the tools will say so.  If you built the PEM file by hand, it could be failing because you missed a single character somewhere, done that myself.  But without the real certificates, this is all guess work. 

 

You really should ask the API supplier to provide a proper bundle file, ideally PFX/PKCS12 format (which Windows also prefers) which is harder to corrupt.

 

Angus

 

 

 

[KMarb 5]

I have a pkcs7 file with file extension .crt. I am new to all this, so forgive me for asking simple questions, but is there a way to use that file with ICS? I have other flavors of files too, but I'm not sure how to give ICS what it needs... to mimic what I did with Postman which does work, so the problem is with me not knowing the domain of info (ssl authentication mechanism) as well as not knowing how to mimic postman settings in ICS.

 

[Angus Robertson 526]

ICS will open PKCS7 files as well, but they are usually certificates only, no private key.  If you have a P12 file, that is the same as PFX, use it as the bundle.

 

The SslCliCert property of the TSslHttpRest component can be loaded with certificate and private key from separate files, with the LoadFromPEMFile() and PrivateKeyLoadFromPemFile() methods, but this is not done in the sample since a bundle is safer, once you have a bundle. 

 

The PEM Tool will also open all these files and save a bundle file, look at the Create Certificates tab.  There are dozens of fields and buttons because there are so many options, But you just need to specify Cert File, click Load Certificate and it will be displayed in the yellow box, then Prv Key File, click Load Key and it will be displayed, then Inter Certs Fiie, click Load Inters, and you should now see all three components displayed.  At the bottom, specify PEM Certs File and PKCS12 Certs File, tick Save Private Key in Cert File and Save Inter Certs, then click Save PEM Cert and save PKCS12 Cert and you will have two new bundles.  

 

You don't need anything else, but this tool will also create new certificates, private keys and certificate requests, and sign requests to issue private certificates for LANs. It will also install certificates into the Windows store and display the various stores and read the contents.

 

Angus

 

 

[KMarb 5]

I followed your steps and now have 2 new bundles. Thank you. 

 

I am trying the example again with the PKCS12 file and my message when I click "Start Rest Request" is this:

 

---------------------------
ICS HTTPS REST and OAuth Demo
---------------------------
Error reading X509 DER certificate - <filename>.pem.
 

I created this file using the steps you sent.

 

When I created the PEM Cert it gave a message that the private key was None, but the pkcs file seemed okay.

 

Is there anyone you know who would consult with me on a paid basis to get the Postman example that I have working converted over to ICS?

[Angus Robertson 526]

Quote

private key was None

So your real problem is a private key that OpenSSL does not recognise or is for a different certificate. 

 

Without seeing the actual files, no-one can help you.  That is why I suggested asking your supplier for a bundle file so the files can not get mixed up.

 

You can email them to me, original files not the bundle you created and I'll check.

 

Angus