Broken proxy support with ICS 8.61

[Kyle_Katarn31 0]

Hello

 

I've updated my app from ICS 8.59 to 8.61and the communication is no longer working when using a proxy (problem reported by one user).

Proxy is reporting error in credentials or network. 

Switching back to the old version built on 8.59 works fine again. No other change in code.

Is it a known problem with 8.61 ? Could 8.62 work better ?

Any additional technical info needed for troubleshooting ?

 

Thanks for your help !

Edited August 21, 2019 by Kyle_Katarn31

[Angus Robertson 577]

There are several different proxy options in different components. 

 

Angus

 

[Kyle_Katarn31 0]

HTTP Client. Do you need more details ? (Code extract). Any change expected from a source code perspective when moving from 59 to 61 ?

[Angus Robertson 577]

I know HTTP proxy support works in V8.62, since I improved it a few months ago and use it regularly myself, to the ICS proxy server.  I simplified the way it is configured so you can use a single URL rather than four separate fields, per the release notes I published here recently. 

 

So what proxy protocol are you using?  Is this HTTP or HTTPS?  Is there any redirection involved?  Where are the protocol logs?

 

Angus

 

[Kyle_Katarn31 0]

Good to read !

Where are the release note explaning the new way to use it ?

I'll test with 62 and I'll come back. 

How can I record protocol logs ?

[Angus Robertson 577]

A message here on 12th August explained enough.  To record the HTTP protocol, you add logging to component events, onCommand and onHeaderData in particular. 

 

But the better way is to change TSslHttpCli to TSslHttpRest which is effectively a higher level version that handles lots of stuff you have to plug into the former manually, like cookies, compression, SSL context, and logging.  Look at http://wiki.overbyte.eu/wiki/index.php/FAQ_Using_TSslHttpRest

 

Angus

 

[Kyle_Katarn31 0]

Thanks, I'll give this a look and will come back with result and / or questions 🙂

[Kyle_Katarn 1]

I'm back with some logs !

Some context :

- works correctly when built with 8.59 from all machines from the network

- works correctly when built with 8.62 from some machine on the network but not all (but all using the same proxy and proxy settings). When fail, it faiuls systematically with the headers below.

 

COMMANDS :

Verbose    [19-08-27 09:38:04]    Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEgAAAAYABgAYAAAAAAAAABAAAAAAAAAAEAAAAAIAAgAQAAAAAAAAAB4AAAABoIAADI3NTYxMVBDQdj/HtneDSpgYGj/Okmg/wONjK3AAZefwUBBHqQfc9vYh+Xzfyhfblng/l0ZcX9z
Verbose    [19-08-27 09:38:04]    Host: www.kc-softwares.com
Verbose    [19-08-27 09:38:04]    User-Agent: Mozilla/5.0
Verbose    [19-08-27 09:38:04]    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Verbose    [19-08-27 09:38:04]    Connection: Keep-alive
Verbose    [19-08-27 09:38:04]    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Verbose    [19-08-27 09:38:04]    GET http://www.kc-softwares.com/sumo/check.php HTTP/1.0

 

Received Headers :

Verbose    [19-08-27 09:38:03]    Content-Length: 921
Verbose    [19-08-27 09:38:03]    Connection: close
Verbose    [19-08-27 09:38:03]    Proxy-Connection: close
Verbose    [19-08-27 09:38:03]    Content-Type: text/html; charset=utf-8
Verbose    [19-08-27 09:38:03]    Pragma: no-cache
Verbose    [19-08-27 09:38:03]    Cache-Control: no-cache
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: BASIC realm="Realm_IWA_BRKRZLCL"
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: NTLM
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: NEGOTIATE
Verbose    [19-08-27 09:38:03]    HTTP/1.1 407 Proxy Authentication Required

Edited August 27, 2019 by Kyle_Katarn
context

[Angus Robertson 577]

Ideally you want to compare the logs to see any differences between 8.59 and 8.62 on the same PC.  Less ideal, between working and non-working PC s on V8.62. 

 

What authentication are you trying to use?  The server seems to accept basic which is simple, you are using NTLM which is complicated and rarely tested.  Certainly never tested NTLM through a proxy.

 

Angus

[Kyle_Katarn 1]

Is NTLM something I can change on my side (client) or does this comes from server configuration ? I kept TSSLHttpsli default settings unchanged (eg : no specific code to "force" NTLM afaik)

[Kyle_Katarn 1]

Meanwhile, i'll try to get same logs with 8.59

[Angus Robertson 577]

I'm not really sure how authentication works with proxies, there are separate authentication properties for server and proxy, both default to off.  Not sure which causes Authorization: NTLMto be added or if it;'s different for a proxy.  You must set ProxyAuth to httpAuthBasic, to use Basic and supply a proxy login.

 

Angus

[Kyle_Katarn 1]

1 hour ago, Angus Robertson said:

I'm not really sure how authentication works with proxies, there are separate authentication properties for server and proxy, both default to off.  Not sure which causes Authorization: NTLMto be added or if it;'s different for a proxy.  You must set ProxyAuth to httpAuthBasic, to use Basic and supply a proxy login.

 

Angus

OK, thanks, i'll test.

 

Meanwhile, isn't my issue close to this "old" discussion ? https://www.mail-archive.com/search?l=twsocket@elists.org&q=subject:"Re%3A+[twsocket]+THttpCli+NTLM+OverbyteIcsBcb100Package"&o=newest&f=1

See message from 2011

It works fine for No Basic Authentication but when I configure for NTLM one of my clients is reporting a failure i.e. cannot download You should realy move to v7, v6 is obsolete and not maintained for years now. Some bugs have been fixed in the NTLM code since then, as I already wrote in a previous PM.

Edited August 27, 2019 by Kyle_Katarn

[Angus Robertson 577]

Sorry, not interested how ICS V6 behaved, it has changed a little since then.

 

Angus

[Kyle_Katarn 1]

5 minutes ago, Angus Robertson said:

Sorry, not interested how ICS V6 behaved, it has changed a little since then.

 

Angus

OK.

I've sent to my users 2 test apps :

- One with 8.59 and Command/Headers logging. Will allow to see if it behaves differently than 8.62

- One with 8.62 forcing httpAuthBasic instead of httpAuthNone

 

I'll share the results here.

[Kyle_Katarn 1]

8.59 and Command/Headers logging : Same traces (407....) then ... gets working after a few failed requests

8.62 with  httpAuthBasic : works fine.

 

so there's definitively something going wrong with AuthNTLM implementation... that got worse from 8.59 to 8.62. Maybe the change mentionned here :

 

[Kyle_Katarn 1]

I forgot to mention that these errors are received in headers (in early 8.59 comms and all 8.62 if not forced to Basic)

 

Verbose    [19-08-27 18:12:11]    Connection: Keep-Alive
Verbose    [19-08-27 18:12:11]    Proxy-Connection: Keep-Alive
Verbose    [19-08-27 18:12:11]    Content-Type: text/html; charset=utf-8
Verbose    [19-08-27 18:12:11]    Pragma: no-cache
Verbose    [19-08-27 18:12:11]    Cache-Control: no-cache
Verbose    [19-08-27 18:12:11]    Proxy-Authenticate: NTLM TlRMTVNTUAACAAAACgAKADgAAAAFgoECWW+CVaJTfpoAAAAAAAAAAIIAggBCAAAABgOAJQAAAA9CAFIASwBSAFoAAgAKAEIAUgBLAFIAWgABAA4ATQBSAFoAQQBEADAANAAEABIAQgBSAEsAUgBaAC4ATABDAEwAAwAiAE0AUgBaAEEARAAwADQALgBCAFIASwBSAFoALgBMAEMATAAFABIAQgBSAEsAUgBaAC4ATABDAEwABwAIAPeN8i7yXNUBAAAAAA==
Verbose    [19-08-27 18:12:11]    HTTP/1.1 407 Proxy Authentication Required

 

[Angus Robertson 577]

Before I comment any further here, I need you to actually explain what your application is trying to do, you have not done so despite me asking.  I was the first to mention NTLM.

 

So two stages here, exactly what proxy configuration, all the actual settings and authentication, xxx where necessary, then what request and authentication.

 

Please bear in mind that NTLM is messy and very hard to test, it's generally only used by private Windows servers that are rarely available for testing.  Microsoft probably also messes with it in new server versions.  So we only look at if we test it, or rely on contributors to test and fix it.  

   

Angus

 

[Kyle_Katarn 1]

The application in question "read" the output of a PHP script running on my server

I use the "proxy" settings (host, port, username, pass) and all other default settings.

 

I never had any issue with ICS comms until I've implemented 8.62 (i generally take every update you make) because of the proxy issues reported above.

 

Sorry for my partial extract from the logs... which comes from a single customer (see scenarii above) the first "block" of error 407 is common to 8.59 (which... finally work after a few attemps) and 8.62 (which systematically fail). With 8.62, when it "fails" in the end, the message was specifically pointing NTLM, which means that for some reason, even with the default value of AuthNone, it ends in branching in NTLM authentication.

 

When "forcing" BasicAuth, everything works fine (but i'll call for logs to check in details).

[Kyle_Katarn 1]

When forcing "Basic", no single error407, works perfect.

 

COMMAND

Verbose    [19-08-28 08:19:16]    Host: www.kcsoftwares.com
Verbose    [19-08-28 08:19:16]    User-Agent: Mozilla/5.0
Verbose    [19-08-28 08:19:16]    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Verbose    [19-08-28 08:19:16]    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Verbose    [19-08-28 08:19:16]    GET https://www.kcsoftwares.com/sumo/check.php HTTP/1.1
Verbose    [19-08-28 08:19:16]    Proxy-Authorization: Basic QXBwSW50ZXJuZXQ6U29tbWVyMjAwNw==
Verbose    [19-08-28 08:19:16]    Content-Length: 0
Verbose    [19-08-28 08:19:16]    Proxy-Connection: Keep-Alive
Verbose    [19-08-28 08:19:16]    Host: www.kcsoftwares.com
Verbose    [19-08-28 08:19:16]    User-Agent: Mozilla/5.0
Verbose    [19-08-28 08:19:16]    CONNECT www.kcsoftwares.com:443 HTTP/1.1

 

HEADERS

Verbose    [19-08-28 08:19:17]    Connection: keep-alive
Verbose    [19-08-28 08:19:17]    Content-Length: 8
Verbose    [19-08-28 08:19:17]    Accept-Ranges: bytes
Verbose    [19-08-28 08:19:17]    X-Cacheable: Cacheable
Verbose    [19-08-28 08:19:17]    X-CDN-Pop-IP: 137.74.120.32/27
Verbose    [19-08-28 08:19:17]    X-CDN-Pop: sbg
Verbose    [19-08-28 08:19:17]    Vary: Accept-Encoding
Verbose    [19-08-28 08:19:17]    X-Request-ID: 238979743
Verbose    [19-08-28 08:19:17]    Set-Cookie: SERVERID102297=24048; path=/; max-age=900
Verbose    [19-08-28 08:19:17]    Content-Type: text/html; charset=UTF-8
Verbose    [19-08-28 08:19:17]    Date: Wed, 28 Aug 2019 06:19:16 GMT
Verbose    [19-08-28 08:19:17]    HTTP/1.1 200 OK

[Angus Robertson 577]

So to sum up, if you tell ICS to connect to a proxy it works OK, if you don't tell it to do so, it does not.   So works as designed.  There is no automatic proxy detection.

 

Angus

 

[Kyle_Katarn 1]

To sum up :

- Default settings + Proxy Host + Proxy Port + Proxy Credentials in 8.59 : hickups (407) then works fine

- Default settings + Proxy Host + Proxy Port + Proxy Credentials in 8.62 : Do no work (407 / NTLM), in the specific context of my user

- BasicAuth + Proxy Host + Proxy Port + Proxy Credentials in 8.62 : works fine

 

 

[Angus Robertson 577]

The main issue here is that if you don't specify a proxy authentication type but do specify credentials, ICS attempts to specify the type for you based on response types, 401, 407, etc, but this is not well documented and may change by accident when other changes are made. 

 

So really it's pot luck, safer to specify the authentication type you require so it's sent with the initial request, rather than after a 40x response.  Also, just because you specify authentication does not mean the proxy or server needs it, which cab confuse testing.

 

Angus 

[Kyle_Katarn 1]

I was not aware of this ! Makes everything much clearer ! Thanks !!

[Kyle_Katarn 1]

Side effect : If you specified host and port but NO credential, should I use AuthNone or would AuthBasic work too ?