Jump to content
Kyle_Katarn31

Broken proxy support with ICS 8.61

Recommended Posts

Hello

 

I've updated my app from ICS 8.59 to 8.61and the communication is no longer working when using a proxy (problem reported by one user).

Proxy is reporting error in credentials or network. 

Switching back to the old version built on 8.59 works fine again. No other change in code.

Is it a known problem with 8.61 ? Could 8.62 work better ?

Any additional technical info needed for troubleshooting ?

 

Thanks for your help !

Edited by Kyle_Katarn31

Share this post


Link to post

I know HTTP proxy support works in V8.62, since I improved it a few months ago and use it regularly myself, to the ICS proxy server.  I simplified the way it is configured so you can use a single URL rather than four separate fields, per the release notes I published here recently. 

 

So what proxy protocol are you using?  Is this HTTP or HTTPS?  Is there any redirection involved?  Where are the protocol logs?

 

Angus

 

Share this post


Link to post

Good to read !

Where are the release note explaning the new way to use it ?

I'll test with 62 and I'll come back. 

How can I record protocol logs ?

Share this post


Link to post

A message here on 12th August explained enough.  To record the HTTP protocol, you add logging to component events, onCommand and onHeaderData in particular. 

 

But the better way is to change TSslHttpCli to TSslHttpRest which is effectively a higher level version that handles lots of stuff you have to plug into the former manually, like cookies, compression, SSL context, and logging.  Look at http://wiki.overbyte.eu/wiki/index.php/FAQ_Using_TSslHttpRest

 

Angus

 

Share this post


Link to post

I'm back with some logs !

Some context :

- works correctly when built with 8.59 from all machines from the network

- works correctly when built with 8.62 from some machine on the network but not all (but all using the same proxy and proxy settings). When fail, it faiuls systematically with the headers below.

 

COMMANDS :

Verbose    [19-08-27 09:38:04]    Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEgAAAAYABgAYAAAAAAAAABAAAAAAAAAAEAAAAAIAAgAQAAAAAAAAAB4AAAABoIAADI3NTYxMVBDQdj/HtneDSpgYGj/Okmg/wONjK3AAZefwUBBHqQfc9vYh+Xzfyhfblng/l0ZcX9z
Verbose    [19-08-27 09:38:04]    Host: www.kc-softwares.com
Verbose    [19-08-27 09:38:04]    User-Agent: Mozilla/5.0
Verbose    [19-08-27 09:38:04]    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Verbose    [19-08-27 09:38:04]    Connection: Keep-alive
Verbose    [19-08-27 09:38:04]    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Verbose    [19-08-27 09:38:04]    GET http://www.kc-softwares.com/sumo/check.php HTTP/1.0

 

Received Headers :

Verbose    [19-08-27 09:38:03]    Content-Length: 921
Verbose    [19-08-27 09:38:03]    Connection: close
Verbose    [19-08-27 09:38:03]    Proxy-Connection: close
Verbose    [19-08-27 09:38:03]    Content-Type: text/html; charset=utf-8
Verbose    [19-08-27 09:38:03]    Pragma: no-cache
Verbose    [19-08-27 09:38:03]    Cache-Control: no-cache
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: BASIC realm="Realm_IWA_BRKRZLCL"
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: NTLM
Verbose    [19-08-27 09:38:03]    Proxy-Authenticate: NEGOTIATE
Verbose    [19-08-27 09:38:03]    HTTP/1.1 407 Proxy Authentication Required

Edited by Kyle_Katarn
context

Share this post


Link to post

Ideally you want to compare the logs to see any differences between 8.59 and 8.62 on the same PC.  Less ideal, between working and non-working PC s on V8.62. 

 

What authentication are you trying to use?  The server seems to accept basic which is simple, you are using NTLM which is complicated and rarely tested.  Certainly never tested NTLM through a proxy.

 

Angus

Share this post


Link to post

Is NTLM something I can change on my side (client) or does this comes from server configuration ? I kept TSSLHttpsli default settings unchanged (eg : no specific code to "force" NTLM afaik)

Share this post


Link to post

I'm not really sure how authentication works with proxies, there are separate authentication properties for server and proxy, both default to off.  Not sure which causes Authorization: NTLMto be added or if it;'s different for a proxy.  You must set ProxyAuth to httpAuthBasic, to use Basic and supply a proxy login.

 

Angus

Share this post


Link to post
1 hour ago, Angus Robertson said:

I'm not really sure how authentication works with proxies, there are separate authentication properties for server and proxy, both default to off.  Not sure which causes Authorization: NTLMto be added or if it;'s different for a proxy.  You must set ProxyAuth to httpAuthBasic, to use Basic and supply a proxy login.

 

Angus

OK, thanks, i'll test.

 

Meanwhile, isn't my issue close to this "old" discussion ? https://www.mail-archive.com/search?l=twsocket@elists.org&q=subject:"Re%3A+[twsocket]+THttpCli+NTLM+OverbyteIcsBcb100Package"&o=newest&f=1

See message from 2011



It works fine for No Basic Authentication but when I configure for NTLM one of my clients is reporting a failure i.e. cannot download You should realy move to v7, v6 is obsolete and not maintained for years now. Some bugs have been fixed in the NTLM code since then, as I already wrote in a previous PM.

Edited by Kyle_Katarn

Share this post


Link to post
5 minutes ago, Angus Robertson said:

Sorry, not interested how ICS V6 behaved, it has changed a little since then.

 

Angus

OK.

I've sent to my users 2 test apps :

- One with 8.59 and Command/Headers logging. Will allow to see if it behaves differently than 8.62

- One with 8.62 forcing httpAuthBasic instead of httpAuthNone

 

I'll share the results here.

Share this post


Link to post

8.59 and Command/Headers logging : Same traces (407....) then ... gets working after a few failed requests

8.62 with  httpAuthBasic : works fine.

 

so there's definitively something going wrong with AuthNTLM implementation... that got worse from 8.59 to 8.62. Maybe the change mentionned here :

 

Share this post


Link to post

I forgot to mention that these errors are received in headers (in early 8.59 comms and all 8.62 if not forced to Basic)

 

Verbose    [19-08-27 18:12:11]    Connection: Keep-Alive
Verbose    [19-08-27 18:12:11]    Proxy-Connection: Keep-Alive
Verbose    [19-08-27 18:12:11]    Content-Type: text/html; charset=utf-8
Verbose    [19-08-27 18:12:11]    Pragma: no-cache
Verbose    [19-08-27 18:12:11]    Cache-Control: no-cache
Verbose    [19-08-27 18:12:11]    Proxy-Authenticate: NTLM TlRMTVNTUAACAAAACgAKADgAAAAFgoECWW+CVaJTfpoAAAAAAAAAAIIAggBCAAAABgOAJQAAAA9CAFIASwBSAFoAAgAKAEIAUgBLAFIAWgABAA4ATQBSAFoAQQBEADAANAAEABIAQgBSAEsAUgBaAC4ATABDAEwAAwAiAE0AUgBaAEEARAAwADQALgBCAFIASwBSAFoALgBMAEMATAAFABIAQgBSAEsAUgBaAC4ATABDAEwABwAIAPeN8i7yXNUBAAAAAA==
Verbose    [19-08-27 18:12:11]    HTTP/1.1 407 Proxy Authentication Required

 

Share this post


Link to post

Before I comment any further here, I need you to actually explain what your application is trying to do, you have not done so despite me asking.  I was the first to mention NTLM.

 

So two stages here, exactly what proxy configuration, all the actual settings and authentication, xxx where necessary, then what request and authentication.

 

Please bear in mind that NTLM is messy and very hard to test, it's generally only used by private Windows servers that are rarely available for testing.  Microsoft probably also messes with it in new server versions.  So we only look at if we test it, or rely on contributors to test and fix it.  

   

Angus

 

Share this post


Link to post

The application in question "read" the output of a PHP script running on my server

I use the "proxy" settings (host, port, username, pass) and all other default settings.

 

I never had any issue with ICS comms until I've implemented 8.62 (i generally take every update you make) because of the proxy issues reported above.

 

Sorry for my partial extract from the logs... which comes from a single customer (see scenarii above) the first "block" of error 407 is common to 8.59 (which... finally work after a few attemps) and 8.62 (which systematically fail). With 8.62, when it "fails" in the end, the message was specifically pointing NTLM, which means that for some reason, even with the default value of AuthNone, it ends in branching in NTLM authentication.

 

When "forcing" BasicAuth, everything works fine (but i'll call for logs to check in details).

Share this post


Link to post

When forcing "Basic", no single error407, works perfect.

 

COMMAND

Verbose    [19-08-28 08:19:16]    Host: www.kcsoftwares.com
Verbose    [19-08-28 08:19:16]    User-Agent: Mozilla/5.0
Verbose    [19-08-28 08:19:16]    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Verbose    [19-08-28 08:19:16]    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Verbose    [19-08-28 08:19:16]    GET https://www.kcsoftwares.com/sumo/check.php HTTP/1.1
Verbose    [19-08-28 08:19:16]    Proxy-Authorization: Basic QXBwSW50ZXJuZXQ6U29tbWVyMjAwNw==
Verbose    [19-08-28 08:19:16]    Content-Length: 0
Verbose    [19-08-28 08:19:16]    Proxy-Connection: Keep-Alive
Verbose    [19-08-28 08:19:16]    Host: www.kcsoftwares.com
Verbose    [19-08-28 08:19:16]    User-Agent: Mozilla/5.0
Verbose    [19-08-28 08:19:16]    CONNECT www.kcsoftwares.com:443 HTTP/1.1

 

HEADERS

Verbose    [19-08-28 08:19:17]    Connection: keep-alive
Verbose    [19-08-28 08:19:17]    Content-Length: 8
Verbose    [19-08-28 08:19:17]    Accept-Ranges: bytes
Verbose    [19-08-28 08:19:17]    X-Cacheable: Cacheable
Verbose    [19-08-28 08:19:17]    X-CDN-Pop-IP: 137.74.120.32/27
Verbose    [19-08-28 08:19:17]    X-CDN-Pop: sbg
Verbose    [19-08-28 08:19:17]    Vary: Accept-Encoding
Verbose    [19-08-28 08:19:17]    X-Request-ID: 238979743
Verbose    [19-08-28 08:19:17]    Set-Cookie: SERVERID102297=24048; path=/; max-age=900
Verbose    [19-08-28 08:19:17]    Content-Type: text/html; charset=UTF-8
Verbose    [19-08-28 08:19:17]    Date: Wed, 28 Aug 2019 06:19:16 GMT
Verbose    [19-08-28 08:19:17]    HTTP/1.1 200 OK

Share this post


Link to post

So to sum up, if you tell ICS to connect to a proxy it works OK, if you don't tell it to do so, it does not.   So works as designed.  There is no automatic proxy detection.

 

Angus

 

Share this post


Link to post

To sum up :

- Default settings + Proxy Host + Proxy Port + Proxy Credentials in 8.59 : hickups (407) then works fine

- Default settings + Proxy Host + Proxy Port + Proxy Credentials in 8.62 : Do no work (407 / NTLM), in the specific context of my user

- BasicAuth + Proxy Host + Proxy Port + Proxy Credentials in 8.62 : works fine

 

 

Share this post


Link to post

The main issue here is that if you don't specify a proxy authentication type but do specify credentials, ICS attempts to specify the type for you based on response types, 401, 407, etc, but this is not well documented and may change by accident when other changes are made. 

 

So really it's pot luck, safer to specify the authentication type you require so it's sent with the initial request, rather than after a 40x response.  Also, just because you specify authentication does not mean the proxy or server needs it, which cab confuse testing.

 

Angus 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×