Using proxy persistent connection disconnected

[Kenny Phong 0]

Hi,

Normal connection without proxy working perfectly. Using proxy (http://www.squid-cache.org/) will have problem, refer to attachments below.

From my understanding, is persistent connection problem, but i can't figure out what is root cause.

Appreciate for your help. Thanks.

Note: i'm using Delphi 2006 with V8.61.

 

 

Edited June 24, 2019 by Kenny Phong

[Angus Robertson 574]

Which component are you using?  What sort of proxy settings in that component?  Is this SSL or non-SSL?  Have you asked for persistent connections?  No idea where those screen dumps come from, but they are not useful for diagnostics without a lot more explanation.

 

Angus

 

[Kenny Phong 0]

6 minutes ago, Angus Robertson said:

Which component are you using?  What sort of proxy settings in that component?  Is this SSL or non-SSL?  Have you asked for persistent connections?  No idea where those screen dumps come from, but they are not useful for diagnostics without a lot more explanation.

 

Angus

 

I'm using THttpCli

 

Proxy := ProxyServerAddress

ProxyPort := ProxyServerPort

 

Is non-SSL

 

Yes, I asked for persistent connections.

 

Screen dumps came from Wireshark. For screen 1, i was successfully signed in to server. And do GET was successfully in screen 2 and screen 3. Then before do POST in screen 4, i was disconnected and return invalid client.

 

Please let me know if you need more information. Thanks.

Edited June 24, 2019 by Kenny Phong

[Angus Robertson 574]

When the connection works fine bypassing the proxy, why do you think it's an ICS issue?  What do you think ICS is doing differently or incorrectly for a proxy connection? Have you compared the headers IC S sends for proxy/non proxy?  Sorry, don't have time this month to study extracts from dumps.

 

Angus

 

[FPiette 383]

In the trace you provide, I see all GET commands are using HTTP 1.0 which don not support persistent connections as far as I know. You should request version 1.1.

 

[Kenny Phong 0]

On 6/24/2019 at 5:10 PM, Angus Robertson said:

When the connection works fine bypassing the proxy, why do you think it's an ICS issue?  What do you think ICS is doing differently or incorrectly for a proxy connection? Have you compared the headers IC S sends for proxy/non proxy?  Sorry, don't have time this month to study extracts from dumps.

 

Angus

 

The headers ICS sends with or without proxy, is the same.

[Kenny Phong 0]

19 hours ago, FPiette said:

In the trace you provide, I see all GET commands are using HTTP 1.0 which don not support persistent connections as far as I know. You should request version 1.1.

 

i changed it's properties it to

Connection := 'Keep-Alive';
RequestVer := '1.1';

Request version became 1.1 but still failed.

 

Edited June 25, 2019 by Kenny Phong

[FPiette 383]

I don't see in your log that the persistent connection fails. I only see that the server is not happy with the request ("401 Invalid client!").

If the request succeed you get a 200 response code, not a 401. You should first resolve that. Maybe a cookie is required or some specific header lines.

 

[Angus Robertson 574]

401 error is usually authentication failure, but since no proper headers are posted in this thread we can not comment on what authentication is being attempted.  Perhaps the server is sufficiently clever to realise the requests have been intercepted and perhaps corrupted by a proxy, thus 401.

 

The newish TSslHttpRest component has header and response logging built-in, and can be used instead of TSslHttpCli, just add the OnHttpRestProg event. 

 

Angus

 

[Kenny Phong 0]

10 hours ago, Angus Robertson said:

401 error is usually authentication failure, but since no proper headers are posted in this thread we can not comment on what authentication is being attempted.  Perhaps the server is sufficiently clever to realise the requests have been intercepted and perhaps corrupted by a proxy, thus 401.

 

The newish TSslHttpRest component has header and response logging built-in, and can be used instead of TSslHttpCli, just add the OnHttpRestProg event. 

 

Angus

 

I change THttpCli to TSslHttpRest and on event OnHttpRestProg write Msg to log file.

Below is what i get:

 

26/06/2019 09:20:25.378: 
Request completed: 200 OK

 

And i try to get log using Web Client Demo in ICS sample.

 

This is what i get:

Connecting to: http://222.22.222.222:20000/[s123abc]Connect?&UseEncryption=1|||
09:26:47:360 State = httpNotConnected
09:26:47:367 Login 127.0.0.1
09:26:47:372 State = httpDnsLookup
09:26:47:380 Socket DNS Lookup Done - 127.0.0.1
09:26:47:386 State = httpDnsLookupDone
09:26:47:393 connect to 127.0.0.1/3128
09:26:47:403 024BA470 Socket handle created handle=732
09:26:47:415 TWSocket will connect to 127.0.0.1:3128
09:26:47:427 SessionConnected, error=0 to 127.0.0.1:3128
09:26:47:432 State = httpConnected
Connected OK to: 127.0.0.1 (127.0.0.1)
09:26:47:447 State = httpWaitingHeader
cmd> GET http://222.22.222.222:20000/[s123abc]Connect?&UseEncryption=1||| HTTP/1.0
cmd> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
cmd> Connection: Keep-Alive
cmd> Accept-Language: en, fr
cmd> User-Agent: Mozilla
cmd> Host: 222.22.222.222:20000
cmd> Proxy-Connection: Keep-Alive
09:26:47:499 7 header lines to send
GET http://222.22.222.222:20000/[s123abc]Connect?&UseEncryption=1||| HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Connection: Keep-Alive
Accept-Language: en, fr
User-Agent: Mozilla
Host: 222.22.222.222:20000
Proxy-Connection: Keep-Alive
cmd> 
09:26:47:511 024BA470 PutDataInSendBuffer handle=732 [1] Data:GET http://222.22.222.222:20000/[s123abc]Connect?&UseEncryption=1||| HTTP/1.0$0D $0A Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*$0D $0A Connection: Keep-Alive$0D $0A Accept-Language: en, fr$0D $0A User-Agent: Mozilla$0D $0A Host: 222.22.222.222:20000$0D $0A Proxy-Connection: Keep-Alive$0D $0A $0D $0A 
09:26:47:517 SendRequest Done
09:26:47:523 024BA470 TriggerDataSent handle=732
09:26:47:528 024BA470 TriggerDataSent handle=732
09:26:47:537 Data available. Len=251
09:26:47:544 >|HTTP/1.1 200 OK|
09:26:47:549 FCloseReq=1
HTTP/1.1 200 OK
09:26:47:559 >|Server: ABCServer|
Server: ABCServer
09:26:47:569 >|Date: Wed Jun 26 09:26:47 MYT 2019|
Date: Wed Jun 26 09:26:47 MYT 2019
09:26:47:578 >|ClientAddr: 45.112.199.3|
ClientAddr: 45.112.199.3
09:26:47:589 >|Content-Type: text/plain;charset=UTF-8|
Content-Type: text/plain;charset=UTF-8
09:26:47:598 >|Content-Length: 0|
Content-Length: 0
09:26:47:608 >|X-Cache: MISS from 12345-abc|
X-Cache: MISS from 12345-abc
09:26:47:617 >|Via: 1.1 12345-abc (squid/3.5.28)|
Via: 1.1 12345-abc (squid/3.5.28)
09:26:47:626 >|Connection: keep-alive|
Connection: keep-alive
09:26:47:637 >||
09:26:47:640 End of header
09:26:47:645 State = httpReady
09:26:47:651 PrepareNTLMAuth begin, FStatusCode = 200 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:47:657 PrepareNTLMAuth end, FStatusCode = 200 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:47:662 RequestDone
Request done, StatusCode #200
hdr>HTTP/1.1 200 OK
hdr>Server: ABCServer
hdr>Date: Wed Jun 26 09:26:47 MYT 2019
hdr>ClientAddr: 45.112.199.3
hdr>Content-Type: text/plain;charset=UTF-8
hdr>Content-Length: 0
hdr>X-Cache: MISS from 12345-abc
hdr>Via: 1.1 12345-abc (squid/3.5.28)
hdr>Connection: keep-alive
09:26:55:870 State = httpClosing
Reset SSL
09:26:55:880 024BA470 SocketCloseCalled handle=732
09:26:55:885 024BA470 TCustomWSocket.Shutdown 1 handle=732
09:26:55:891 SessionClosed Error: 0
09:26:55:897 State = httpReady
09:26:55:903 PrepareNTLMAuth begin, FStatusCode = 200 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:55:907 PrepareNTLMAuth end, FStatusCode = 200 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:55:914 RequestDone
Request done, StatusCode #200
hdr>HTTP/1.1 200 OK
hdr>Server: ABCServer
hdr>Date: Wed Jun 26 09:26:47 MYT 2019
hdr>ClientAddr: 45.112.199.3
hdr>Content-Type: text/plain;charset=UTF-8
hdr>Content-Length: 0
hdr>X-Cache: MISS from 12345-abc
hdr>Via: 1.1 12345-abc (squid/3.5.28)
hdr>Connection: keep-alive
Connecting to: http://222.22.222.222:20000/[s123abc]Sector?[FAST]=KL,SG,MY,SS,SZ,HK,N,O,JK,WIDX,BK,FX,NAR,NM|||
09:26:57:324 State = httpNotConnected
09:26:57:330 Login 127.0.0.1
09:26:57:335 State = httpDnsLookup
09:26:57:341 Socket DNS Lookup Done - 127.0.0.1
09:26:57:347 State = httpDnsLookupDone
09:26:57:354 connect to 127.0.0.1/3128
09:26:57:359 024BA470 Socket handle created handle=768
09:26:57:367 TWSocket will connect to 127.0.0.1:3128
09:26:57:378 SessionConnected, error=0 to 127.0.0.1:3128
09:26:57:384 State = httpConnected
Connected OK to: 127.0.0.1 (127.0.0.1)
09:26:57:400 State = httpWaitingHeader
cmd> GET http://222.22.222.222:20000/[s123abc]Sector?[FAST]=KL,SG,MY,SS,SZ,HK,N,O,JK,WIDX,BK,FX,NAR,NM||| HTTP/1.0
cmd> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
cmd> Connection: Keep-Alive
cmd> Accept-Language: en, fr
cmd> User-Agent: Mozilla
cmd> Host: 222.22.222.222:20000
cmd> Proxy-Connection: Keep-Alive
09:26:57:456 7 header lines to send
GET http://222.22.222.222:20000/[s123abc]Sector?[FAST]=KL,SG,MY,SS,SZ,HK,N,O,JK,WIDX,BK,FX,NAR,NM||| HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Connection: Keep-Alive
Accept-Language: en, fr
User-Agent: Mozilla
Host: 222.22.222.222:20000
Proxy-Connection: Keep-Alive
cmd> 
09:26:57:474 024BA470 PutDataInSendBuffer handle=768 [2] Data:GET http://222.22.222.222:20000/[s123abc]Sector?[FAST]=KL,SG,MY,SS,SZ,HK,N,O,JK,WIDX,BK,FX,NAR,NM||| HTTP/1.0$0D $0A Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*$0D $0A Connection: Keep-Alive$0D $0A Accept-Language: en, fr$0D $0A User-Agent: Mozilla$0D $0A Host: 222.22.222.222:20000$0D $0A Proxy-Connection: Keep-Alive$0D $0A $0D $0A 
09:26:57:481 SendRequest Done
09:26:57:486 024BA470 TriggerDataSent handle=768
09:26:57:490 024BA470 TriggerDataSent handle=768
09:26:57:517 Data available. Len=261
09:26:57:521 >|HTTP/1.1 401 Unauthorized|
09:26:57:526 FCloseReq=1
HTTP/1.1 401 Unauthorized
09:26:57:537 >|Server: ABCServer|
Server: ABCServer
09:26:57:547 >|Date: Wed Jun 26 09:26:57 MYT 2019|
Date: Wed Jun 26 09:26:57 MYT 2019
09:26:57:558 >|ClientAddr: 45.112.199.3|
ClientAddr: 45.112.199.3
09:26:57:569 >|Content-Type: text/plain;charset=UTF-8|
Content-Type: text/plain;charset=UTF-8
09:26:57:581 >|Content-Length: 0|
Content-Length: 0
09:26:57:593 >|X-Cache: MISS from 12345-abc|
X-Cache: MISS from 12345-abc
09:26:57:605 >|Via: 1.1 12345-abc (squid/3.5.28)|
Via: 1.1 12345-abc (squid/3.5.28)
09:26:57:617 >|Connection: keep-alive|
Connection: keep-alive
09:26:57:626 >||
09:26:57:633 End of header
09:26:57:640 State = httpReady
09:26:57:646 PrepareNTLMAuth begin, FStatusCode = 401 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:57:652 PrepareNTLMAuth end, FStatusCode = 401 FProxyAuthNTLMState=0 FAuthNTLMState=0
09:26:57:657 RequestDone
Request done, StatusCode #401
hdr>HTTP/1.1 401 Unauthorized
hdr>Server: ABCServer
hdr>Date: Wed Jun 26 09:26:57 MYT 2019
hdr>ClientAddr: 45.112.199.3
hdr>Content-Type: text/plain;charset=UTF-8
hdr>Content-Length: 0
hdr>X-Cache: MISS from 12345-abc
hdr>Via: 1.1 12345-abc (squid/3.5.28)
hdr>Connection: keep-alive
09:28:57:066 Data available. Len=0
09:28:57:088 **data available. Len=0
09:28:57:098 SessionClosed Error: 0
09:28:57:108 024BA470 SocketCloseCalled handle=768
09:28:57:115 024BA470 TCustomWSocket.Shutdown 1 handle=768

[Kenny Phong 0]

15 hours ago, FPiette said:

I don't see in your log that the persistent connection fails. I only see that the server is not happy with the request ("401 Invalid client!").

If the request succeed you get a 200 response code, not a 401. You should first resolve that. Maybe a cookie is required or some specific header lines.

 

Normal connection without proxy is working fine. And i don't know what specific header lines needed.

 

And for your information, i did try MyApp > Proxifier (https://www.proxifier.com/) > Squid > Server, working perfectly.

[Angus Robertson 574]

5 hours ago, Kenny Phong said:

I change THttpCli to TSslHttpRest and on event OnHttpRestProg write Msg to log file.

Below is what i get:

 

26/06/2019 09:20:25.378: 
Request completed: 200 OK

 

You also need to set the DebugLevel property to the level of logging required from THttpDebugLevel, DebugHdr in this case to get all the headers.  Most applications don'r need a high level of logging.

 

Angus

[Kyle_Katarn31 0]

Could your problem be linked to this one ?