TIcsProxy most often forwards the content sent from target only partially to source

[uso 0]

Hi,

 

I'm trying to implement a TLS wrapping proxy to "secure" a TLS incapable web service,

so the intended information flow is like this:

client -(TLS encrypted data)-> proxy -(plain data)-> server.local:8080
server.local -(plain data)-> proxy -(TLS encrypted data)-> client

In Wireshark I can see that "server.local" sends the response as expected to the proxy,

but I'm unable to get the proxy to return the data as expected to the client.

The proxy does not always return the whole response from "server.local".

Requesting small entities (10000 byte >= response size) seems to work (success.log).

Whereas bigger entities (~16000 byte <= response size) fail (error.log).

 

My proxy config:

[Proxy]
RxBuffSize=65536
MaxClients=999

ServerHeader=Foo Bar Baz
LocalAddr=0.0.0.0
DebugLevel=DebugAll
TarSecLevel=sslSecLevel128bits
CertVerTar=CertVerWinStore
SslReportChain=True

[Source1]
AuthForceSsl=True
AuthSslCmd=True
BindIpAddr=0.0.0.0
BindNonPort=80
BindSslPort=443
Descr=Bar
Enabled=True
Hosts=foo.local
HostTag=FooBarBaz
CertChallange=ChallNone
CertSignDigest=Digest_sha256
CerSupplierProto=SuppProtoNone
CliCertMethod=sslCliCertNone
SslLoadSource=CertWinStoreMachine
SslSrvSecurity=sslSrvSecHigh

[Target1]
Descr=Foo
HostEnabled=True
HostTag=FooBarBaz
IdleTimeout=60
;SrcPath=
TarHost=localhost
TarPort=8080
TarSsl=False
UpdateHtml=True
UpdateHttp=True

Is there anything obvious I'm missing?

 

Used ICS version:

ICS V9.2 - Part 6, Revision 50

error.log

success.log

Edited June 10 by uso

[Angus Robertson 574]

Nothing obvious missing, why did you create a certificate in the Windows store instead of a PEM/PFX that ICS would create for you?  I find it better to test with real IP addresses, your server won't start if there are any web servers running on 80/443. 

 

The two logs look the same except for a couple of values, not sure what they are supposed to show. 

 

All my mail travels through TIcsProxy, and I use the Forward Proxy on my public servers to access my local servers remotely, so the component is well tested, and rarely touched to break it

 

Angus

 

 

[uso 0]

Hello Angus,

 

we are using the Windows cert store functionality because certificates shall be enrolled by AD.

Quote

 I find it better to test with real IP addresses, your server won't start if there are any web servers running on 80/443. 

Sure, but this is not an issue as it can be taken as granted, that port 80, 8080 and 443 on the used interfaces are free for our purposes.

 

Quote

 

The two logs look the same except for a couple of values, not sure what they are supposed to show. 

 

To logs show, that in the case of "success" the amount of bytes received from "target" and sent to "source" are equal:

# [...]

[yyyy-MM-dd hh:56:57.56]
FooBarBaz - Bar (1) Target 0 - Remote closed, Data sent 73, Data recvd 6798

# [...]

[yyyy-MM-dd hh:56:57.56]
FooBarBaz - Bar (1) Source 0 - Client disconnection from 127.0.0.1, Data sent 6798, Data recvd 73

# [...]

In case of "error" there is a difference:

#[...]

[yyyy-MM-dd hh:32:06.32]
FooBarBaz - Bar (1) Target 0 - Remote closed, Data sent 92, Data recvd 48462

#[...]

[yyyy-MM-dd hh:32:06.32]
FooBarBaz - Bar (1) Source 0 - Client disconnection from 127.0.0.1, Data sent 16384, Data recvd 92

#[...]

 

Quote

[...] and I use the Forward Proxy on my public servers to access my local servers remotely [...]

I tried to toggle the ForwardProxy property:

If I set it to true, the proxy does not send any data back to "Source".

 

I doubt, that the error is based on the TIcsProxy component as I know that you are using the components in production environments.

But ATM I'm lost. I don't know where to look to solve that issue, or how to debug it any further.

Edited June 10 by uso

[Angus Robertson 574]

To fix a bug I must be able to reproduce it.  Sure the logs sure some different numbers, but I've no idea what data or requests you are sending, no headers logged or anything useful for debugging. 

 

Forward proxy is intended to direct to another web site, such as this one that generates a lot of data, very useful for testing in a live environment, the internals of the proxy moving data are the same for forward and reverse HTTP proxy. 

 

Angus

 

 

[uso 0]

Hello Angus,

 

I was able to repoduce the problem in another environment.

Backend service test app is attached (ServiceTest.zip), a vanilla OverbyteIcsProxySslServer was used as proxy, a browser (Edge, Firefox, Chrome, etc.) as client.

Proxy configuration was the same as above.

 

What happens:

Connecting http://localhost:80 - success
Connecting http://localhost:8080 - success
Connecting http://localhost:80/wsdl/IServiceTest - success
Connecting http://localhost:8080/wsdl/IServiceTest - success
Connecting https://localhost:443 - success
Connecting https://localhost:443/wsdl/IServiceTest - ERROR

 

ServiceTest.zip

Edited June 11 by uso

[Angus Robertson 574]

Thanks, I'll look at reproducing it on my servers, main problem is all my web sites are https only and force http to https, so needs something set-up for testing.

 

Angus

 

[uso 0]

2 hours ago, Angus Robertson said:

Thanks, I'll look at reproducing it on my servers, main problem is all my web sites are https only and force http to https, so needs something set-up for testing.

 

Angus

 

I'm unable to reproduce the problem in plain HTTP mode.

 

ServiceTest is standalone. You do not need a certificate other than the "ICS Intermediate Short" automagically delivered by OverbyteIcsProxySslServer.

To sum up, on my test machine I have ServiceTest, OverbyteIcsProxySslServer, its config and a browser - all run locally.

In my case thats enought to reproduce the problem (but remote access (with valid certificates) showed the same behaviour).

[Angus Robertson 574]

I'll be testing using my own public proxy server, remembered I do have one http server, svn.overbyte.be.  But I have some stuff to catch up with first.

 

Angus

 

[uso 0]

GSSL_BUFFER_SIZE seems to have directed influence on whether the problem is reproducible or not.

The default value of 16kb is less than the size of most of the data "my" Target emits.

For testing purposes I set the value of GSSL_BUFFER_SIZE to be larger than the largest data blob Target could possibly send and the problem went away.

Edited June 12 by uso

[Angus Robertson 574]

I've tested the ICS proxy as a reverse HTTP proxy for HTTPS<>HTTP, I've emailed you the URL, everything seems to work fine with a browser, with web pages of varying sizes.

 

The servicetest zip you posted seems to use web broker, never used that, and nothing to do with ICS.  I'm afraid I don't have the spare time to build, install and test such applications that are unconnected to ICS. 

 

Angus

 

 

[uso 0]

Hello Angus,

 

service app is not unrelated to the problem or ICS - at least not in this case.

The app enabled me to reproduce the behaviour I observed in a simple, uncluttered manner:

On 6/11/2024 at 9:04 AM, uso said:

[...]

Backend service test app is attached (ServiceTest.zip), a vanilla OverbyteIcsProxySslServer was used as proxy, a browser (Edge, Firefox, Chrome, etc.) as client.

Proxy configuration was the same as above.

[...]

As far as I know OverbyteIcsProxySslServer is part of the ICS sources and the "unrelated" ServiceTest is a simplified form of the service backend we use behind the proxy.

But the service is not aborting the transmission, or crashing:

On 6/10/2024 at 5:15 PM, uso said:

[...]

In Wireshark I can see that "server.local" sends the response as expected to the proxy,

but I'm unable to get the proxy to return the data as expected to the client.

The proxy does not always return the whole response from "server.local".

Requesting small entities (10000 byte >= response size) seems to work (success.log).

Whereas bigger entities (~16000 byte <= response size) fail (error.log).
[...]

 

 

So this is, as of today, the only way I can reproduce the problems I observe in my environment.

How else should I enable others to reproduce the error, if not with a demo service and the OverbyteIcsProxySslServer example delivered with ICS itself?!

 

Regards,

uso

Edited June 17 by uso

[Angus Robertson 574]

To sum up, you are unable to make the ICS proxy work with a specific web server or web broker, really no idea what that is.

 

How do you know this is an ICS problem?  Have you tried using the ICS proxy with other more modern web servers like Apache, as I have it, 

 

Your service example is undocumented, no setup instructions, I've no idea what is supposed to do, and I don't have time to debug it for you.

 

Angus

 

[uso 0]

1 hour ago, Angus Robertson said:

To sum up, you are unable to make the ICS proxy work with a specific web server or web broker, really no idea what that is.

 

How do you know this is an ICS problem?  Have you tried using the ICS proxy with other more modern web servers like Apache, as I have it, 

 

Your service example is undocumented, no setup instructions, I've no idea what is supposed to do, and I don't have time to debug it for you.

 

Angus

 

Hello Angus,
 

the sample is undocumented because it is a simple app generated by SOAP wizzard of Delphi.

Not much changes, only added autogenerated dummy functions to have a big WSDL file produced.

 

Just start ServiceTest, hit [Start] to start the service endpoint, start OverbyteIcsSslProxyServer (using the config from above),

hit [Open Browser] button on the ServiceTest and try to connect to localhost using a browser.

8080 connects directly to the backend service: No problem

80 connects to the backend service using the proxy (if configured like above): No problem

443 (using https protocoll) connects to the backend service using the proxy: Data partitially missing if SizeOf(Data) > ~16000 byte.

 

So there must be something I'm missing. I'm calling for help here and tried to describe my problem and my observations.

If it sounds like an insult against you or TIcsProxy, I beg your pardon!

On 6/10/2024 at 7:53 PM, uso said:

[...]

I doubt, that the error is based on the TIcsProxy component as I know that you are using the components in production environments.

But ATM I'm lost. I don't know where to look to solve that issue, or how to debug it any further.

 

Maybe the ServiceTest is ending the session in an unusal way after transmitting the response data to the proxy

and in response to that unexpected behaviour the proxy aborts the connection to the client bevore the TLS Tx buffer chunks are fully processed.

I can't tell it, because I do not know the internals of ICS TLS processing, nor do I know how to test for this case.

[Angus Robertson 574]

Please take this problem to email, it seems specific to your application rather than a general ICS problem.  I've replied.

 

Angus

 

[Angus Robertson 574]

These problems all came down to a missing line in the configuration file, so the ICS proxy behaved as a TCP proxy instead of an HTTP proxy, and did not modify HTTP headings.

 

I'll look at writing a configuration GUI for the IcsHosts files, to make it all easier and self documenting, it's not obvious which settings relate to the several protocols IcsHosts servers support. 

 

Angus

 

[uso 0]

1 hour ago, Angus Robertson said:

These problems all came down to a missing line in the configuration file, so the ICS proxy behaved as a TCP proxy instead of an HTTP proxy, and did not modify HTTP headings.

 

I'll look at writing a configuration GUI for the IcsHosts files, to make it all easier and self documenting, it's not obvious which settings relate to the several protocols IcsHosts servers support. 

 

Angus

 

Thanks a million for your help!