Cross platform HTTP client with proxy support?

[softtouch 9]

For macos, I tried Indy, which need openssl, and does not seem to work with newer openssl libs or tls 1.3, or even with the systems ssl implementation.

I also tried using THttpClient, which works but crash on mac when trying to set a proxy. There is also stated that it will always use the system proxy.

I need to use public and private proxies, multiple requests, each request has to use another proxy.

Does anybody know any 3rd. party component that can use ssl without the need of openssl, and can set individual proxies for each request?

[Patrick PREMARTIN 76]

Perhaps ICS ? http://www.overbyte.eu/frame_index.html?redirTo=/products/ics.html

[softtouch 9]

Unfortunately, I cannot get that to work with macOS/Cross platform and it seems to use also openssl.

[Fr0sT.Brutal 900]

ICS works on MacOS but not on Linux and uses OpenSSL.

 

Why not implement proxy in RTL client by yourself?

[esegece 47]

Indy supports openSSL 1.1 using the following pull request https://github.com/IndySockets/Indy/pull/299, not sure if works on OSX

 

I publish a commercial library based on Indy that supports openSSL 1.1 and openSSL 3.0 for all Delphi personalities (windows, OSX, linux, iOS...), you can use a TsgcHTTP1Client which inherits  from TIdHTTP Indy client and connect using openSSL 1.1 or 3.0

Find below an example using openSSL 1.1 that is cross-platform

 

uses
  sgcHTTP, sgcWebSocket_Types;



function GetHTTP(const aURL: string): string;
var
  oHTTP: TsgcHTTP1Client;
  vResponse: string;
begin
  oHTTP := TsgcHTTP1Client.Create(nil);
  Try
    oHTTP.TLSOptions.OpenSSL_Options.APIVersion := oslAPI_1_1;
    oHTTP.TLSOptions.Version := tls1_3;
    oHTTP.Proxy.Enabled := True;
    oHTTP.Proxy.Host := '3.215.142.228';
    oHTTP.Proxy.Port := 3128;
    oHTTP.Proxy.ProxyType := pxyHTTP;
    result := oHTTP.Get(aURL);
  Finally
    oHTTP.Free;
  End;
end;

 

This still requires the openSSL libraries but I provide already compiled openSSL libraries for registered customers for Windows, OSX, iOS and Android. More info:

 

https://www.esegece.com/

https://www.esegece.com/help/sgcWebSockets/#t=Components%2FHTTP%2FHTTP1%2FHTTP1.htm

https://www.esegece.com/help/sgcWebSockets/#t=QuickStart%2FOpenSSL%2FOpenSSL_OSX.htm

 

The package that supports this is sgcWebSockets Enterprise that comes with a custom indy version that support openSSL 1.1 and 3.0

 

Kind Regards,

Sergio

[Rollo62 538]

43 minutes ago, esegece said:

The package that supports this is sgcWebSockets Enterprise that comes with a custom indy version that support openSSL 1.1 and 3.0

Thanks for the info, I see two products of yours supporting OpenSSL:

 - The sgcWebSockets ( which needs to be the Enterprise version, wich comes with SSL and many other features )

 - or the sgcIndy ( which seems to be SSL alone addition to normal Indy, but I assume this will solve the SSL issues )

 

Can you please tell us the strategy behind:

Is the "WebSockets Enterprise" = "WebSockets package" + "sgcIndy package" ?

Or are the SSL implementations in sgcWebSockets Enterprise and scgIndy somewhat different ?

Both should come with all cross-platform binaries of latest SSL.

 

Can I start with sgcWebSocket Professional and then add sgcIndy later, will I got the same as in sgcWebSockets Enterprise ?

[esegece 47]

1 hour ago, Rollo62 said:

Thanks for the info, I see two products of yours supporting OpenSSL:

 - The sgcWebSockets ( which needs to be the Enterprise version, wich comes with SSL and many other features )

 - or the sgcIndy ( which seems to be SSL alone addition to normal Indy, but I assume this will solve the SSL issues )

 

Can you please tell us the strategy behind:

Is the "WebSockets Enterprise" = "WebSockets package" + "sgcIndy package" ?

Or are the SSL implementations in sgcWebSockets Enterprise and scgIndy somewhat different ?

Both should come with all cross-platform binaries of latest SSL.

 

Can I start with sgcWebSocket Professional and then add sgcIndy later, will I got the same as in sgcWebSockets Enterprise ?

 

Hello,

 

Yes, sgcWebSockets Enterprise edition comes with a custom indy package with support for openSSL 1.1 & 3.0, ALPN... The sgcIndy package, which is used for users that want to add support for the latest openSSL versions with the Indy components (ftp, smtp...) is currently included too with the sgcWebSockets Enterprise Source code package. The SSL implementations are equal for both, but in the sgcWebSockets package, the required indy files are renamed, so you can have the standard indy library installed in your IDE and the sgcWebSockets enterprise package without problems. Currently, the sgcIndy Package can be purchased separetely because some users are only interested in use the latest openSSL libraries for Indy.

 

No, sgcWebSockets Enterprise has more features than sgcWebSockets Professional + sgcIndy, the enterprise edition has more components: HTTP.SYS server, http/2 support, DTLS over UDP... Anyway, you can upgrade from Professional to Enterprise edition at any time. More about feature matrix:

 

https://www.esegece.com/websockets/features/feature-matrix-delphi-cbuilder

 

Only keep in mind that if you purchase the sgcWebSockets professional edition and later you add sgcIndy, in order to use the latest openSSL libraries that comes with the sgcIndy package, you need the Professional Edition with source code, because the Professional BASIC edition compiles against the standard indy library.

 

Kind Regards,

Sergio

[softtouch 9]

@esegece your post is unrelated to my request of a 3rd party component that works without openssl on macos.

[Daniel 417]

18 minutes ago, softtouch said:

@esegece your post is unrelated to my request of a 3rd party component that works without openssl on macos.

Well, yes, unrelated to your question, but a direct answser to Rollo62. Well, this is how it works sometimes.

We try to stay on topic of course, but this is also not StackOverflow. So there is always a gray line ... 

[Fr0sT.Brutal 900]

What's so wrong with OpenSSL? ICS could compile this lib statically with additional paid 3rd party lib.

 

Having read the code I have (10.3), I'm afraid you can't easily plug into RTL version as it doesn't operate sockets calling more high-level OS functions instead. You could run HTTP CONNECT request to a custom proxy but then you'll have problems doing TLS to a destination host. So I suspect all you have to do is first curse MacOS designers as they deserve and then lower your requirements to include 3rd party.

Edited January 30, 2023 by Fr0sT.Brutal

[Rollo62 538]

Yes. Sorry that all here consume too much time to find a workable solution or alternative for softtouch's issues, with or without Indy.

It was also claimed that newer OpenSSL was not working and he was looking for alternatives, so then better tell us clearly in big letters that Indy and which specific 3rd Parties are a no-go.

Edited January 30, 2023 by Rollo62

[Lars Fosdal 1792]

@softtouch- ref. THttpClient on Mac - Access Violation - did you register an issue on the Quality portal?

[Fr0sT.Brutal 900]

2 minutes ago, Lars Fosdal said:

@softtouch- ref. THttpClient on Mac - Access Violation - did you register an issue on the Quality portal?

Mac doesn't allow setting a custom proxy

[Lars Fosdal 1792]

I found this issue: https://quality.embarcadero.com/browse/RSP-40392 which appears to be the same, and it is in status Open.

The example code looks a bit weird, though.

[softtouch 9]

8 minutes ago, Lars Fosdal said:

I found this issue: https://quality.embarcadero.com/browse/RSP-40392 which appears to be the same, and it is in status Open.

The example code looks a bit weird, though.

The example code is just the smallest possible code causing the crash. The moment I set a proxy, it crash in the code seen in the screenshot there in System.Net.HttpClient.Mac, thats why I cant use THttpClient, even I would like to (because of the ssl thing).

[Lars Fosdal 1792]

15 minutes ago, Fr0sT.Brutal said:

Mac doesn't allow setting a custom proxy

But, it looks like you are correct. MacOS and iOS doesn't allow the app to override the settings in the OS.

 

Platform	Behavior
Windows	The HTTP Client uses the system proxy settings. You can bypass the system proxy settings and you can also provide alternative proxy settings for the HTTP Client. To bypass the system proxy settings, create proxy settings for the HTTP Client and specify http://direct as the URL.
macOS	The HTTP Client always uses the system proxy settings. Even if you provide alternative proxy settings for the HTTP Client, the HTTP Client uses the system proxy settings.
iOS	The HTTP Client always uses the system proxy settings. Even if you provide alternative proxy settings for the HTTP Client, the HTTP Client uses the system proxy settings.
Android	The HTTP Client uses the system proxy settings. You cannot bypass those settings, but you can provide alternative proxy settings for the HTTP Client.
Linux	The HTTP Client uses the system proxy settings. You cannot bypass those settings, but you can provide alternative proxy settings for the HTTP Client.

[Lars Fosdal 1792]

1 minute ago, softtouch said:

the smallest possible code causing the crash

h:=THTTPClient.Create;
h.ProxySettings.Create(<IP>,<PORT>,<USER>,<PASS>);
h.Get(<URL>);
h.Free;

The AV problem is that line two should have read:

h:=THTTPClient.Create;
h.ProxySettings := TProxySettings.Create(<IP>,<PORT>,<USER>,<PASS>);
h.Get(<URL>);
h.Free;

but, as the documentation says - it would not have made a difference, as the OS settings always take effect on MacOS (and iOS).

[softtouch 9]

3 minutes ago, Lars Fosdal said:

h:=THTTPClient.Create;
h.ProxySettings.Create(<IP>,<PORT>,<USER>,<PASS>);
h.Get(<URL>);
h.Free;

The AV problem is that line two should have read:

h:=THTTPClient.Create;
h.ProxySettings := TProxySettings.Create(<IP>,<PORT>,<USER>,<PASS>);
h.Get(<URL>);
h.Free;

but, as the documentation says - it would not have made a difference, as the OS settings always take effect on MacOS (and iOS).

I tried both variations, both crash at the same line. Even setting a proxy would not work on macOS, it still should not crash the program. I used this same code already on older macOS versions, and there was no crash. Possible macOS 13.x cause it, I have no clue and also cant spend much time to investigate this, I just need to find a working component which works under Windows and macOS target, not using openssl.

[Lars Fosdal 1792]

I have a MBP M1 Pro, but I don't have a proxy service to try it on, otherwise I would.

[softtouch 9]

5 minutes ago, Lars Fosdal said:

I have a MBP M1 Pro, but I don't have a proxy service to try it on, otherwise I would.

I could provide a private proxy if needed. I can also provide the small test program.

 

When I set the proxy with 

h.ProxySettings := TProxySettings.Create(<IP>,<PORT>,<USER>,<PASS>);

it will call the procedure TMacHTTPClient.SetMacProxySettings in System.Net.HTTPClient.Mac, which calls various functions like "LDict.setValue(TNSNumber.OCClass.numberWithInt(1), kCFNetworkProxiesHTTPEnable);", but all the calls to kCFNetworkProxiesHTTPEnable and other "kCFNetworkProxies..." return always an empty string, causing somehow the app to crash.

[Remy Lebeau 1403]

11 hours ago, softtouch said:

For macos, I tried Indy, which need openssl, and does not seem to work with newer openssl libs or tls 1.3

Indy's support for OpenSSL 1.1+/TLS 1.3 is a WIP that is available in this pull request: https://github.com/IndySockets/Indy/pull/299

Quote

or even with the systems ssl implementation.

Correct, Indy does not support platform-specific implementations at this time.  Delphi's native HTTP client libraries do, though.

Quote

I also tried using THttpClient, which works but crash on mac when trying to set a proxy.

Did you report that to Embarcadero?

Quote

Does anybody know any 3rd. party component that can use ssl without the need of openssl, and can set individual proxies for each request?

Note that Indy is not specifically tied to OpenSSL exclusively.  That is just Indy's default SSL/TLS library since it is cross-platform, but you can use any other library you want, all you need is a TIdSSLIOHandlerSocketBase-derived class that wraps the library's API.  Some 3rd party SSL/TLS libraries provide such a class, but for others you will have to write (or find) your own class.  There are only a handful of virtual methods that need to be implemented (namely, to handle the handshake, to read/write data, etc).

Edited January 30, 2023 by Remy Lebeau

[aehimself 396]

10 hours ago, Daniel said:

but this is also not StackOverflow

So we can use ChatGPT for reputation farming? 🙂