Angus Robertson

The ICSLogger is designed for our internal debugging of the components, and needs extra information added to be useful for application level debugging, such as when you start a request, and the events called. It does not log the NTLM requests,. which is where your statuscode is coming from, not the real request.  Beware we don't often test NTLM since it so rarely used on the public internet.  

Angus

Not looked at the code, but generally status=0 means an internal error or something unrelated to the HTTP protocol like SSL or disk I/O, hopefully your application logs all the protocol commands and responses which will help indicate at what point the error happened.  If you use TSslHttpRest instead of THttpCli, logging is built-in.   

I would not rely on the component internal state completely, you should not start a new request until after OnRequestDone has been called, ie post a message from that event that triggers the next request in the queue,  

Angus

If you set the properties separately, don't use ProxyURL, that override them all.  It was add added to avoid needing to save four or five different properties, and pass them up from derived components.

Angus

The proxy sample handles non-SSL as well.  the four properties Proxy, ProxyPort, ProxyUsername and ProxyPassword are all set by ProxyURL, so don't use them together.  Never used ProxyConnection.  

Angus

ICS includes an HTTP proxy component and sample project which you can build and run locally for testing, OverbyteIcsProxySslServer.dpr. 

Assuming you are using V8.62 or later, there is a new property ProxyURL property which combines four proxy properties as a URL for simplicity, ie http://[user[:password]@]host:port.  You still need to set ProxyAuth if that is needed.  A proxy listens on a specific IP address and port, and then forwards traffic to the original URL.  For testing, you can set 127.0.0.1 and port 81, and set-up the proxy to listen on the same, so ProxyURL would be http://127.0.0.1:81. 

Angus

Did you get anywhere using the Google RSA-PSS private key?  I've made IcsJoseFindAlg recognise them OK, but then hit a problem in IcsJoseJWKPubKey because the OpenSSL RSA functions don't seem to work on RSA-PSS keys so I can not read the exponents needed, I think this was why I have up testing RSA-PSS 18 months ago hoping OpenSSL would fix this, but not yet.  There are possible workarounds.

Angus

Update: OpenSSL changed the RSA functions to recognise RSA-PSS keys in October, but not had a new release of 1.1.1 since, so we need to be patient and it will work soon.

No, the ICS HTTP server is independent of the Windows http.sys API, While a server running at kernel level is potentially more efficient than one at application level, all the REST and authentication stuff would still be at application level.  You are also restricted to server facilities Microsoft chooses to offer, which are present means no TLS/1,3 or modern ciphers, for instance.  So really only an advantage for very heavy load servers.

Angus

Not much error handling for opening the file, it might not exist or be protected, or whether you read it correctly, I set all the output parameters for PKCS12_parse to nil before calling it, unless this is a very old Delphi your password is not AnsiString, just a few things to try, OpenSSL error handling might give you some ideas.  Your last line does not work with any newer versions of OpenSSL, and 1.0.2 is out of support in four weeks.

ICS has a TX509 certificate class that does all this for you, including getting all certificate fields, and another that renews it automatically before expiry.  You can use these with internet libraries. 

Angus

ICS is a project developed by volunteers and offered free of charge to the community.  Volunteers come and go, and currently there are none helping with C++ and MacOS, so our level of support depends on reports from end users, and we try to react.  We spend our time developing for platforms used by the majority of Delphi users, if those on other platforms don't help, they should not expect support.

V8.60 earlier this year added a lot of new components and it seems no MacOS user has tried to build this, thus the errors were not found.  So download V8.59 or earlier which should be okay for MacOS. The last bug specifically fixed against MacOS was in V8.52.   

I will fix or workaround GetComputerNameW and GetThreadLocale,  etc etc etc does not really help, there are not many new APIs.  I really don't have the time spend hours on this stuff, I just need to be told what to fix.

Angus

Did you try removing the offending Windows only SystemTime functions?  Did ICS then build okay on MacOs64?  I've corrected that problem in master, but need to know if other MacOS errors then occur.

Angus

It was concept code, how to get the port of an open socket. and can be simplified somewhat to:

BindIpPortStr := Socket.GetXPort;

since it's a built function that the FTP client seems not to use.  In this case Socket is your TSocketServer component, provided you are not using IcsHosts. 

The IcsHosts implementation up to V8.63 does not allow a zero port, since that means there should be an SSL port specified instead, each IcsHost is designed to listen on two ports at once.  But this was a bad design, so I'll change it for the next release so that both ports being zero uses a non-SSL random port.  I'll also return the random port allocated, somewhere. 

Angus

The port number property does not change if you specify zero, it remains zero.  You need to find the port number allocated by windows:

    saddr        : TSockAddrIn6;
    saddrlen     : Integer;
 

           { Get the port number as assigned by Windows }
            saddrLen  := SizeOf(saddr);
            ListenSocket.GetSockName(PSockAddrIn(@saddr)^, saddrLen);
            DataPort  := WSocket_ntohs(saddr.sin6_port);
 

This should work with the listen socket in SocketServer, but other things that need the port won't know about it. And not tested for zero.

Angus

TWSocket allows the port to set to zero and listen called with a random port allocated by Windows, this is used by the ICS FTP client for the PORT command.  

Not sure about TWSocketServer, never tried it but it's rather more complex listening on multiple ports so ignoring zero is quite possible, TWSocketServer is certainly not designed to listen on a random port.   Again the FTP client has code to allocated ports sequentially from a specific range, trying the next if in use, so you could borrow that.

Angus

A RSA-PSS key should be PS256 not RS256, but this is really down the other end, whether it accepts both type of RSA keys for RS256.  Easy to relax our code, if the server is also relaxed.  I'll look at this stuff when I do SASL for gmail. 

Angus

And what happens when you build for MacOS?  I think you will get the same errors.  It seems GetSystemTime is Windows only so need some conditional code there, guess no-one has tried to build for MacOS for a few months since that was added.  You can try making those two UTC function dummies removing the real code, I think they are only used for the time client and server.  Sorry, I won't be looking at this immediately.

Angus

I did finally get a Twitter developer account a while back, but always seem to be fire fighting something else when I try to get back to OAUth1.

Angus

I was testing Google Accounts last week, it's OAuth2 implementation is unusual but now supported by ICS, did not get as far as trying a service account.  But if Google is returning RSA-PSS keys, we need to support them, so on my list.

Angus

Quite happy to make ICS compatible with MacOS64, but you need to tell me what errors appear in which units, I don't have any Apple hardware to test.  Or is this a DEFINE issue, with MACOS or POSIX not being declared on MacOS64?

Angus

This list is possible additions to ICS, new protocols and functionality, none of which is guaranteed...  Open to suggestions for other possible protocol additions or improvements.  Personally, I'm unlikely to look at any of this stuff for several months, unless my company has an urgent need for something new.  But if several other users are all looking for the same thing, I can help co-ordinating improvements.

Protocol: STUN client and server
Why: Session Traversal Utilities for NAT allows finding a public IP address while behind a NAT router, by contacting a STUN server.  Used by public servers and clients that need to tell other applications how to contact them.  Also some client protocols like a host name, like SMTP.
Difficulty: low, simple protocol, easy to implement.
Benefits: medium, saves configuring the IP manually.

Protocol: RDAP Client
Why: Registration Data Access Protocol is the replacement for the Whois protocol, using HTTPS REST and Json protocols. Both domains and IP addresses.
Difficulty: low, simple protocol, easy to implement.
Benefits: low, Whois is heavily censored now.

Protocol: Roughtime client and server
Why: replacement for NTP and SNTP network time protocols (from Google), with security.
Difficulty: low, simple protocol.
Benefits: low, usually get time from Windows.

Protocol: HTTP/2 for HTTP client and server
Why: More efficient version of HTTP/1.1, particularly for web pages with dozens of elements, compresses headers.
Difficult: moderate, extra DLL, messy, lots to change.
Benefits: low, ICS is rarely used to download complex web pages, perhaps
more important for the HTTP server.

Protocol: SASL for SMPT and POP3 clients
Why: Simple Authorisation and Security Layer adds OAuth2 for SMTP and POP3, safer than clear authentication.
Difficulty: low, OAuth2 already done.
Benefits: high, where the email provider requires it.

Protocol: OAuth1 for Twitter
Why: Twitter uses OAuth1 rather than the easier and more recent OAuth2 almost everyone else uses.
Difficulty: low, uses HMAC which is done already.
Benefits: high, if you want to send tweets.

Protocol: DNS over HTTPS (DOH)
Why: secure DNS can not be intercepted and modified.
Difficulty: ICS already has a TDnsQueryHttps component and sample, but causing it to be used by TWSocket and other components at low level could get messy and link in all the REST and Json units.
Benefits: low, Microsoft is threatening to support DOH, probably only
Windows 10/2019.

Protocol: MQTT
Why: MQ Telemetry Transport is used to send messages between devices, including IoT.
Difficulty: ICS MQTT project n GutHub, needs integration.
Benefits: high, if you need the protocol.

Protocol: Websockets server
Why: A full duplex version of HTTP often used for server push to dynamically update web pages.
Difficulty: ICS websockets project in GutHub, needs update and integration.
Benefits: high, if you need the protocol.

Protocol: SChannel SSL/TLS Support
Why: avoid distributing OpenSSL DLLs by using SSL/TLS protocol APIs built into Windows. Downside is Microsoft often takes years to support new protocols and often only in the latest operating systems

Difficulty: high, needs to be done at the lowest levels, risks adding bugs for OpenSSL if both supported, need to replace a lot of OpenSSL encryption APIs with Windows APIs, and certificate APIs.
Benefits: low, unless you really hate DLLs.

Protocol: POP3 Server, IMAP Client and Server
Why: because these are missing and we all use email.
Difficulty: moderate, lot of new new code.
Benefits: high, if you need them.

Platform: Better C++ and MacOS Support
Why: we don't do much testing on C++ and MacOS due to lack of volunteers to do this regularly.  We lack samples for C++ and MacOS.
Difficulty: high, users want someone else to do the work.
Benefits: high, for C++ and MacOS users.

Platform: Support for mobile apps and Linux
Why: more platforms.
Difficulty: very high, probably at least one man year effort, maybe more.
Benefits: high, for mobile apps and Linux.
 

Angus

Which previous version did you successfully build under MacOs64?  The announcement said MacOS and Win64, not MacOs64. 

Angus

V8.63 was tested yesterday on RAD Studio 10.3 Rio Release 3 (aka 10.3.3) without any problems.

Angus

Sorry, RSA-PSS key support is only partially implemented, it is shown as pending at the top of the unit. 

I spent a lot of time learning and implementing the Jose stuff, and started to code new keys types, but never got to test them properly since they are rarely used even 18 months later.  In the end I have to move on and implement the Let's Encrypt stuff this was written for. 

Looking at the code, RSA-PSS keys are actually checked later in that function for jsigRsaPss256, or are you saying they can also be used with jsigRsa256?

Angus

ICS V8.63 has been released at:

http://wiki.overbyte.eu/wiki/index.php/ICS_Download

ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10 Seattle, 10.1 Berlin, 10.2 Tokyo and 10.3 Rio, and C++Builder 2006 to XE3, 10.2 Tokyo and 10.3 Rio. ICS supports VCL and FMX, Win32, Win64 and MacOS targets.

The distribution zip includes the latest OpenSSL 1.1.1d win32, with other versions of OpenSSL being available from the download page.

1 - IcsHosts is now supported in all the main ICS server components, TWSocketServer, TSslHttpServer, TSslHttpAppSrv, TIcsProxy, TIcsHttpProxy, TIcsIpStrmLog and TSslFtpServer.  This simplifies server applications which do not need to setup an SSL context, can easily support multiple listeners and hosts, and can automatically order and install SSL/TLS certificates from Let's Encrypt and commercial suppliers.

2 - There are several changes relating to automatic certificate ordering, mostly cosmetic based on experience adding the feature into more applications with a few issues fixed during testing with better logging.

3 - Made improvements to handle the OAuth2 version used by Google Accounts, allowing the REST component to access Google APIs such as Gmail.

4 - Did a major refresh of the three SSL/TLS trusted root certificate bundles included with ICS, to add new roots from Amazon and others, and remove untrusted certificates.

5 - There are improvements in TFtpCli to access FTP servers behind NAT routers where the internal and external IP addresses are different and not correctly handled by the FTP server.

6 - The TSslFtpServer component now uses the IcsHosts concept added to the web and proxy servers two years ago, see above. There is a new IcsHosts property which allows multiple hosts to be specified. Also fixed
ftpsNoPasvIpAddrInLan and ftpsNoPasvIpAddrSameSubnet options to work correctly to present local passive IP address on LANs.

7 - The TIcsIpStrmLog component will now start despite non-fatal SSL/TLS certificate warnings, and may be used with self signed certificates. The OverbyteIcsIpStmLogTst sample shows how to restart the TCP server after the first SSL/TLS certificate is automatically installed.

More detailed release notes are at: http://wiki.overbyte.eu/wiki/index.php/ICS_V8.63
 

Angus

ICS contains dozens of units, only a few of which are updated for new releases.  So many units have older versions.  But units they depend on may be later and fix bugs in that unit.  The overnight zip from SVN is the latest, V8.63 will be finally released later this week.

Angus

If you want help with connection problems you need to give us real URLs, we can not waste time looking for code problems when the input or site is faulty.  Your previous post was a perfect example.

You should also test the site using the REST sample which has proper logging to show what is happening.  It will probably work, and you can then check why your own code fails. 

Please don't post ICS source code here, we have it already.

Angus