Angus Robertson

I will need to reproduce connection failures before I look into fixing them. So I need a Websocket server to test against. My only suggestion is to try the client against other servers and see if any fail, because no-one else has reported any problems. Angus

Are you suggesting there is a problem with the Base64Encode function creating incorrect output? Have you attempted to debug and fix the problem yourself, or are you expecting someone else to do it? I've been testing the OverbyteIcsWebSocketCli unit this week and it works fine. Angus

As the comments in the code say, the MsUserAuthority setting depends on the account type that the credentials relate to, mainly for corporate accounts, common and consumer are for general email accounts, the older version defaulted to consumer. Angus

The URLs used for Rest Email are at the top of the OverbyteIcsSslHttpOAuth.pas unit, and have not changed since the unit was created, except to allow <MsUserAuth> instead of consumer, are you setting MsUserAuthority correctly? Angus

No idea why this client is not using keep-alive, perhaps their REST library does not support it? The basic Websocket API is working, and the client can access another Websocket interface on the server so seem to know their stuff, now just need to design the pipelining, so that queued queries get returned to the correct client. I'll look at JSON-RPC 2.0 for next time. Angus

Sorry, no quick solution. There are OpenSSL providers to access private keys on TPM2 devices and to access certificates in the Windows Store (which ICS can now do), but not to access the Windows crypto signing APIs so that other dongles can be supported, at least to my knowledge. Angus

Digital certificates are sometimes distributed on USB tokens precisely to stop them being copied by software applications. ICS can only access private keys that are stored in local windows store. The USB token is not a simple storage device, it has an operating system that allows applications to pass data to the token, which is then signed by the private key and passed back to the application as a hash. This signing is normally done by a Windows or OpenSSL API, and they need to be aware of the token and use that instead. All token suppliers provide drivers that allow Windows applications to use their tokens, but not usually for OpenSSL. OpenSSL needs extra code to access USB tokens, in the obsolete versions it was called an engine, with 3.0 and later it's called a provider, but it's just another DLL. I understand there are OpenSSL providers to allow use signing with USB tokens, but they are supplied as C source code not Windows DLLs, and I've never tried any. Integrating such a provider into ICS would not be quick, or maybe some-one has done it already? Angus

Thanks for the comment, yes record ending is important, since this API has simple URL encoded arguments a single CRLF will be fine. In fact, allowing multiple requests in a single message is probably what is really needed. This client tends to do 10,000 odd requests in a single block during the night, takes about 25 minutes at the moment, single server thread, with a new session each time, eight per second. Will need to decide how many requests can be queued, in case they decide to try 1 million. The responses do include the main argument, ie {"success":true,"reccount":16,"records":[{"number":"118118","number_from":"2010-03-10", etc. I should add the API type to the outer wrapper. Then send them back one message at a time. Either the server or client can ping/pong to keep the connection open, I was planning on the server doing that to avoid complexity at the client, they simply close the connection when the batch is over. Angus

Indeed, no need for threads in ICS. But Windows uses a thread to connect a TCP socket, and typically waits 30 to 40 seconds for a response before the Close event is called with an error, you can not abort it earlier. So the socket can not be re-used for another connection immediately, unlike ICMP. If you are checking a lot of hosts, you need a socket pool where they are not re-used until closed by Windows. Angus

You have already discovered two reasons why TCP ping is unreliable, you need a remote open port and TCP timeouts are horrible to work around, that is why everyone uses ICMP ping which is 100 times more useful. If you want to persist with TCP (why???) you should be using the TIcsIpStrmLog component, with the OverbyteIcsIpStmLogTst sample, try to connect to your TCP address and port as TCP Client, there are various setting you can change. That component is used for similar purposes in other places in ICS, despite the name, it is really a high level version of WSocket that can be used as a client or server for TCP or UDP, sending data between instances of itself or other applications. Angus

As the others have said, TCP ping does not exist, you can try and open a specific TCP port at an IP address, but Windows has a long timeout while this is attempted and the socket can not be reused until that is over, so it's slow, and you need lots of parallel sockets to make it work, and hope some ports are open. Build the ICS Network Tools sample that got added to ICS V9, the LAN Devices tab uses a new component that scans ranges of IPv4 and IPv6 addresses for devices, using APR, Neighbourhood IPs and pings, and builds a table with host names, MAC addresses and vendor, etc. It's very similar to the excellent Nirsoft Wireless Network Watcher tool I've run continually for a decade to monitor my LAN. Angus

Your code looks the same as the original suggestion, which was added to the version in SVN last month. There are other fixes to this unit pending. Seems to be quite an interest in the new ICS websocket components. Angus

I've asked him to send me the changed units, I'll incorporate any safe Android changes I've not already made in the next couple of weeks, busy with low level SSL improvements. Angus

Thanks, those checks are not in the POST handler and I thought I'd removed them since my PUT tests all worked, but now gone. SVN may be a few days. Angus

My pre and post command jobs do other stuff as well as signing, copy files to various directories. Beware the signtool /a command only works when there is just one certificate available... Angus

I have batch command files that handle all the signing, and then sign the final setup file, using InnoSetup pre and post compile steps. Better to use CMD extension for Windows 11, BAT is deprecated. [PostCompile] Name: "signsetup.bat" [PreCompile] Name: "compress.bat" Angus

Not used the signcode /sha1 argument before, it is more common to use : /a /s MY /n "Common name" (this all replaces /f in my earlier example) where /a automatically looks for a signing certificate, /s is the Windows store (MY is the name for Personal), and the Common Name is usually your company name, but whatever is shown on the General certificate dialog tab for 'Issued to', or CN= under Subject on the Details tab. Angus

No, because the HSM in the token has a program that takes data from the application, signs it with the private key in the HSM and returns a digest to the application as part of the signing process. The whole point is the private key is not available outside the HSM so can not be copied. The HSM/token can be used remotely only if the data to be signed is passed to the PC/server with the HSM, and the digest returned to the originating PC. Angus

You did not qualify your comment about PFX files only being useful for self signed certificates by mentioning code signing. Not sure how useful self code signing would be. But you are correct, new code signing is all dongles, although my own Comodo PFX files still work fine with signtool: signtool sign /p "xx" /f "c:\certificates\magenta-systems-certkey.pfx" /d "Copyright (c) 1998-2023 The OpenSSL Project" /as /fd sha256 /tr http://timestamp.sectigo.com /td sha256 "c:\svn-repos\signed-openssl\openssl-3.2.0-win32\libcrypto-3.dll" That command only needs a minor change to make it useful a dongle certificate, assuming the drivers for the dongle are installed. Angus

I would dispute that statement, PFX or PKCS#12 are Microsoft's preferred format since they contain certificate, private key and intermediate certificates, so one file per host. Sure you can create PEM/CER bundles with a private key, but more than one certificate can confuse servers. But Apache does not support PFX, you have to use PEM, that is the main reason PFX is not often seen. Angus

Is this a Code Signing certificate? If so, you can not export the private key from the token, so no reason to export the certificate either. If not, why did Digicert supply it on a token? Angus

PemTool has both, two buttons, Create Request from Props and Create Request from Cert, the former uses properties from another tab, lots of them. PemTool takes a while to understand, it is a development tool to test all the ICS certificate functions, and does not have a friendly GUI. It only writes files, no database like XCA. There is a second sample OverbyteIcsX509CertsTst that orders Let's Encrypt certificates and includes an 'Own CA' allowing you to sign your own local certificates with a private CA. Angus

ICS has a lot of tools for certificate manipulation, reading and writing different formats, so can convert between them, and can read and write to the Windows Store. The OverbyteIcsPemTool sample does all this, it has more features than the XCA tool. A compiled version can be downloaded if you don't want to build it. Angus

OpenSSL has released new minor version 3.2.0, which has a lot of new features. It is compatible with the current versions of ICS, but has only been tested briefly with clients, it needs at least a week of testing with servers before I'm comfortable adding the DLLs to ICS as the defaults. The major change in 3.2.0 is support for client side QUIC protocol. QUIC is based on UDP rather than TCP and allows multiple streams in parallel, typically for downloading web pages with hundreds of elements, QUIC combined with HTTP/2 becomes HTTP/3. There is a DLL solution that has been used to add HTTP/2 to Indy but not native Delphi implementation I'm aware of, it's a lot of work. So no possibility of ICS having HTTP/3 soon. Other changes in 3.2.0 include: Certificate compression in TLS, including support for zlib, zstd and Brotli Deterministic ECDSA. Support for Ed25519ctx, Ed25519ph and Ed448ph. AES-GCM-SIV. Argon2 and supporting thread pool functionality. Hybrid Public Key Encryption (HPKE). The ability to use raw public keys in TLS. Support for Brainpool curves in TLS 1.3. SM4-XTS. Support for using the Windows system certificate store as a source of trusted root certificates. Some of the above cipher and hash changes may be used by TLS connections without change to ICS, if negotiated with the other end, but certificate related changes will need updates to ICS. Windows binaries are available in SVN and the overnight zip file and separately from https://wiki.overbyte.eu/wiki/index.php/ICS_Download or https://www.magsys.co.uk/delphi/magics.asp In addition to the three DLL files, the zip includes a compiled RES resource file that contains the same DLLs, text files and version information, see the RC file. The RES file may be linked into application EXE files and code then used to extract the DLLs from the resource to a temporary directory to avoid distributing them separately. ICS V9.1 and later optionally support loading the resource file, currently in SVN and the overnight zip. Angus

ICS V9.0 has been released at: https://wiki.overbyte.eu/wiki/index.php/ICS_Download ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10 Seattle, 10.1 Berlin, 10.2 Tokyo, 10.3 Rio, 10.4 Sydney and 11 and C++ Builder 2006 to XE3, 10.2 Tokyo, 10.3 Rio, 10.4 Sydney and 11. ICS supports VCL and FMX, Win32, Win64 and MacOS 32-bit targets. The distribution zip includes the latest OpenSSL 3.1.2 win32, with other versions of OpenSSL being available from the download page. ICS V9 is planned to be a long term support release with no new components or major features added, just bug fixes as needed, major changes will be for V10. V9.0 has been tested with Delphi 7, but I had to make changes to several new components due to missing language features and TWebBrowser does not exist, so V9.0 will be the last tested with D7. Delphi 2007 is easier to support for those building ANSI projects and will become the oldest version supported. An overview of the new samples and components was posted in this group a few days ago, and is included in the download page. The full release notes for V9.0 are at https://wiki.overbyte.eu/wiki/index.php/ICS_V9.0 All ICS active samples are available as prebuilt executables, to allow ease of testing without needing to install ICS and build them all. There are four separate zip files split into clients, servers, tools and miscellaneous samples which can be downloaded from https://wiki.overbyte.eu/wiki/index.php/ICS_Samples The Getting Started page at https://wiki.overbyte.eu/wiki/index.php/ICS_Getting_Started has also been updated with all the new components. Angus