Angus Robertson

Yes, TWSocketServer has been tested with hundreds of clients, the main limit is how many new clients can connect each second, particularly with SSL which has a high connection overhead. Once connected, lots of large files can be download in parallel. But blocking the server with SQL requests, SHA1 sums, etc, will stop new connections, file I/O, etc, The ICS FTP server uses a thread for such operations. The ICS web and FTP servers are good examples, although quite complicated. Angus

I've not used FtpUpOneFile myself for many years, it was written for another client and worked in their circumstance, need to test it. Angus

There are new versions of the FTP client and server in SVN and the overnight zip, fixing the FTP server options being ignored and improving passive mode on the client. I've added logging for both client and server IPv4 passive mode to log various IP addresses before the data connection is opened, to make it easier to debug why a connection fails. The FTP client also has a new Option ftpFixPasvLanIP for when '227 Entering Passive Mode ()' returns a LAN IP instead of a WAN IP, so use control IP instead. This fixes failed downloads if the FTP server is behind a NAT router and is not configured to present the external IP. An example log is: > PASV < 227 Entering Passive Mode (192,168,1,161,82,10). ! Passive connection requested to: 192.168.1.161:21002, control channel: 217.146.115.83 ! Suspicious LAN IP changed to control channel address > LIST < 150 Opening data connection for directory list. www3.magsys.co.uk SSL Connected OK with TLSv1.2 < 226 File sent ok Note this example won't work now, I had to misconfigure the server to return the local IP address. Angus

Your three Xfer fixes are now in SVN and the overnight zip. Angus

The FTP server Options ftpsNoPasIpAddrInLAN and ftpsNoPasvIpAddrSameSubnetoptions were broken because the remote client IP was always returned as 0.0.0.0, now fixed, will be in SVN next week with other FTP changes. Angus

Sorry, I don't do C++ Builder, it needs peer support from other users. Angus

No, it gets an A rating and supports TLS/1,2 so should work with ICS. The OverbyteIcsHttpRestTst sample connects fine from here: www.kcsoftwares.com SSL Connected OK with TLSv1.2, cipher ECDHE-RSA-AES128-GCM-SHA256, key auth RSA, key exchange ECDH, encryption AESGCM(128), message auth AEAD Angus

SSL connections fail all the time for dozens of different reasons. The client might require protocols or ciphers the server does not support, or vice versa. Much depends on how the SslContext is set-up, it's easier on modern components like TSslHttpRest that do most of the hard work for you. If it's an obscure server, bad SSL configuration is always a possibility and https://www.ssllabs.com/ssltest/ is always useful to find how good or bad the server really is. Angus

Misconfiguration of NAT'd FTP servers is common, I only corrected one of mine this week. So if the client can help, it's worth a little effort. The STUN protocol allow determination of public IP address behind NAT, and a STUN component has been on my ICS wish list for a few years, maybe I'll finally write it. Won't help me though, since my development servers behind NAT have several different IPv4 and IPv6 addresses on two different ISPs. Angus

Yes this issue was discussed here last week relating to the ICS FTP server behind a NAT router. In the past, routers were able to manipulate the control channel to change the private IP address to public, but it seems FTP is now considered a legacy protocol and newer routers don't bother. Also SSL stops the router seeing the control channel, one reason you can configure SSL on data but leave the control channel clear. I like FileZilla's idea of re-using the earlier public address, I'll have a look and see if that can be done in ICS, as an option. Angus

On my list. Angus

TriggerDataAvailable was never designed to be extendable. Apart from the three version in wsocket no other ICS units override it. So while you can override it, you need to replicate all the functionality like parsing line endings. Angus

Yes, I was being lazy and hoping no-one would use that feature on Delphi 7, I'll add your fix next week. Angus

The ICS library is designed to be used through event handlers, not by overriding internal functions. That requires a high level understanding of the library structure and is beyond the support I can offer here. Angus

I am not trying to write your code for you, just explaining why your function does not work. You have a conceptual issue trying to override library functionality. Angus

You are not using the Inherited code in other versions of TriggerDataAvailable so need to handle line mode yourself. Angus

When using LineMode, in the onDataAvailable event you use ReceiveStrA to get a complete raw line without needing to check line endings, but that just calls Receive and does all the hard work for you. If you are creating descendent components, you need to ensure the correct ancestor, which is not easy due to the deep hierarchy of wsocket. LineMode is handled in TCustomLineWSocket. Angus

You just ignore errors you don't want to handle. Not a good idea though. Angus

This depends on whether you are connecting synchronously or asynchronously. For sync, a simple loop is all that is needed, but your loop will be blocked for up to a minute during each connection attempt depending on TCP/IP timeouts, look at function FtpLogon in OverbyteIcsFtpMulti which does this for different IP addresses. For async, you use the onSessionConnected event which is called whether a connection is OK or fails. Look at SocketSessionConnected in OverbyteIcsIpStreamLog which handles retries after a delay. Angus

Sorry, don't understand the question. There is no TClientSocket in any ICS source unit, but client would suggest you referring to TWSocketServer. OnError is a notification event, you can not suppress errors there. Angus

Thanks, will fix that next week. Don't use the threaded version very much, just to test how many hundred simultaneous FTP sessions the ICS FTP server and client can support. Angus

For completeness, I have now configured my port forwarded FTP server correctly, opened passive ports in the Draytek router, and it is now working correctly again for internal and external traffic. However the ftpsNoPasIpAddrInLAN and ftpsNoPasvIpAddrSameSubnet are not working as expected, it never uses a local IP address, which I need to investigate. I suspect IPv6 may also be an issue on passive data connections, need to test that as well. But you can get around the LAN issue by changing the Passive IP in the event. Angus

ICS includes three CA CA Trusted Store, two as PEM bundle files, one in a source unit, and access to the Window Certificate Store directly: 1 - RootCaCertsBundle.pem is a large file that was originally created 15 years ago by exporting the Windows certificate store using the OverbyteIcsPemTool sample. But Windows 10 no longer has a complete local certificate store and instead downloads new certificates as needed by Windows browsers. So with ICS V8.63, it is now the same as the new microsoft_windows.pem bundle mentioned above. It currently contains 289 certificates and is 595 bytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 2 - TrustedCABundle.pem is a smaller file, with certificate for major commercial issuers manually updated as newer sites are found to have missing root certificates. But this file is more dynamic than RootCaCertsBundle.pem. It currently contains 51 certificates and is 88 Kbytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 3 - To avoid distributing bundle files and as a fail safe if a file can not be found, ICS includes 34 built-in hard coded certificates in OverbyteIcsSslX509Utils.pas which can be returned as a string by the function sslRootCACertsBundle. Again this unit may be dynamic with new certificates added as needed. Note only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in bundle by default, other components need to add it manually to avoid the extra program code involved. 4 - ICS also includes a component TMsCertChainEngine in the unit OverbyteIcsMsSslUtils.pas which allows applications to avoid using bundle files and instead access the Windows Certificate Store directly to validate certificates. There is a very slight overhead as the store is opened and Windows may need to download missing root certificates. Only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components includes TMsCertChainEngine by default, with the CertVerMethod property selecting CertVerNone, CertVerBundle or CertVerWinStore. TMsCertChainEngine does include one extra optional feature to check if any certificates in the chain have been revoked by their issuer, perhaps for fraudulent use, beware revoke checks involve contacting each issuer and can slow down chain validation by a few seconds or even longer. Either of the certificate bundle files may be loaded into an SslContext by using the SslCAFile property. The built in bundle may be specified before the SslContext is initialised using SslCALines.Text property, or the LoadCAFromString method after initialisation. If checking a certificate chain, OpenSL will issue the error message 'unable to get local issuer certificate' if a trusted certificate is not found in the store. The contents the three certificates bundles are listed at http://wiki.overbyte.eu/wiki/index.php/FAQ_ICS_SSL/TLS_CA_Trusted_Store_Contents Angus

CA Trusted Store Background The whole SSL/TLS certificate verification process depends upon finding a trusted root certificate that signed the next certificate up the chain, and so on. But who chooses those trusted roots? The answer is the author of the application that accepts an SSL/TLS certificate, or maybe the operating system the application is running on, or the SSL library used to build the application. The bad news is that root certificates come and go, old roots stop being accepted for various reasons, and new roots are introduced for new security standards or new businesses. Historically most roots were RSA Sha1 digest signed and most still are, despite Sha1 being banned for new certificates. Increasingly Sha256 and ECC root certificates are being introduced and required by new intermediate certificates. Typically, active certificate authorities have at least four roots, with 2,048 and 4,096 bit RSA keys and 256 and 394 bit ECDSA keys (which are much smaller than RSA). Sources of CA Trusted Stores SSL/TLS trusted root certificate bundles and always changing, annually perhaps for major changes, although Microsoft officially Windows roots every two months. The Common CA Database (CCADB) https://www.ccadb.org/ is a repository of information about Certificate Authorities (CAs), and is used by a number of different root store operators to manage their root stores. But it's not easy to create root bundles from CCADB and another developer got frustrated with updating roots, and created a Trust Stores Observatory Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 500 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM bundle files, one each for the different operating systems. New PEM Bundle CA Trusted Store Files There are six different PEM CA bundle files, built from the Trust Stores Observatory Git repository: apple.pem - 174 Certificates google_aosp.pem - 137 Certificates microsoft_windows.pem - 289 Certificates mozilla_nss.pem - 137 Certificates openjdk.pem - 88 Certificates oracle_java.pem - 88 Certificates Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store. These bundles may be downloaded at: https://www.magsys.co.uk/download/software/ca-root-bundles.zip Magenta Systems Ltd will periodically update these bundles, as needed. ICS V8.63 contains the content of microsoft_windows.pem. Angus

The client and server logging should show if the control channel is closed prematurely. If you not already using my old TMagFtp client component, I'd suggest you update your client to use the latest version which is now in ICS, TIcsFtpMulti which uses special directory commands when working with the ICS FTP server, and automatically retries transfers when they fail due to control channel closing or other errors. Angus