Angus Robertson

This depends on whether you are connecting synchronously or asynchronously. For sync, a simple loop is all that is needed, but your loop will be blocked for up to a minute during each connection attempt depending on TCP/IP timeouts, look at function FtpLogon in OverbyteIcsFtpMulti which does this for different IP addresses. For async, you use the onSessionConnected event which is called whether a connection is OK or fails. Look at SocketSessionConnected in OverbyteIcsIpStreamLog which handles retries after a delay. Angus

Sorry, don't understand the question. There is no TClientSocket in any ICS source unit, but client would suggest you referring to TWSocketServer. OnError is a notification event, you can not suppress errors there. Angus

Thanks, will fix that next week. Don't use the threaded version very much, just to test how many hundred simultaneous FTP sessions the ICS FTP server and client can support. Angus

For completeness, I have now configured my port forwarded FTP server correctly, opened passive ports in the Draytek router, and it is now working correctly again for internal and external traffic. However the ftpsNoPasIpAddrInLAN and ftpsNoPasvIpAddrSameSubnet are not working as expected, it never uses a local IP address, which I need to investigate. I suspect IPv6 may also be an issue on passive data connections, need to test that as well. But you can get around the LAN issue by changing the Passive IP in the event. Angus

ICS includes three CA CA Trusted Store, two as PEM bundle files, one in a source unit, and access to the Window Certificate Store directly: 1 - RootCaCertsBundle.pem is a large file that was originally created 15 years ago by exporting the Windows certificate store using the OverbyteIcsPemTool sample. But Windows 10 no longer has a complete local certificate store and instead downloads new certificates as needed by Windows browsers. So with ICS V8.63, it is now the same as the new microsoft_windows.pem bundle mentioned above. It currently contains 289 certificates and is 595 bytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 2 - TrustedCABundle.pem is a smaller file, with certificate for major commercial issuers manually updated as newer sites are found to have missing root certificates. But this file is more dynamic than RootCaCertsBundle.pem. It currently contains 51 certificates and is 88 Kbytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 3 - To avoid distributing bundle files and as a fail safe if a file can not be found, ICS includes 34 built-in hard coded certificates in OverbyteIcsSslX509Utils.pas which can be returned as a string by the function sslRootCACertsBundle. Again this unit may be dynamic with new certificates added as needed. Note only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in bundle by default, other components need to add it manually to avoid the extra program code involved. 4 - ICS also includes a component TMsCertChainEngine in the unit OverbyteIcsMsSslUtils.pas which allows applications to avoid using bundle files and instead access the Windows Certificate Store directly to validate certificates. There is a very slight overhead as the store is opened and Windows may need to download missing root certificates. Only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components includes TMsCertChainEngine by default, with the CertVerMethod property selecting CertVerNone, CertVerBundle or CertVerWinStore. TMsCertChainEngine does include one extra optional feature to check if any certificates in the chain have been revoked by their issuer, perhaps for fraudulent use, beware revoke checks involve contacting each issuer and can slow down chain validation by a few seconds or even longer. Either of the certificate bundle files may be loaded into an SslContext by using the SslCAFile property. The built in bundle may be specified before the SslContext is initialised using SslCALines.Text property, or the LoadCAFromString method after initialisation. If checking a certificate chain, OpenSL will issue the error message 'unable to get local issuer certificate' if a trusted certificate is not found in the store. The contents the three certificates bundles are listed at http://wiki.overbyte.eu/wiki/index.php/FAQ_ICS_SSL/TLS_CA_Trusted_Store_Contents Angus

CA Trusted Store Background The whole SSL/TLS certificate verification process depends upon finding a trusted root certificate that signed the next certificate up the chain, and so on. But who chooses those trusted roots? The answer is the author of the application that accepts an SSL/TLS certificate, or maybe the operating system the application is running on, or the SSL library used to build the application. The bad news is that root certificates come and go, old roots stop being accepted for various reasons, and new roots are introduced for new security standards or new businesses. Historically most roots were RSA Sha1 digest signed and most still are, despite Sha1 being banned for new certificates. Increasingly Sha256 and ECC root certificates are being introduced and required by new intermediate certificates. Typically, active certificate authorities have at least four roots, with 2,048 and 4,096 bit RSA keys and 256 and 394 bit ECDSA keys (which are much smaller than RSA). Sources of CA Trusted Stores SSL/TLS trusted root certificate bundles and always changing, annually perhaps for major changes, although Microsoft officially Windows roots every two months. The Common CA Database (CCADB) https://www.ccadb.org/ is a repository of information about Certificate Authorities (CAs), and is used by a number of different root store operators to manage their root stores. But it's not easy to create root bundles from CCADB and another developer got frustrated with updating roots, and created a Trust Stores Observatory Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 500 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM bundle files, one each for the different operating systems. New PEM Bundle CA Trusted Store Files There are six different PEM CA bundle files, built from the Trust Stores Observatory Git repository: apple.pem - 174 Certificates google_aosp.pem - 137 Certificates microsoft_windows.pem - 289 Certificates mozilla_nss.pem - 137 Certificates openjdk.pem - 88 Certificates oracle_java.pem - 88 Certificates Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store. These bundles may be downloaded at: https://www.magsys.co.uk/download/software/ca-root-bundles.zip Magenta Systems Ltd will periodically update these bundles, as needed. ICS V8.63 contains the content of microsoft_windows.pem. Angus

The client and server logging should show if the control channel is closed prematurely. If you not already using my old TMagFtp client component, I'd suggest you update your client to use the latest version which is now in ICS, TIcsFtpMulti which uses special directory commands when working with the ICS FTP server, and automatically retries transfers when they fail due to control channel closing or other errors. Angus

The line that matters here relating to passive IP addresses is ; 227 Entering Passive Mode (27,316,15,428,191,207); which indicates a public address is being correctly advertised to the client, although you have not logged the server IP address so I'm guessing here. You are also using a two year old version of ICS. If even one file transfers correctly, that suggests the correct public IP address is being used. You need to check the logs to make sure the same public IP is being sent for each file. Another possibility is the public IP changed during the session, while most of us have stable public IP addresses that stay the same for weeks or years, some ISPs (mobile in particular) use CNAT and the public IP address may change every few minutes. Angus

I need to see the logs I asked you to keep. Angus

Just did a test with my FTP behind a NAT router, it currently fails to download files. It used to work with my old Sonicwall router which was intelligent enough to manipulate the FTP protocol to change the IP address, but no longer works with my current Draytek Vigor router which apparently does not understand FTP (but does work with IPv6 which the Sonicwall did not). Or maybe I've not configured something in the Draytek. It's probably PasvIpAddr would fix it, but I'd have to change my FTP server and I don't have time as the moment - I have three other public FTP servers so not a priority, sorry. Angus

Unfortunately the use of PasvIpAddr in the FTP server is undocumented and the developer that added and used those options is no longer able to answer questions about them. You'll need to examine your server and client logs and compare them with FileZilla to so what is wrong. I have an ICS FTP server behind a NAT router that works fine without any special settings except PasvPortRangeStart=21001 and PasvPortRangeSize=997, but I'm aware different NAT routers have varying abilities to cope with FTP. Angus

Your example is not thread safe, since AnswerString is being sent from within your own thread rather than the main thread where the client connection was opened. You should have AnswerDelayed := TRUE; in the handler, and then sent the Answer after the thread terminates. Web clients are unlikely to wait hours for a request to conclude, the web server has timeouts. Angus

i have made RFC1123_StrToDate more robust with error handling, it will be in SVN later today with other changes. Angus

Thanks, RFC1123_StrToDate is a very old function that probably precedes TryEncodeTime being added to Delphi, and has worked fine ever since. Using it would avoid an internal exception if the web server returns bad date headers, which is very rare, and you'd get a date at least if the time was invalid. I'll look into it. Angus

The correct way to resume FTP is to use the new TIcsFtpMulti component in OverbyteIcsFtpMulti.pas, with a demo in OverbyteIcsXferTst.dpr, it will automatically repeat failed transfers using resume if possible. There is a function to FTP a single file if you don't want it to automatically upload or download a nested directory structure. Angus

Which component or unit are you referring to? Angus

Sorry, ICS does not support SSH. There is an SSH terminal application on the ICS User Made web page, gut it was written in 2002 using old versions of OpenSSL and ICS,and is unlikely to still work. Angus

OpenSSL does have newer functions to format certificate information, but most users want single fields, not a list of cryptic fields. Angus

All the common subject and issuer fields are available as properties, ie SubjectCName, SubjectOName, IssuerCName, SubjectCOName, SubAltNameDNS, etc, or you can use GetNameEntryByNid with the NID literal for obscure ones like STREET, not even sure that exists... Look at the function ListCertDetail in OverbyteIcsPemtool1.pas which creates a string detailing most certificate fields. Angus

Only concerned about internal DER ASN_ENCODING format which ICS already supports, not file formats. OverbyteIcsPemtool1.pas already has code for opening a store, getting certificate contexts and converting them to X509 and PEM, need to move that into one of the library units, create a PCCERT_CONTEXT and add it to the correct store. Angus

ICS has code to read certificates from the Windows certificate store, but not to add them. Not needed for ICS applications generally, but for servers using SChannel. Angus

Thanks, added to my master, will be in SVN in a couple of days with other minor SSL improvements. Are you working with the Windows certificate store by any chance, by pending list includes putting certificates (Let's Encrypt) into the store. Angus

All Windows services run in their own thread already, TWSocketServer often runs in Windows services without needing extra threads. Using a thread for some client responses and not for others is exactly what the FTP servers does, it is no difficult, just make sure the response is sent in the main thread once SQL is done. Bringing TWSocketThrdServer up to date and creating TsslHttpThrdSrv is a major project and needs someone to sponsor it. My company effectively sponsors most ICS development since I create features mostly that are needed for our applications but which are then used by others without any cost. But so far I've not needed TsslHttpThrdSrv. It has been disscused in the past, it was going to be designed to handle x clients in a single thread before using another thread for x more, typically x would be 100 or more for a simple web server but could be one so each client gets a thread. Angus

Look at the new TIcsMailQueue component added to ICS earlier this year, it does exactly what you need. You can queue hundreds of emails at the same time, and they will be sent and retried over several minutes or days. The component is described half way down the page at http://wiki.overbyte.eu/wiki/index.php/ICS_V8.60 Angus

Not sure why you want to run the server in a thread, that just listens on one or more ports and creates clients. Running the client in a thread can be done, ICS has TWSocketThrdServer that does exactly that, but it is rarely used, has not been updated for 10 years, lacks newer multi-listen and SSL stuff and there is no web server derived from it, lack of demand from end users. My own TSslHttpAppSrv SQL driven web server runs happily in a single thread, the SQL stored procedures rarely take very long so are not blocking others users significantly, obviously this depends on volume, I'm only handling couple of thousand SQL requests a day, from the logging, SQL proc took 16ms, 31ms, 141ms, 63ms, 390ms, 16ms. So I never bothered with threads. As Francois says, using a thread just for the database lookup and returning the page in the main thread is the safest solution, look at class TClientProcessingThread in the ICS FTP server which uses a thread to calculate MD5sums and directory listing, Angus