Angus Robertson

Yes this issue was discussed here last week relating to the ICS FTP server behind a NAT router. In the past, routers were able to manipulate the control channel to change the private IP address to public, but it seems FTP is now considered a legacy protocol and newer routers don't bother. Also SSL stops the router seeing the control channel, one reason you can configure SSL on data but leave the control channel clear. I like FileZilla's idea of re-using the earlier public address, I'll have a look and see if that can be done in ICS, as an option. Angus

On my list. Angus

TriggerDataAvailable was never designed to be extendable. Apart from the three version in wsocket no other ICS units override it. So while you can override it, you need to replicate all the functionality like parsing line endings. Angus

Yes, I was being lazy and hoping no-one would use that feature on Delphi 7, I'll add your fix next week. Angus

The ICS library is designed to be used through event handlers, not by overriding internal functions. That requires a high level understanding of the library structure and is beyond the support I can offer here. Angus

I am not trying to write your code for you, just explaining why your function does not work. You have a conceptual issue trying to override library functionality. Angus

You are not using the Inherited code in other versions of TriggerDataAvailable so need to handle line mode yourself. Angus

When using LineMode, in the onDataAvailable event you use ReceiveStrA to get a complete raw line without needing to check line endings, but that just calls Receive and does all the hard work for you. If you are creating descendent components, you need to ensure the correct ancestor, which is not easy due to the deep hierarchy of wsocket. LineMode is handled in TCustomLineWSocket. Angus

You just ignore errors you don't want to handle. Not a good idea though. Angus

This depends on whether you are connecting synchronously or asynchronously. For sync, a simple loop is all that is needed, but your loop will be blocked for up to a minute during each connection attempt depending on TCP/IP timeouts, look at function FtpLogon in OverbyteIcsFtpMulti which does this for different IP addresses. For async, you use the onSessionConnected event which is called whether a connection is OK or fails. Look at SocketSessionConnected in OverbyteIcsIpStreamLog which handles retries after a delay. Angus

Sorry, don't understand the question. There is no TClientSocket in any ICS source unit, but client would suggest you referring to TWSocketServer. OnError is a notification event, you can not suppress errors there. Angus

Thanks, will fix that next week. Don't use the threaded version very much, just to test how many hundred simultaneous FTP sessions the ICS FTP server and client can support. Angus

For completeness, I have now configured my port forwarded FTP server correctly, opened passive ports in the Draytek router, and it is now working correctly again for internal and external traffic. However the ftpsNoPasIpAddrInLAN and ftpsNoPasvIpAddrSameSubnet are not working as expected, it never uses a local IP address, which I need to investigate. I suspect IPv6 may also be an issue on passive data connections, need to test that as well. But you can get around the LAN issue by changing the Passive IP in the event. Angus

ICS includes three CA CA Trusted Store, two as PEM bundle files, one in a source unit, and access to the Window Certificate Store directly: 1 - RootCaCertsBundle.pem is a large file that was originally created 15 years ago by exporting the Windows certificate store using the OverbyteIcsPemTool sample. But Windows 10 no longer has a complete local certificate store and instead downloads new certificates as needed by Windows browsers. So with ICS V8.63, it is now the same as the new microsoft_windows.pem bundle mentioned above. It currently contains 289 certificates and is 595 bytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 2 - TrustedCABundle.pem is a smaller file, with certificate for major commercial issuers manually updated as newer sites are found to have missing root certificates. But this file is more dynamic than RootCaCertsBundle.pem. It currently contains 51 certificates and is 88 Kbytes in size and may be found in the Samples/Delphi/SslInternet/ directory. 3 - To avoid distributing bundle files and as a fail safe if a file can not be found, ICS includes 34 built-in hard coded certificates in OverbyteIcsSslX509Utils.pas which can be returned as a string by the function sslRootCACertsBundle. Again this unit may be dynamic with new certificates added as needed. Note only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in bundle by default, other components need to add it manually to avoid the extra program code involved. 4 - ICS also includes a component TMsCertChainEngine in the unit OverbyteIcsMsSslUtils.pas which allows applications to avoid using bundle files and instead access the Windows Certificate Store directly to validate certificates. There is a very slight overhead as the store is opened and Windows may need to download missing root certificates. Only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components includes TMsCertChainEngine by default, with the CertVerMethod property selecting CertVerNone, CertVerBundle or CertVerWinStore. TMsCertChainEngine does include one extra optional feature to check if any certificates in the chain have been revoked by their issuer, perhaps for fraudulent use, beware revoke checks involve contacting each issuer and can slow down chain validation by a few seconds or even longer. Either of the certificate bundle files may be loaded into an SslContext by using the SslCAFile property. The built in bundle may be specified before the SslContext is initialised using SslCALines.Text property, or the LoadCAFromString method after initialisation. If checking a certificate chain, OpenSL will issue the error message 'unable to get local issuer certificate' if a trusted certificate is not found in the store. The contents the three certificates bundles are listed at http://wiki.overbyte.eu/wiki/index.php/FAQ_ICS_SSL/TLS_CA_Trusted_Store_Contents Angus

CA Trusted Store Background The whole SSL/TLS certificate verification process depends upon finding a trusted root certificate that signed the next certificate up the chain, and so on. But who chooses those trusted roots? The answer is the author of the application that accepts an SSL/TLS certificate, or maybe the operating system the application is running on, or the SSL library used to build the application. The bad news is that root certificates come and go, old roots stop being accepted for various reasons, and new roots are introduced for new security standards or new businesses. Historically most roots were RSA Sha1 digest signed and most still are, despite Sha1 being banned for new certificates. Increasingly Sha256 and ECC root certificates are being introduced and required by new intermediate certificates. Typically, active certificate authorities have at least four roots, with 2,048 and 4,096 bit RSA keys and 256 and 394 bit ECDSA keys (which are much smaller than RSA). Sources of CA Trusted Stores SSL/TLS trusted root certificate bundles and always changing, annually perhaps for major changes, although Microsoft officially Windows roots every two months. The Common CA Database (CCADB) https://www.ccadb.org/ is a repository of information about Certificate Authorities (CAs), and is used by a number of different root store operators to manage their root stores. But it's not easy to create root bundles from CCADB and another developer got frustrated with updating roots, and created a Trust Stores Observatory Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 500 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM bundle files, one each for the different operating systems. New PEM Bundle CA Trusted Store Files There are six different PEM CA bundle files, built from the Trust Stores Observatory Git repository: apple.pem - 174 Certificates google_aosp.pem - 137 Certificates microsoft_windows.pem - 289 Certificates mozilla_nss.pem - 137 Certificates openjdk.pem - 88 Certificates oracle_java.pem - 88 Certificates Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store. These bundles may be downloaded at: https://www.magsys.co.uk/download/software/ca-root-bundles.zip Magenta Systems Ltd will periodically update these bundles, as needed. ICS V8.63 contains the content of microsoft_windows.pem. Angus

The client and server logging should show if the control channel is closed prematurely. If you not already using my old TMagFtp client component, I'd suggest you update your client to use the latest version which is now in ICS, TIcsFtpMulti which uses special directory commands when working with the ICS FTP server, and automatically retries transfers when they fail due to control channel closing or other errors. Angus

The line that matters here relating to passive IP addresses is ; 227 Entering Passive Mode (27,316,15,428,191,207); which indicates a public address is being correctly advertised to the client, although you have not logged the server IP address so I'm guessing here. You are also using a two year old version of ICS. If even one file transfers correctly, that suggests the correct public IP address is being used. You need to check the logs to make sure the same public IP is being sent for each file. Another possibility is the public IP changed during the session, while most of us have stable public IP addresses that stay the same for weeks or years, some ISPs (mobile in particular) use CNAT and the public IP address may change every few minutes. Angus

I need to see the logs I asked you to keep. Angus

Just did a test with my FTP behind a NAT router, it currently fails to download files. It used to work with my old Sonicwall router which was intelligent enough to manipulate the FTP protocol to change the IP address, but no longer works with my current Draytek Vigor router which apparently does not understand FTP (but does work with IPv6 which the Sonicwall did not). Or maybe I've not configured something in the Draytek. It's probably PasvIpAddr would fix it, but I'd have to change my FTP server and I don't have time as the moment - I have three other public FTP servers so not a priority, sorry. Angus

Unfortunately the use of PasvIpAddr in the FTP server is undocumented and the developer that added and used those options is no longer able to answer questions about them. You'll need to examine your server and client logs and compare them with FileZilla to so what is wrong. I have an ICS FTP server behind a NAT router that works fine without any special settings except PasvPortRangeStart=21001 and PasvPortRangeSize=997, but I'm aware different NAT routers have varying abilities to cope with FTP. Angus

Your example is not thread safe, since AnswerString is being sent from within your own thread rather than the main thread where the client connection was opened. You should have AnswerDelayed := TRUE; in the handler, and then sent the Answer after the thread terminates. Web clients are unlikely to wait hours for a request to conclude, the web server has timeouts. Angus

i have made RFC1123_StrToDate more robust with error handling, it will be in SVN later today with other changes. Angus

Thanks, RFC1123_StrToDate is a very old function that probably precedes TryEncodeTime being added to Delphi, and has worked fine ever since. Using it would avoid an internal exception if the web server returns bad date headers, which is very rare, and you'd get a date at least if the time was invalid. I'll look into it. Angus

The correct way to resume FTP is to use the new TIcsFtpMulti component in OverbyteIcsFtpMulti.pas, with a demo in OverbyteIcsXferTst.dpr, it will automatically repeat failed transfers using resume if possible. There is a function to FTP a single file if you don't want it to automatically upload or download a nested directory structure. Angus

Which component or unit are you referring to? Angus