Angus Robertson

If you are trying to add support for DTLS and PSK to ICS using existing code, I might be able to help, but not to write a totally new SSL implementation using APIs, ignoring all our tested high level code.  I have no experience of DTLS or PSK.

Angus

libeay32.dll and ssleay.dll were used by obsolete versions of OpenSSL, support ceased for those versions at the end of last year, so they have been removed from the ICS distribution.  But we;'ll leave support for the old version in ICS itself for nine more months, until OpenSSL 3.0 is released and supported by ICS, which will be libss1-3_0.dll, etc.

None of this relates to the PORT problem you reported.

Angus

The only web page kept up to date with downloads is: http://wiki.overbyte.eu/wiki/index.php/ICS_Download

Can not diagnose anything from a one line error, but this mostly happens when you are NAT routers with public and local IP addresses.  You should be using Passive mode to avoid this if you using TLS then passive mode should be forced, so not sure what you doing without more of the log.

Angus

Beware when updating OverbyteIcsSSLEAY.pas and OverbyteIcsLIBEAY.pas they change when I'm adding new features to ICS, make sure you use the latest from SVN last week, there are minor changes not in SVN yet. 

We replicates macros as functions in OverbyteIcsLIBEAY which is where you should add any more you need, look at function f_BIO_get_ssl which calls f_BIO_ctrl, you need to add f_BIO_set_conn_hostname similarly, etc.,

Use of macros is horrible for those of us not using C++, even Google has turned them all into APIs in BoringSSL.  The main problem is when OpenSSL convert macros back into APIs, this is never documented clearly so we keep using our macro function which then usually fails. 

Angus

You are probably using an old version of ICS, Samples\Delphi\SslInternet\OverbyteIcsXferTst.dpr was added about a year ago. OverbyteIcsFtpMultipartDownload is something totally different, no use to you.

Angus

Build the OverbyteIcsXferTst.dpr sample which uses the OverbyteIcsFtpMulti, the sample has a simple menu with the various secure FTP options.

Angus

Where ever you check the 404 error send the windows message to start another GetAsync request.  Or use sync methods when you can do a simple loop and break when it works, Get instead of GetAsync it all depends on how your application designed. 

Angus

Have you actually tried the OverbyteIcsXferTst.dpr sample I mentioned last week yet?  I think you are trying to complicate something that is almost working. 

Angus

ICS FTP server and client support extra commands based on MLSD: XDMSLD takes an argument -subdirs for recursive directories, while XCMSLD is similar but returns directory listings on the control channel to avoid opening a data connection to download what is often only a few lines of directory listings.  These commands make synchronising local and remote directory structures very efficient, which is what the ICS TIcsFtpMulti component does. 

I did think about writing an RFC back in 2008, but did not really expect any other FTP servers to implement the commands, even then FTP was going out of fashion.

Angus

Using a self signed certificate for localhost is not going to be a very friendly OAuth2 user experience, since the browser will display certificate warnings, and not everyone can work out which buttons to click to display the real page. 

Does Visma have suggestions for that?  To me sounds like someone thought SSL would sound good without actually testing it.

Angus

Look at StartDomSrv in OverbyteIcsSslX509Certs.pas which sets up the simple web server with a newly generated localhost certificate, and CreateAcmeAlpnCert which will create a normal non-ALPN certificate if you leave KeyAuth blank, but that does not matter for your purposes.  

Which OAuth2 system requires HTTPS for the callback?  Seems over the top since only your local browser displays the result.  We should probably handle that in the REST component properly.

Angus

Most people consider FTP dead now, replaced by HTTP POST/PUT, and never bothered to update their 20 or 25 year servers with more efficient commands introduced since then.  

MLSD actually has an RFC somewhere, while the data returned by LIST is undocumented, and does not always have a year in the date, it was historically a Unix directory listing, thus only line feeds.  FileZilla Server supports MLSD. 

Angus

When you improve the logging in the server to show errors and events, similarly to the sample you copied code from,. you'll get some debugging information.  See SslFtpServer1SslVerifyPeer,  SslFtpServer1SslHandshakeDone, 

And also fix the logging bug that is adding this path, not coming from the clients, HOST C:\TEMP\[127.0.0.1]

Angus

I always test the ICS FTP server with the ICS OverbyteIcsXferTst sample, then you just click a single button to perform a complete FTP session, with proper logging.  

Angus

What it should say is:

02:10:08  > AUTH TLS
02:10:08  < 234 Using authentication type TLS
02:10:08  Check for Old SSL Session
02:10:08  No Old SSL Session Cached
02:10:08  Starting SSL Session
02:10:08  Cache SSL Session: New
02:10:08  ! SSL Connected OK with TLSv1.2, cipher ECDHE-RSA-CHACHA20-POLY1305, key auth RSA, key exchange ECDH, encryption CHACHA20/POLY1305(256), message auth AEAD
 

But that depends on how many other events you are logging, you cam see in the sample where all this comes from (although this log is from a real server).  I suspect you are still testing with the OpenSSL client which to be honest is a waste of time, it does not understand the FTP protocol.   With TCP, it is rarely obvious which end caused disconnection. 

Angus

The FTP server sample does all the logging properly,SslFtpServer1AnswerToClient.

If you use IcsHosts in the latest server, you don't need an SslContext, all that is handled by the component. You set the certificates and an SSL security level and everything else is done automatically, including installing Let's Encrypt SSL certificates on a public server.  This all works now, just no new sample yet. 

Angus

Improving your logging will help, you did not report the response or errors to commands received which is essential to error tracing with bad clients. 

The latest ICS in SVN has FTP server improvements to simplify SSL configuration in the same way as the web and proxy servers two years ago using IcsHosts, but there is no sample to copy yet, you need to look at the multi-web server sample and see how that configures listeners and at http://wiki.overbyte.eu/wiki/index.php/FAQ_Using_IcsHosts. 

Angus.

The ICS TFtpClient component has no directory handling, there are no formal standards for directory formats with the LIST command,  you should use the MLSD command which is supported by all proper FTP servers and is standardised. Otherwise your application is responsible for the different directory listing formats when using TFtpClient.  The ICS FTP server has even more efficient directory listing commands that handle sub-directories.

But you are probably using the wrong component, 15 years ago I wrote a higher level ICS FTP client component which is now part of the main ICS distribution as TIcsFtpMulti which is described briefly at http://wiki.overbyte.eu/wiki/index.php/ICS_V8.60 with a new sample application OverbyteIcsXferTst.dpr.  This handles all the directory listing stuff for you, at least for most known common FTP servers, there are always developers with strange implementations. and automates uploading and downloading whole directories and sub-directories of files. 

Angus

I have no idea how the openssl client tool works or why you are using it to test FTP, but 'AUTH C:\TEMP\TLS' is not a valid command, it should be AUTH TLS'. You have not shown what smartftp sent.

Since the ICS sample works, you have missed something in your implementation but I can not debug your code. 

Angus

You don't log the IP address and port your server is using, although your client says port 5420 which is not an SSL FTP port, so the server does not attempt to negotiate SSL   Using non standard ports for SSL requires extra code.  V8.64 has a new Option ftpsAuthForceSsl that might do it simply.

Not sure why you are testing an FTP servers with a non-FTP client that does not send the correct protocol. 

Angus

Not sure what you are trying to achieve, but I suggest you get the original ICS sample working first, then work out what is different with your project. 

The IcsLogger is for our development purposes, not debugging applications.  Your code has lots of Logit commands, it's what those report that helps with debugging.

Angus

Quote

The one and only way to do this is how i described .. what i told you is best practice and how it is done, there is no other way 

So how do you suggest the password is entered for a background service application on a hosted server?   Using a token from an authentication server is fine, but how do you get it?

Angus

The code itself is probably OK if you copied it correctly. the issue is more likely all the settings and SSL/TLS certificates that are needed to make a secure server work, of which you show none.  

You could also have shown the protocol that you logged with all the errors.   What SSL certificate are you using, and what intermediate, since you are setting SslCaFile.  But servers don't use SslCAPath.

Angus

Updated OpenSSL to 1.1.1f, this reverts a change in 1.1.1e that caused problems in some applications, but I don't believe ICS was effected.

Angus

Encrypting a string or password is easy, harder is how you protect the key used for the encryption. 

Angus