active FTP confusion (how is this even possible?)

[merijnb 4]

I'm currently in the situation where I'm using an active FTP client while setting a port range for the data connection.

If a port range is given, the TCustomFtpCli.PortAsync() method in OverbyteIcsFtpCli will see which of the ports is available from given range, simply by trying to start a listening socket on the first port, go to the next if that excepts, etc.

 

During tests we have seen the strange behavior that when we start multiple FTP connections in rapid succession, it sometimes happens that multiple clients pick the same port for the data connection (so for example client b picks the same port as client a, while client a is still using that port). It's a bit hard to replicate in a consistent way, but the smallest I've been able to make it is by making an application which creates three TFtpClient objects, let them all connection to the same server, set to active FTP with a port range (7000-7010) to upload a small file. This happens from a single thread.

Something needed for this to happen seems timing related, I've not been able to reproduce this with an FTP server from ICS, but I did with Filezilla server. Obviously it shouldn't matter that much, since the port used for the data connection is chosen by the client, not the server.

 

I'm quite at a loss how this can happen, for two reasons:

 

1. The code in PortAsync() tries to setup a listening socket for each connection, apparently it's sometimes able to setup 2 different listening sockets on the same port, how is this possible?
2. How is it possible that (besides there are two sockets listening on the same port) it's actually possible to get data to the two different listening sockets while they're listening on the same port, how does the tcp stack in Windows know for which of the two sockets the data should be sent to?

 

I've added a zip file with logging from both the server and clients from my test, in it there is logging from filezilla server, a wireshark capture from the server's perspective, application log from the three clients (things like OnDisplayMessage from the TFtpClient), ICS log from all three clients (ICSLogger) and a wireshark capture from the client machine.

 

Besides I'm really curious what I'm missing here I wonder if this is a bug in ICS code (how is it possible it's able to start two listening sockets on the same port) and if it's something I should be worried about.

 

Thanks in advance, I'm really curious what comes out of this 🙂

ftp test log.zip

[Angus Robertson 577]

What is your FTP actually attempting to do?  Download from multiple FTP servers at the same time?   In one thread? 

 

It's unlikely I'll have time to investigate this or look at your logs or code unless this is a widespread problem. 

 

Angus

 

[merijnb 4]

2 hours ago, Angus Robertson said:

What is your FTP actually attempting to do?  Download from multiple FTP servers at the same time?   In one thread? 

 

It's unlikely I'll have time to investigate this or look at your logs or code unless this is a widespread problem. 

 

Angus

 

I'm uploading files to an FTP server, these uploads are triggered from outside of my app and handled by a single thread. Each trigger will start up an FTP client to upload a single file.

[Angus Robertson 577]

So you may have several copies of your FTP application running at the same time, uploading to the same server in parallel  Why not run a queue so the files are uploaded sequentially. 

 

Angus

 

 

[merijnb 4]

49 minutes ago, Angus Robertson said:

So you may have several copies of your FTP application running at the same time, uploading to the same server in parallel  Why not run a queue so the files are uploaded sequentially. 

 

Angus

 

 

No, it's one application.

 

Queuing is a possibility (or a fall back), but why? There is no technical reason to do so.
Multiple files can be uploaded simultaneously without problems.

[Remy Lebeau 1436]

9 hours ago, merijnb said:

The code in PortAsync() tries to setup a listening socket for each connection, apparently it's sometimes able to setup 2 different listening sockets on the same port, how is this possible?

Is ICS using the SO_REUSEADDR/SO_REUSEPORT socket options?  If not, then it should not be possible at all.

9 hours ago, merijnb said:

How is it possible that (besides there are two sockets listening on the same port) it's actually possible to get data to the two different listening sockets while they're listening on the same port, how does the tcp stack in Windows know for which of the two sockets the data should be sent to?

It is possible to have multiple TCP sockets listening on the same port at the same time ONLY if they are listening on different IPs.  Two sockets cannot listen on the same IP/port pair at the same time.

[Angus Robertson 577]

ICS does have a property ExclusiveAddr which sets SO_EXCLUSIVEADDRUSE, but it's not used by the FTP client, mostly by servers.  FTP client is really half client half, server in active mode.

 

I've still not had a proper response to how this application is designed, but it sounds like there are multiple instances of the FTP client running in the same thread, sharing a small pool of 10 ports, and each client starts from the beginning of that pool for each transfer getting errors if the port is in-use, then trying the next port.  If it was a single client, the ports would simply increment each time, ditto if Windows was allowed to assign the port.  This scenario really requires a common port pool between clients, but nobody has reported this problem before. 

 

But I'm not looking at this further until I understand what problem we are trying to fix.

 

One simply solution is for each FTP client to use a separate range of 10 ports, 21001 to 21010, 21011 to 21020, etc.  That avoids all conflicts and errors. 

 

Angus

 

Edited November 9, 2021 by Angus Robertson

[merijnb 4]

43 minutes ago, Angus Robertson said:

I've still not had a proper response to how this application is designed, but it sounds like there are multiple instances of the FTP client running in the same thread, sharing a small pool of 10 ports, and each client starts from the beginning of that pool for each transfer getting errors if the port is in-use, then trying the next port.  If it was a single client, the ports would simply increment each time, ditto if Windows was allowed to assign the port. 

Apologies, I thought that was clear by now. Your description is about right, this application will at some point need to upload a file to an FTP server, at that point it will create an FTPClient object and start the upload. In a specific case active FTP with a specified port range is required, and then we ran into this interesting behavior.

 

Quote

This scenario really requires a common port pool between clients, but nobody has reported this problem before.

I don't think that will really solve the issue since (at least for me) it's not clear yet how this is happening in the first place. If I debug this I can see clearly that every time one of the clients tries to decide what port to use (at the moment PORT command should be sent), it will try to start listening on the first port in the range, if it fails (since already in use) it will try the next. The principle in code seems to be ok. Yet somehow there seems to be a gap between a socket calling Listen() and that port actually being unavailable for another socket to listen on, at least, that is the only way I can imagine two separate FTP client instances will give the same port to the server to make the data connection on.

 

To me it seems there are two strange things about this:

 

1) how is it possible that two clients give the same port to the server for the active connection, that implies that two sockets were able to listen on the same port simultaneously which is not possible.

2) how is it possible the file transfers (seem) to work (see the wireshark logs). The only thing I can imagine is that FTP client 1 starts his data socket listening on port A, FTP client 2 for some reason (see point above) thinks he's also listening on port A, the server makes two connections back, both on port A, and they both actually connect to the listening socket from FTP client 1 (as far as I can see there is no check how many clients connect to the listening socket started by the FTP client). This kind of makes sense from socket perspective, but makes no sense at all from application perspective, that could never work (how would FTP client 1 be able to handle both data streams).

 

Now I can work around this in several ways, so this question is more out of interest (I see a learning opportunity here), what the heck is going on here 🙂

Edited November 9, 2021 by merijnb

[Angus Robertson 577]

Quote

at that point it will create an FTPClient object and start the upload.

Which resets the port pool to the start, it is not necessary to continually recreate components. 

 

As Remy pointed out, Windows can listen on the same port more than once deliberately, but applications are mostly written to avoid that happening and use unique ports.  Your use of the component is exposing the difficulties in handling these errors.  The solution is to use the component properly so that errors don't happen, as I've tried to explain. 

 

I will update with FTP client with ExclusiveAddr to stop duplicate listeners, but you should not rely on this.

 

Angus

[merijnb 4]

If I see correctly ReuseAddr isn't used, that means that TFtpClient itself cannot be the cause of reusing a port, you agree?

 

Quote

The solution is to use the component properly so that errors don't happen, as I've tried to explain. 

As I said, I have ways around this, this is more to satisfy my curiosity.

 

When you say I'm not using the component properly, what do you mean exactly? It's now allowed to use multiple instances of TFtpClient at the same time? I understand there are ways around this, but is there a technical reason you say this?

 

[Fr0sT.Brutal 900]

Honestly, "active" FTP mode is ugly, it violates client-server relationships and not gateway-friendly. So IMHO it's used rather rarely.

[merijnb 4]

Just now, Fr0sT.Brutal said:

Honestly, "active" FTP mode is ugly, it violates client-server relationships and not gateway-friendly. So IMHO it's used rather rarely.

I totally agree, unfortunately in this case I don't have any other option.

[Angus Robertson 577]

I have tested the threaded multi FTP component with hundreds of simultaneous sessions to the ICS FTP server, but not using a tiny pool of 10 ports, as I thought I explained clearly.  I would have been mostly testing SSL which always uses passive mode.  

 

That tiny port pool and always starting from the beginning of the pool is your design failure in attempting to support multiple sessions.  Sure the component handling of in-use ports could be better, but should rarely be needed in well designed applications.

 

Angus