Problems downloading files from a FTP server using the ICS FTP Client. The DIR command has no carriage return or line feeds

[M-Brig 0]

Hello,

We are working with a number of companies that have devices with FTP servers on them. We are having problems extracting the files from these servers.

There are 2 devices we are having problems with all their other devices we have no problem downloading the files with the ICS FTP Client component.

The first problem we encountered was after requesting a directory all we would get back from the stream is the first file.

 

Code to request directory:
      FTPClient.hostname:=192.168.0.12;
      FTPClient.Passive:=True;
      FTPClient.timeout:=5;
      FTPClient.options:=options-[ftpbandwidthcontrol];
      FTPClient.ConnectionTpe:=ftpdirect;
      FTPClient.Multithreaded:=false;
      FTPClient.DisplayFileFlag:=false;
      FTPClient.Binary:=True;
      FTPClient.Port:=21;
      FTPClient.username:=ccccc;
      FTPClient.password:=hhhhhh;
      FTPClient.LocalFileName:=''MyList.LST'; 
      FTPFTPClient.directoryasync

Code inside FTPClientRequestDone Procedure
      Strm:=TFileStream.Create(ftpclient(sender).LocalFileName,fmOpenRead);
      try
        SetLength(S,Strm.Size);
        Strm.Read(PAnsiChar(S)^,Length(S));
        ACodePage:=ftpclient(sender).CodePage;
        portvars^.ftpmaps[ftpclient(sender).ftpdev].ftpstr.Text:=ConvertCodepage(S, ACodePage, CP_ACP);
        if portvars^.ftpmaps[ftpclient(sender).ftpdev].ftpstr.Count>0 then
          parse_ftp_directory(ftpclient(sender).ftpdev,ftpclient(sender).ftpprt);
      finally Strm.Free; end;

 

So, instead of using the stream we read directly from the MyList.LST file we defined in the FTPClient.localfilename. This allowed us to read all the available files in the path.  It turns out there are no carriage returns or line feeds after each entry in the DIR coming from the stream but fine when we read directly from the file. We asked our client to connect using the command line prompt on the devices that we are having problems with and with the devices that we have no problems with.  Please see the attached Word document.
Once we got past the problem with reading the directory we ran into another problem. After requesting the files using GetAsync we can only get the first file listed in the DIR out. All other attempts we get a 550 error code "Requested action not taken. File unavailable".

It turns out all the files listed in the DIR have a #0 at the end of every line. The attached MyList.LST file shows the null character at the end of every line.  

To do further troubleshooting we created a power shell application using the Powershell FTP client to test if it would be able to download the files and still no luck. If we keep the #0 at the end of the file when requesting the file we can only get the first file out. If we remove the #0 then we can not get any files out.

 

Has anyone seen a server act like this and is there anyway we can get these files out. The only way to get the files out without using the Device's manufacturer software is through File Explorer. 

 

Thanks for all your help.

 

FTP Ls and Dir commands.docx

MyList.LST

[Angus Robertson 574]

The ICS TFtpClient component has no directory handling, there are no formal standards for directory formats with the LIST command,  you should use the MLSD command which is supported by all proper FTP servers and is standardised. Otherwise your application is responsible for the different directory listing formats when using TFtpClient.  The ICS FTP server has even more efficient directory listing commands that handle sub-directories.

 

But you are probably using the wrong component, 15 years ago I wrote a higher level ICS FTP client component which is now part of the main ICS distribution as TIcsFtpMulti which is described briefly at http://wiki.overbyte.eu/wiki/index.php/ICS_V8.60 with a new sample application OverbyteIcsXferTst.dpr.  This handles all the directory listing stuff for you, at least for most known common FTP servers, there are always developers with strange implementations. and automates uploading and downloading whole directories and sub-directories of files. 

 

Angus

 

 

 

[Fr0sT.Brutal 900]

Very weird case. I'd recommend you to check that server with FileZilla and sniff the traffic of your app with simple SmSniff program. Then test with ICS sample FTP client.

[M-Brig 0]

Hello. Thank you for the answers.

 

We did not know about the MLSD command. Thank you we will use that in the future.

 

However, we can read the directory from the file but not from the stream. So that part we got past. We now need to get the files out. We can only get the first file in the directory from the server using GetAsync.

 

Have you ever heard of a #0 at the end of every directory listing. And it is required to be in the name when we request it from the server using GetAsync.  Does the #0 effect the server some how?

 

In our Powershell app we get the first file also and any other request we get the error: 421 Service Not Available, closing control connection.

 

Thanks for all your help. Stay safe and have a great weekend.

[Fr0sT.Brutal 900]

I've been working with FTP for decade and never heard of #0s in listings. The protocol is text one, it's not meant to use non-literal chars. However, these devices could run some weird servers where everything could happen. Probably filenames on these devices contain #0 (I'm not aware if Posix allows that but maybe).

 

Btw, I recently checked the servers I communicate with (pretty large ones, therefore running advanced server software) and neither of them allowed MLSD.

[Angus Robertson 574]

Most people consider FTP dead now, replaced by HTTP POST/PUT, and never bothered to update their 20 or 25 year servers with more efficient commands introduced since then.  

 

MLSD actually has an RFC somewhere, while the data returned by LIST is undocumented, and does not always have a year in the date, it was historically a Unix directory listing, thus only line feeds.  FileZilla Server supports MLSD. 

 

Angus

 

[Fr0sT.Brutal 900]

17 minutes ago, Angus Robertson said:

Most people consider FTP dead now, replaced by HTTP POST/PUT

Yeah and it makes me sad. HTTP has no standard listing format either so is very hard to automate.

[Remy Lebeau 1394]

11 hours ago, Angus Robertson said:

MLSD actually has an RFC somewhere

RFC 3659

11 hours ago, Angus Robertson said:

while the data returned by LIST is undocumented

Well, not just undocumented, but implementation-defined.  Servers can (and do) use whatever format they want for LIST.

 

[Angus Robertson 574]

ICS FTP server and client support extra commands based on MLSD: XDMSLD takes an argument -subdirs for recursive directories, while XCMSLD is similar but returns directory listings on the control channel to avoid opening a data connection to download what is often only a few lines of directory listings.  These commands make synchronising local and remote directory structures very efficient, which is what the ICS TIcsFtpMulti component does. 

 

I did think about writing an RFC back in 2008, but did not really expect any other FTP servers to implement the commands, even then FTP was going out of fashion.

 

Angus

 

[Fr0sT.Brutal 900]

Nevertheless there's no good replacement at all. HTTP interface won't let you use a client app where you can just select multiple directories and download them all recursively.

[M-Brig 0]

Hello,

 

Thank you again for your responses. Unfortunately, these devices are brand new.  

 

I checked in their manual and they do not support HTTP Post/Put. FTP is the only way to get files out of these devices. 

 

Only the manufacturers software can extract the files and File Explorer. However, File Explorer is a manual process and our clients are looking for an automated process which our software supports.

 

Does anyone know what type of FTP client File Explorer uses? This may help us figure out the problem. The FTP from the command prompt and Power shell's FTP client do not work on these devices.

 

Thanks for all your help.   

Edited April 6, 2020 by M-Brig

[Angus Robertson 574]

Have you actually tried the OverbyteIcsXferTst.dpr sample I mentioned last week yet?  I think you are trying to complicate something that is almost working. 

 

Angus

 

[Fr0sT.Brutal 900]

8 hours ago, M-Brig said:

Does anyone know what type of FTP client File Explorer uses?

First you've got to tell what File Explorer is...

Why don't you just simply sniff the traffic instead and see what's really going on?

[aehimself 396]

11 hours ago, M-Brig said:

Does anyone know what type of FTP client File Explorer uses? This may help us figure out the problem. The FTP from the command prompt and Power shell's FTP client do not work on these devices.

If File Explorer is the standard Windows file management application which pops up when you click on My Computer / This PC; and you just type ftp://host_or_ip_of_device it's hardwired in the shell for sure. I am also certain that Microsoft did not maintain this Client as they saw the future in WebDav; not FTP; but to be honest it's irrelevant in your case. You said your only issue is that your device is inserting a null character at the end of each file upon listing, which you save to a file. I'm not sure if Delphi is handling #0 characters in strings since Unicode; but you can try:

Var
 sl: TStringList;
 a: Integer;
Begin
 sl := TStringList.Create;
 Try
  sl.Delimiter := #0;
  sl.LoadFromFile('MyList.lst');
  For a := 0 To sl.Count - 1 Do
   // Do something with the file at sl[a]
 Finally
  sl.Free;
 End;
End;

Or...

Uses System.IOUtils;

Var
 sa: TArray<String>;
 fname: String;
Begin
 sa := TFile.ReadAllText('MyList.lst').Split([#0]);
 For fname In sa Do
  // Do something with the file in fname
End;

and then using any FTP client component you issue a download request.

 

Edit:

 

It does.

Var
 sa: TArray<String>;
 fname: String;
Begin
 sa := String('Hello' + #0 + 'World' + #0 + 'Zero char' + #0 + 'separation').Split([#0]);
 For fname In sa Do
  ShowMessage(fname);
End;

works, popping up one segment at a time; so it should do the trick.

Edited April 6, 2020 by aehimself

[M-Brig 0]

Hello Everyone,

 

Thank you for your responses. 

 

Angus, we have no access to these devices. We are relying on the techs to go in and do the testing.

 

But, good news. I checked with our development team and they did get to test with removing the #0 before requesting the file. It worked from our Power Shell application but we have not had a chance to test with Delphi yet. 

 

Since we have to wait days between testing I thought they performed this test already and it did not work but I was wrong.

 

If anyone encounters any problems extracting files from an FTP Server we suggest checking to make sure there are no non literal characters in the LST or DIR commands. 

 

Thanks for all your help.

 

Stay Safe.