Scanning for files while they are created or deleted

[Marus 0]

Hi !

I try to make a backup application... and I use FindFirstFile & FindNextFile Windows APIs to scan the directories. I need a strategy on how to make the scanning if the files are created or deleted in this time. It is difficult for me to create one because I don't understand how this APIs work... I made a test. I created 5 files in a folder and then I started scanning with FindFirstFile. Then I deleted the last 4 of them and continue to enumerate with FindNextFile, which, instead of ending the search with "no more files", because they were deleted, FindNextFile, always returns one file from the deleted ones, as if nothing had happened. It's as if they were cached since the execution of FindFirstFile... I'm really lost here. How would you do ?

[programmerdelphi2k 235]

maybe because after you trigger the procedure that verifies the files, it already has the names of the files that existed before your other procedure (or line) that triggers the "DeleteFile" command or similar... so, the list "contains the values old" before his "new" command!
Perhaps it would be better for you, right after sending the "delete" command, to cancel the old task and launch a new task to verify the new situation. Or who knows, find an API function that does this "in real time", but I think you may have the previous result or even a "false positive". So maybe you have to have a "delay" between the two tasks....

[DelphiUdIT 132]

I haven't used FindFile... for many years.

To list the files I suggest you to use the functions of the System.IOUtils unit, for example:

 

var FileList1: TStringDynArray

begin
FileList1 := TDirectory.GetFiles('.\Images\',  '*.*', TSearchOption.soAllDirectories);
for var ix := Low(FileList1) to High(FileLis1t) do
    begin
         //Do Something
         //This catches the last modified time of the file
         //TFile.GetLastWriteTime(FileList1[ix])
         //You can use the next line to check if the file really exists
         //FileExists(FileList1[ix]))
    end;
//Clear the list
SetLength(FileList1,0);
end;

 

Edited May 6, 2023 by DelphiUdIT

[programmerdelphi2k 235]

I think that is better this way:

• if necessary use "WinApi.Window.XXXXXX" to avoid conflict with others units with function-names, like "FINDClose" in "System.SysUtils and Winapi.Window"

function MyFindDeletedFilesOnList(AList: TArray<string>; AValue: string): boolean;
begin
  result := false;
  //
  if (length(AList) = 0) or (AValue.IsEmpty) then
    exit;
  //
  for var N in AList do
    if (N = AValue.Trim) then
      exit(true);
end;

procedure TForm1.Button2Click(Sender: TObject);
var
  LWFDw        : _WIN32_FIND_DATAW;
  LHandle      : THandle;
  LPathFolder  : PWideChar;
  LTrue        : boolean;
  LFilename    : string;
  i            : integer;
  LDeletedFiles: TArray<string>;
begin
  Memo1.Text  := '';
  i           := 0;
  LPathFolder := 'd:\MyFolder';
  LHandle     := FindFirstFile(PWideChar(LPathFolder + '\*.*'), LWFDw);
  LTrue       := (LHandle <> INVALID_HANDLE_VALUE);
  //
  // Nothing is guaranteed here, as another process (internal or external) can delete the file at any time!
  try
    while LTrue do
      begin
        LFilename := string(LWFDw.cFileName);
        //
        if (LFilename.Trim > '..') and MyFindDeletedFilesOnList(LDeletedFiles, LFilename) then
          begin
            LTrue := FindNextFile(LHandle, LWFDw);
            Memo1.Lines.Add('.... continue your search! the file ' + LFilename + ' was deleted');
            //
            continue; // next round...
          end;
          // else, you can create your list of all files not deleted too!
        //
        i := i + 1;
        //
        if (LFilename.Trim <= '..') then
          LFilename := '...... dir root and ancestral dir ......';
        //
        Memo1.Lines.Add(i.ToString + ', ' + LFilename);
        //
        // (i=2) it will avoid verify all time "if exists or not"!
        if  (i = 2) and FileExists(LPathFolder + '\a.jpg') then // delete after: "." and ".." ... next round will be "a.jpg"
          begin
            DeleteFile(LPathFolder + '\a.jpg');
            LDeletedFiles := LDeletedFiles + ['a.jpg'];
          end;
        //
        LTrue := FindNextFile(LHandle, LWFDw);
      end;
  finally
    if (LHandle <> INVALID_HANDLE_VALUE) then
      Winapi.Windows.FindClose(LHandle);
  end;
end;

initialization

ReportMemoryLeaksOnShutdown := true;

end.

Edited May 7, 2023 by programmerdelphi2k

[FPiette 363]

I don't see any issue getting file names which are already deleted. You will discover it when you open the file the the backup. Just handle that case. Anyway, you have to handle it is case of any other error.

You can be notified about changes in the file system by using FindFirstChangeNotification (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-findfirstchangenotificationa). Even if you get delete notification, you will need to check when you open the file because there will always be a race condition.

You'll also have an issue with files which are open at the time you open it for backup. To solve this, you need Windows Volume Shadow Copy Service (VSS) (https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service)

[dummzeuch 1389]

9 hours ago, Marus said:

I try to make a backup application... and I use FindFirstFile & FindNextFile Windows APIs to scan the directories. I need a strategy on how to make the scanning if the files are created or deleted in this time. It is difficult for me to create one because I don't understand how this APIs work... I made a test. I created 5 files in a folder and then I started scanning with FindFirstFile. Then I deleted the last 4 of them and continue to enumerate with FindNextFile, which, instead of ending the search with "no more files", because they were deleted, FindNextFile, always returns one file from the deleted ones, as if nothing had happened. It's as if they were cached since the execution of FindFirstFile... I'm really lost here. How would you do ?

FindFirstFile creates a snapshot of the files that exist at the time it is called and returns a handle to that list. FindNextFile then only enumerates over that already existing list. (You also need to call FindClose afterwards to free the internally used resources.) If during that time a file is created or deleted, the list remains unchanged. You will have to check for this in your code.

 

As fpiette said: You might want to use FindFirstChangeNotificaton etc. instead. But even then you will have to check whether the files still exist.

[Marus 0]

@dummzeuch And if the directory I scan has 5000 files, it will make a snapshot of all of them at start ? I think it will take a lot of time and FindNextFile will not make sense anymore.

 

@FPiette You are right ! But when I open the file at backup time and I got an error, I must report it in the log, but the files deleted by the user are not considered errors.  

 

I forgot to mention that my app is not the one who create or delete the files, the app is running in the background and it can happen that the user of the system do this.

 

I would have liked to read one file at a time, then copy it, and then move on to the next one, because if I read the entire list of files and then start copying them, during this time, some files may be deleted...

[Attila Kovacs 617]

You are overcomplicating the task.

The file can be deleted even when its last byte is read.

Just copy the files, catch all IO errors, and ignore the ones from the currenct file.

The backup will be still valid for that moment.

[dummzeuch 1389]

5 hours ago, Marus said:

@dummzeuch And if the directory I scan has 5000 files, it will make a snapshot of all of them at start ? I think it will take a lot of time and FindNextFile will not make sense anymore.

It takes a snapshot of the metadata of the files (name, size, properties ...), not the files themselves. That's how that function works, as far as I know. Whether you think it makes sense or not, doesn't affect that.

[timfrost 73]

Quote

I think it will take a lot of time

If you are concerned about performance, use the options in FindFirstFileEx to use a larger buffer and not to return 8.3 filenames.