dummzeuch

I must be doing something wrong here: var Stopwatch: TStopwatch; begin Stopwatch.Start; sleep(3000); Stopwatch.Stop; m_Result.Lines.Add(format('Took %.3f seconds', [Stopwatch.Elapsed.TotalSeconds])); end; The result is "-130847749795,796 seconds", I was, of course, expecting something around 3. What am I missing?

One of the bad habits I have picked up over the years is not looking into the documentation. The only reason I can give is that the documentation was so bad for some time that the time it took usually was wasted. Since documentation has become much better (but is still far from perfect, especially some of the examples), this no longer applies though.

What i don't need are snide remarks like this.

Conclusion: It doesn't matter whether I use Breadth First or Depth First Search, the performance is the same. Switching from SysUtils.FindFirst/FindNext to Windows.FindFirstFileEx/FindNextFile with only basic file information and the flag FIND_FIRST_EX_LARGE_FETCH brought me a performance gain of about 25% which is not too bad. I found no way to benefit from using TDirectory. The only test I made got worse performance than FindFirst/Next. Test environment: Windows 8.1 client accessing a share on a Samba server over 1 GBit Ethernet with minimal traffic (it's a weekend). The computer is a reasonably fast Intel Xeon E3 with 3.4 GHz with 8 cores and 16 GByte RAM. The Server has an Intel Core I9-9940X processor with 16 cores and with 64 GBytes of RAM. The share is stored on a 2 TB Intel M2 NVMe SSD. It's running Ubuntu 18.04.4 LTS and Samba 4.7.6-Ubuntu. The test program was compiled with Delphi 10.4. There was a no noticable difference between building with Debug and Release config. The directory tree is 4 levels deep and contains only a few directories on the first 3 levels but a total of 898 directories on the deepest level. Each of these directories on the deepest level contain from several hundred up to several thousand jpeg files. The purpose of the code is to find all directories on any level containing at least one jpeg file but no subdirectories. All tests were run several times in varying order to prevent any caching effects to distort the result.

As promised, here the BFS version using FindFirstFileEx. And as expected, it also takes about 45 seconds in my test setup. function FindFirstFileEx(lpFileName: LPWSTR; fInfoLevelId: TFindexInfoLevels; lpFindFileData: Pointer; fSearchOp: TFindexSearchOps; lpSearchFilter: PWin32FindData; dwAdditionalFlags: DWORD): THandle; stdcall; external kernelbase Name 'FindFirstFileExW'; function FindNextFile(hFindFile: THandle; lpFindFileData: PWin32FindData): BOOL; stdcall; external kernelbase Name 'FindNextFileW'; procedure TForm1.b_FindFirstExNextBFSClick(Sender: TObject); var DirsWithJpg: TArray<string>; DirsToSearch: TStringList; procedure CheckDirectory(const _Dir: string); const FIND_FIRST_EX_LARGE_FETCH = 2; var DirBs: string; SearchHandle: THandle; sr: TWin32FindData; ContainsFiles: boolean; ContainsSubdirs: boolean; fn: string; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); SearchHandle := FindFirstFileEx(PChar(DirBs + '*.*'), FindExInfoBasic, @sr, FindExSearchNameMatch, nil, FIND_FIRST_EX_LARGE_FETCH); if SearchHandle <> INVALID_HANDLE_VALUE then begin try repeat fn := sr.cFileName; if (fn = '.') or (fn = '..') then begin // ignore end else if (sr.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) <> 0 then begin // directory ContainsSubdirs := true; // add to list for later processing DirsToSearch.Add(DirBs + fn); end else if sr.dwFileAttributes and (FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM or FILE_ATTRIBUTE_REPARSE_POINT) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(fn), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until not FindNextFile(SearchHandle, @sr); finally Winapi.Windows.FindClose(SearchHandle); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; DirsToSearch := TStringList.Create; try DirsToSearch.Add('\\server\share\dir'); while DirsToSearch.count > 0 do begin CheckDirectory(DirsToSearch[0]); DirsToSearch.Delete(0); end; finally FreeAndNil(DirsToSearch); end; Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirstExNext BFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end;

OK, so here is the DFS implementation using FindFirstFileEx and only getting basic info. In my tests on the same computer and with the same Samba server and share it only takes about 45 seconds (rather than the 60 seconds with SysUtils.FindFirst). The result is the same. For completeness I am going to implement BFS for this too, but I don't expect any change really. function FindFirstFileEx(lpFileName: LPWSTR; fInfoLevelId: TFindexInfoLevels; lpFindFileData: Pointer; fSearchOp: TFindexSearchOps; lpSearchFilter: PWin32FindData; dwAdditionalFlags: DWORD): THandle; stdcall; external kernelbase Name 'FindFirstFileExW'; function FindNextFile(hFindFile: THandle; lpFindFileData: PWin32FindData): BOOL; stdcall; external kernelbase Name 'FindNextFileW'; procedure TForm1.b_FindFirstExNextDFSClick(Sender: TObject); var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); const FIND_FIRST_EX_LARGE_FETCH = 2; var DirBs: string; SearchHandle: THandle; sr: TWin32FindData; ContainsFiles: boolean; ContainsSubdirs: boolean; fn: string; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); SearchHandle := FindFirstFileEx(PChar(DirBs + '*.*'), FindExInfoBasic, @sr, FindExSearchNameMatch, nil, FIND_FIRST_EX_LARGE_FETCH); if SearchHandle <> INVALID_HANDLE_VALUE then begin try repeat fn := sr.cFileName; if (fn = '.') or (fn = '..') then begin // ignore end else if (sr.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) <> 0 then begin // directory ContainsSubdirs := true; // recursive Depth First Search CheckDirectory(DirBs + fn); end else if sr.dwFileAttributes and (FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM or FILE_ATTRIBUTE_REPARSE_POINT) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(fn), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until not FindNextFile(SearchHandle, @sr); finally Winapi.Windows.FindClose(SearchHandle); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir); Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirstExNext DFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; I redeclared FindFirstFileEx and FindNextFile for this because the declarations in WinApi.Windows are inconsistent in the declaration of the lpfFindFileData parameter. Now both declare it as PWin32FindData.

My tests show no performance difference at all between DFS und BFS using FindFirst/FindNext. Both take nearly exactly 60 seconds on my test machine traversing the same directory tree on a Samba server. Trying to use TDirectory did not result in any improvement. It takes about twice as long. DFS code: var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); var DirBs: string; sr: TSearchRec; ContainsFiles: boolean; ContainsSubdirs: boolean; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); if FindFirst(DirBs + '*.*', faAnyFile, sr) = 0 then begin try repeat if (sr.Name = '.') or (sr.Name = '..') then begin // ignore end else if (sr.Attr and faDirectory) <> 0 then begin // directory ContainsSubdirs := true; // recursive Depth First Search CheckDirectory(DirBs + sr.Name); end else if sr.Attr and (faHidden or faSysFile or faSymLink) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(sr.Name), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until FindNext(sr) <> 0; finally FindClose(sr); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir'); Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirst/Next DBF: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; BFS code: var DirsWithJpg: TArray<string>; DirsToSearch: TStringList; procedure CheckDirectory(const _Dir: string); var DirBs: string; sr: TSearchRec; ContainsFiles: boolean; ContainsSubdirs: boolean; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); if FindFirst(DirBs + '*.*', faAnyFile, sr) = 0 then begin try repeat if (sr.Name = '.') or (sr.Name = '..') then begin // ignore end else if (sr.Attr and faDirectory) <> 0 then begin // directory ContainsSubdirs := true; // add to list for later processing DirsToSearch.Add(DirBs + sr.Name); end else if sr.Attr and (faHidden or faSysFile or faSymLink) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(sr.Name), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until FindNext(sr) <> 0; finally FindClose(sr); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; DirsToSearch := TStringList.Create; try DirsToSearch.Add('\\server\share\dir'); while DirsToSearch.count > 0 do begin CheckDirectory(DirsToSearch[0]); DirsToSearch.Delete(0); end; finally FreeAndNil(DirsToSearch); end; Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirst/Next BFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; The naive approach using TDirectory takes about twice as long because of the calls to .GetFiles and .GetDirectories each call FindFirst/FindNext per directory. (It also finds 2 less directories containing jpg files, so there is probably a bug somewhere, but I didn't investigate this any further.) var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); var Files: TArray<string>; Dirs: TArray<string>; i: Integer; begin Files := TDirectory.GetFiles(_Dir, '*.jpg'); Dirs := TDirectory.GetDirectories(_Dir); if Length(Files) > 0 then begin if Length(Dirs) > 0 then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else DirsWithJpg := DirsWithJpg + [_Dir]; end else begin for i := Low(Dirs) to High(Dirs) do CheckDirectory(Dirs[i]); end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir'); Stopwatch.Stop; m_Result.Lines.Add(Format('DirGetFiles: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; I thought about using TDirectory.FileSystemEntries instead but could not think of a simple way to implement this. It's probably possible using a Predicate but that's not really any simpler than directly using FindFirst/FindNext. There is probably still some space for improvements and the code is not really clean, e.g. accessing variables of the outer procedure. So, now I'll have a look at calling Windows.FindFirstFileEx instead of FindFirst, but I don't have much hope that this will help much.

Yes, I'm sure it could.

Ouch! Forget it: I didn't reset the stopwatch first: var Stopwatch: TStopwatch; begin Stopwatch.Reset; //<<<--- this was missing Stopwatch.Start; sleep(3000); Stopwatch.Stop; m_Result.Lines.Add(format('Took %.3f seconds', [Stopwatch.Elapsed.TotalSeconds])); end;

On the other hand, if I only want to create one installer, it's easier to temporarily edit the makeinstallers.cmd, where I can simply copy one existing line rather than having to remember exactly which parameters I'd have to pass: rem [...] rem *** copy one line from the list below here *** rem *** and remove two REMs *** rem call :makeinst RS103 RS10.3 rem pause rem goto :eof :list call :makeinst Delphi6 D6 call :makeinst Delphi7 D7 call :makeinst Delphi2005 call :makeinst BDS2006 call :makeinst Delphi2007 call :makeinst RS2009 call :makeinst RS2010 call :makeinst RSXE1 call :makeinst RSXE2 call :makeinst RSXE3 call :makeinst RSXE4 call :makeinst RSXE5 call :makeinst RSXE6 call :makeinst RSXE7 call :makeinst RSXE8 call :makeinst RS100 RS10 call :makeinst RS101 RS10.1 call :makeinst RS102 RS10.2 call :makeinst RS103 RS10.3 call :makeinst RS104 RS10.4 rem [...]

That might for a change actually be an option, as the program in question is developed with Delphi 10.2 rather than my usual work horse Delphi 2007. I'm not yet used to those "new" utility units. I just had a look at the sources. As @Vandrovnik already said: It internally also uses FindFirst / FindNext in the same way my program did, so I don't expect any improvements. WalkThroughDirectory also recursively does a depth first search. In addition it calls two different callbacks, which won't improve performance either. I'll now try to use FindFirstEx and also implemtent BFS and do some timing on each.

Thanks, I missed that one. As for the others I now know why I didn't see those errors: I compiled these projects with Delphi 2007 only. Maybe I should switch to a more modern version.

all should be fixed now in svn

This works fine for me without the change and regardless whether I do a command line build (__build_Project.cmd) or use the IDE. The GXpertsRS103 project contain the ..\..\ExternalSource directory in the search path for "All configurations - All platforms", so the change should not be necessary. Or was this for one of the stand alone projects too?

Works for me, but is a bit sluggish.

In my installation nothing has changed in this respect. The project window can be pinned and unpinned, where "unpinned" means it's just a small panel to the right that makes the window visible when clicking on it. Maybe undocking and docking it again will solve the problem?

If you want to use the"stand alone" tools in GExperts, you might find that they don't work if the bpl files it needs are not in the search path. These tools call the dll, the dll is an IDE plugin, an IDE plugin needs some bpls, so there is no way around that (other than making those tools not use the dll).

That might for a change actually be an option, as the program in question is developed with Delphi 10.2 rather than my usual work horse Delphi 2007. I'm not yet used to those "new" utility units. Thanks for the pointer.

Thanks for the hints. I haven't checked this stand alone tools for some time. I'm surprised about the change to gx_dbugintf. I'll need to have a closer look. Also thanks for the changed prebuild.cmd

bug #192 Message Dialog Expert form is too small in Delphi 10.4

https://blog.dummzeuch.de/2018/08/18/gnugettext-support-for-the-gexperts-message-dialog-expert/ You can also turn it off, there is a checkbox for it in the dialog.

No, you will get new licenses for all included Delphi versions and in the case of the named user licenses for each user. The old licenses will continue to work for a while and then Embarcadero will invalidate them. I also got a link for a special web page that lists downloads for all my included Delphi versions. I don't remember if I had to ask for it or got it automatically. Unfortunately, the notification email I get when a new Delphi release is available always contained a wrong download link and I keep forgetting about the special page I mentioned above. But support was always helpful in that case.

So, does that mean we have a Delphi 10.4 release?

For what it's worth: We hired a new Delphi developer about 4 years ago (because the one we hired 10 years ago left). And we are constantly creating new and improving old programs that are used internally, and some that are used in our parent company and sold to costumers who buy our custom built measurement vehicles or data. We recently started a new tool that required Delphi 10.2 features and even bought two new licenses for that. And I'm waiting for Delphi 10.4 in the hope that one particular bug in the IDE has been fixed that prevented another new tool to move forward. But we might be an exception, because I'm a fan of Delphi and I'm making most of the decisions regarding the development environment. Maybe my successor will decide to drop all that "legacy" stuff and start rewriting everything with "modern" technology. If he is lucky, the company might even survive such a move, barely, and with a software department of 10 instead of 3 people. But hey, that would create 7 new jobs.

The answer is equally simple: Once it has been activated, No.