dummzeuch

OK, so I understand that there might be the possibility of an Int64 overflow, here: which could result in Pos becoming negative. On the other hand that would mean a stream with about MaxInt64 bytes of data, which is quite a lot: 9,223,372,036,854,775,807 or 2^33 gibibytes. Do 64 bit processors actually have the capacity of addressing that much (virtual) memory? Are there any other bugs in my implementation? There might also be no performance gain at all due to highly sophisticated features of the CPU. And of course: There might be other possible improvements If I want this to be optimized I should submit a report to Embarcadero Why did I actually bring this up? I was looking at some code that was extensively using TMemoryStream as a buffer for various operations, where it accessed the Position and Size properties all the time and thought about the possible overhead of these method calls. Then I looked at the implementation of GetSize and found 3 calls to Seek in there, just to eventually get the value of a field. And then I looked at various other methods of T(Custom)MemoryStream and thought WTF is this? That I used the Write method as an example above was pure chance, it was just the last method I had been looking at.

In ISO 9001 consistency is everything. You must have rules and follow them. It doesn't matter whether the result is correct or not, it must reproducible every single time. (OK, I'll stop now, I just have suffered too much pain due to this idiocy.)

No, there is no way to change that order. But there is the configurable Editor Popup Menu (in Editor Experts), which might help you there.

Given a large directory tree with many files and sub directories and a depth of about 5 located on a Samba server share which must be fully traversed: What is more efficient: Depth First Search: Recursive FindFirst/Next which immediately handles each sub directory when it is encountered. Breadth First Search: Iterative FindFirst/Next which maintains a list of directories to process and appends sub directories to this list when they are encountered to be processed later. I have so far implemented DFS, simply because it was the easiest. It takes a very long time (about 60 seconds in my test scenario). Before I try to implement BFS: Has anybody ever compared the performance of these approaches on Windows + network shares? I googled, but could not find anything definitive (which really surprised me), maybe my google fu has let me down again.

I did some more tests: On Windows 10 1909 on a physical computer and a local drive I found that it makes no difference whether I use FindFirstFileEx or FindFirstFile. On Windows 10 1909 on a Samba share (same as in the first tests) there is still an advantage of using FindFirstFileEx, but it isn't as big as in my tests on Windows 8.1. Since the contents of the drive have changed in the meantime and the Windows 10 installation is on a virtual machine while the Windows 8.1 installation is on a physical computer I can't meaningfully compare them to the original tests.

I'm happy to report that the problem with the icons in the popup menus causing empty entries seems to have been fixed in Delphi 10.4. So I have now enabled them in GExperts again.

It doesn't do anything at all. And if you remove the conditional define testWizard, it doesn't get installed. That item was just a convenient way to test some possible wizard functionality without having to write a new plugin.

The GExperts Rename Components expert has some limited functionality similiar to Quick Edit and it has a keyboard shortcut. https://blog.dummzeuch.de/2018/07/21/selecting-alignment-and-anchors-in-the-rename-components-expert/ Adding a keyboard shortcut for Quick Ediit is on my wish list, but not as easy as I expected. I already wasted several hours on this.

Once I've restored the backup of my data hard disk (which crashed on me last weekend), I will have a look and tell you what it does. Until then, you will either have to look yourself or live with the suspense. 😉

I wonder what that menu item does? 😉 Acutally I have no idea, it's probably a leftover from some testing code which I forgot to remove again. This is the first time I heard from somebody else who actually used that plugin.

I have seen this error several times already: I close a project and get the error in the title. Has anybody else experienced this?

Hey, give us something to laugh! Would make for a nice change for me right now.

Since I have no idea which functionality of the IDE FixInsight might hook (or even which events or maybe what else it might in stall), I have no idea where to look. I know that there are still some issues with GExperts crashing when exiting the IDE but none of them so far was reproducible enough to track down the issue. And as David Heffernan said: It could be anything. There aren't many safe guards inside the IDE to separate itself from plugins and plugins from each other.

GExperts has an option to automatically close it on successful compile. Maybe you have enabled it? (Default is disabled though.)

I revisited my dzBdsLauncher tool again – no idea why, it just occurred to me 😉 – and added quite a few improvements: It now supports .dof (Delphi 6 and 7) and .bdsproj (Delphi 2005 and 2006) files. In addition to the previous checks it now also looks at the disabled packages list to determine which Delphi version to start. That’s the only option for Delphi 2005 and 2006 because these files are nearly identical. It can now also handle .dpr files by looking for corresponding .dproj, .bdsproj and .dof files (in that order) and taking these to determine the correct Delphi version. As a side effect I found a problem with the Delphi 10.1 version of the GExperts .dproj file. It had a wrong ProjectVersion entry. The blog post with download link is here.

I must be doing something wrong here: var Stopwatch: TStopwatch; begin Stopwatch.Start; sleep(3000); Stopwatch.Stop; m_Result.Lines.Add(format('Took %.3f seconds', [Stopwatch.Elapsed.TotalSeconds])); end; The result is "-130847749795,796 seconds", I was, of course, expecting something around 3. What am I missing?

One of the bad habits I have picked up over the years is not looking into the documentation. The only reason I can give is that the documentation was so bad for some time that the time it took usually was wasted. Since documentation has become much better (but is still far from perfect, especially some of the examples), this no longer applies though.

What i don't need are snide remarks like this.

Conclusion: It doesn't matter whether I use Breadth First or Depth First Search, the performance is the same. Switching from SysUtils.FindFirst/FindNext to Windows.FindFirstFileEx/FindNextFile with only basic file information and the flag FIND_FIRST_EX_LARGE_FETCH brought me a performance gain of about 25% which is not too bad. I found no way to benefit from using TDirectory. The only test I made got worse performance than FindFirst/Next. Test environment: Windows 8.1 client accessing a share on a Samba server over 1 GBit Ethernet with minimal traffic (it's a weekend). The computer is a reasonably fast Intel Xeon E3 with 3.4 GHz with 8 cores and 16 GByte RAM. The Server has an Intel Core I9-9940X processor with 16 cores and with 64 GBytes of RAM. The share is stored on a 2 TB Intel M2 NVMe SSD. It's running Ubuntu 18.04.4 LTS and Samba 4.7.6-Ubuntu. The test program was compiled with Delphi 10.4. There was a no noticable difference between building with Debug and Release config. The directory tree is 4 levels deep and contains only a few directories on the first 3 levels but a total of 898 directories on the deepest level. Each of these directories on the deepest level contain from several hundred up to several thousand jpeg files. The purpose of the code is to find all directories on any level containing at least one jpeg file but no subdirectories. All tests were run several times in varying order to prevent any caching effects to distort the result.

As promised, here the BFS version using FindFirstFileEx. And as expected, it also takes about 45 seconds in my test setup. function FindFirstFileEx(lpFileName: LPWSTR; fInfoLevelId: TFindexInfoLevels; lpFindFileData: Pointer; fSearchOp: TFindexSearchOps; lpSearchFilter: PWin32FindData; dwAdditionalFlags: DWORD): THandle; stdcall; external kernelbase Name 'FindFirstFileExW'; function FindNextFile(hFindFile: THandle; lpFindFileData: PWin32FindData): BOOL; stdcall; external kernelbase Name 'FindNextFileW'; procedure TForm1.b_FindFirstExNextBFSClick(Sender: TObject); var DirsWithJpg: TArray<string>; DirsToSearch: TStringList; procedure CheckDirectory(const _Dir: string); const FIND_FIRST_EX_LARGE_FETCH = 2; var DirBs: string; SearchHandle: THandle; sr: TWin32FindData; ContainsFiles: boolean; ContainsSubdirs: boolean; fn: string; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); SearchHandle := FindFirstFileEx(PChar(DirBs + '*.*'), FindExInfoBasic, @sr, FindExSearchNameMatch, nil, FIND_FIRST_EX_LARGE_FETCH); if SearchHandle <> INVALID_HANDLE_VALUE then begin try repeat fn := sr.cFileName; if (fn = '.') or (fn = '..') then begin // ignore end else if (sr.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) <> 0 then begin // directory ContainsSubdirs := true; // add to list for later processing DirsToSearch.Add(DirBs + fn); end else if sr.dwFileAttributes and (FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM or FILE_ATTRIBUTE_REPARSE_POINT) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(fn), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until not FindNextFile(SearchHandle, @sr); finally Winapi.Windows.FindClose(SearchHandle); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; DirsToSearch := TStringList.Create; try DirsToSearch.Add('\\server\share\dir'); while DirsToSearch.count > 0 do begin CheckDirectory(DirsToSearch[0]); DirsToSearch.Delete(0); end; finally FreeAndNil(DirsToSearch); end; Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirstExNext BFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end;

OK, so here is the DFS implementation using FindFirstFileEx and only getting basic info. In my tests on the same computer and with the same Samba server and share it only takes about 45 seconds (rather than the 60 seconds with SysUtils.FindFirst). The result is the same. For completeness I am going to implement BFS for this too, but I don't expect any change really. function FindFirstFileEx(lpFileName: LPWSTR; fInfoLevelId: TFindexInfoLevels; lpFindFileData: Pointer; fSearchOp: TFindexSearchOps; lpSearchFilter: PWin32FindData; dwAdditionalFlags: DWORD): THandle; stdcall; external kernelbase Name 'FindFirstFileExW'; function FindNextFile(hFindFile: THandle; lpFindFileData: PWin32FindData): BOOL; stdcall; external kernelbase Name 'FindNextFileW'; procedure TForm1.b_FindFirstExNextDFSClick(Sender: TObject); var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); const FIND_FIRST_EX_LARGE_FETCH = 2; var DirBs: string; SearchHandle: THandle; sr: TWin32FindData; ContainsFiles: boolean; ContainsSubdirs: boolean; fn: string; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); SearchHandle := FindFirstFileEx(PChar(DirBs + '*.*'), FindExInfoBasic, @sr, FindExSearchNameMatch, nil, FIND_FIRST_EX_LARGE_FETCH); if SearchHandle <> INVALID_HANDLE_VALUE then begin try repeat fn := sr.cFileName; if (fn = '.') or (fn = '..') then begin // ignore end else if (sr.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) <> 0 then begin // directory ContainsSubdirs := true; // recursive Depth First Search CheckDirectory(DirBs + fn); end else if sr.dwFileAttributes and (FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM or FILE_ATTRIBUTE_REPARSE_POINT) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(fn), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until not FindNextFile(SearchHandle, @sr); finally Winapi.Windows.FindClose(SearchHandle); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir); Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirstExNext DFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; I redeclared FindFirstFileEx and FindNextFile for this because the declarations in WinApi.Windows are inconsistent in the declaration of the lpfFindFileData parameter. Now both declare it as PWin32FindData.

My tests show no performance difference at all between DFS und BFS using FindFirst/FindNext. Both take nearly exactly 60 seconds on my test machine traversing the same directory tree on a Samba server. Trying to use TDirectory did not result in any improvement. It takes about twice as long. DFS code: var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); var DirBs: string; sr: TSearchRec; ContainsFiles: boolean; ContainsSubdirs: boolean; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); if FindFirst(DirBs + '*.*', faAnyFile, sr) = 0 then begin try repeat if (sr.Name = '.') or (sr.Name = '..') then begin // ignore end else if (sr.Attr and faDirectory) <> 0 then begin // directory ContainsSubdirs := true; // recursive Depth First Search CheckDirectory(DirBs + sr.Name); end else if sr.Attr and (faHidden or faSysFile or faSymLink) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(sr.Name), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until FindNext(sr) <> 0; finally FindClose(sr); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir'); Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirst/Next DBF: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; BFS code: var DirsWithJpg: TArray<string>; DirsToSearch: TStringList; procedure CheckDirectory(const _Dir: string); var DirBs: string; sr: TSearchRec; ContainsFiles: boolean; ContainsSubdirs: boolean; begin ContainsFiles := False; ContainsSubdirs := False; DirBs := IncludeTrailingPathDelimiter(_Dir); if FindFirst(DirBs + '*.*', faAnyFile, sr) = 0 then begin try repeat if (sr.Name = '.') or (sr.Name = '..') then begin // ignore end else if (sr.Attr and faDirectory) <> 0 then begin // directory ContainsSubdirs := true; // add to list for later processing DirsToSearch.Add(DirBs + sr.Name); end else if sr.Attr and (faHidden or faSysFile or faSymLink) = 0 then begin // regular file if not ContainsFiles then begin if SameText(ExtractFileExt(sr.Name), '.jpg') then begin ContainsFiles := true; end; end; end else begin // ignore special files end; until FindNext(sr) <> 0; finally FindClose(sr); end; end; if ContainsFiles then begin if ContainsSubdirs then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else begin DirsWithJpg := DirsWithJpg + [_Dir]; end; end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; DirsToSearch := TStringList.Create; try DirsToSearch.Add('\\server\share\dir'); while DirsToSearch.count > 0 do begin CheckDirectory(DirsToSearch[0]); DirsToSearch.Delete(0); end; finally FreeAndNil(DirsToSearch); end; Stopwatch.Stop; m_Result.Lines.Add(Format('FindFirst/Next BFS: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; The naive approach using TDirectory takes about twice as long because of the calls to .GetFiles and .GetDirectories each call FindFirst/FindNext per directory. (It also finds 2 less directories containing jpg files, so there is probably a bug somewhere, but I didn't investigate this any further.) var DirsWithJpg: TArray<string>; procedure CheckDirectory(const _Dir: string); var Files: TArray<string>; Dirs: TArray<string>; i: Integer; begin Files := TDirectory.GetFiles(_Dir, '*.jpg'); Dirs := TDirectory.GetDirectories(_Dir); if Length(Files) > 0 then begin if Length(Dirs) > 0 then begin m_Result.Lines.Add(Format('Directory %s contains files and subdirectories, ignoring it.', [_Dir])); end else DirsWithJpg := DirsWithJpg + [_Dir]; end else begin for i := Low(Dirs) to High(Dirs) do CheckDirectory(Dirs[i]); end; end; var Stopwatch: TStopwatch; begin Stopwatch.Reset; Stopwatch.Start; CheckDirectory('\\server\share\dir'); Stopwatch.Stop; m_Result.Lines.Add(Format('DirGetFiles: Found %d dirs in %.3f seconds', [Length(DirsWithJpg), Stopwatch.Elapsed.TotalSeconds])); end; I thought about using TDirectory.FileSystemEntries instead but could not think of a simple way to implement this. It's probably possible using a Predicate but that's not really any simpler than directly using FindFirst/FindNext. There is probably still some space for improvements and the code is not really clean, e.g. accessing variables of the outer procedure. So, now I'll have a look at calling Windows.FindFirstFileEx instead of FindFirst, but I don't have much hope that this will help much.

Yes, I'm sure it could.

Ouch! Forget it: I didn't reset the stopwatch first: var Stopwatch: TStopwatch; begin Stopwatch.Reset; //<<<--- this was missing Stopwatch.Start; sleep(3000); Stopwatch.Stop; m_Result.Lines.Add(format('Took %.3f seconds', [Stopwatch.Elapsed.TotalSeconds])); end;

On the other hand, if I only want to create one installer, it's easier to temporarily edit the makeinstallers.cmd, where I can simply copy one existing line rather than having to remember exactly which parameters I'd have to pass: rem [...] rem *** copy one line from the list below here *** rem *** and remove two REMs *** rem call :makeinst RS103 RS10.3 rem pause rem goto :eof :list call :makeinst Delphi6 D6 call :makeinst Delphi7 D7 call :makeinst Delphi2005 call :makeinst BDS2006 call :makeinst Delphi2007 call :makeinst RS2009 call :makeinst RS2010 call :makeinst RSXE1 call :makeinst RSXE2 call :makeinst RSXE3 call :makeinst RSXE4 call :makeinst RSXE5 call :makeinst RSXE6 call :makeinst RSXE7 call :makeinst RSXE8 call :makeinst RS100 RS10 call :makeinst RS101 RS10.1 call :makeinst RS102 RS10.2 call :makeinst RS103 RS10.3 call :makeinst RS104 RS10.4 rem [...]