Remy Lebeau

Not if you pump the message queue whenever MsgWaitForMultipleObjects() tells you that messages are waiting.

Something that has been annoying me for awhile. When posting a code snippet and choosing the Pascal syntax highlighter, backslashes are treated as C/C++ escape sequences, which throws off the coloring. The actual Pascal language doesn't treat backslash as an escape character. Can this be fixed?

Can you be more specific? What is the ACTUAL problem? That is a LOT of code for a seemingly simple problem. Can you narrow it down to a much simpler test case?

Good point. I have removed that suggestion.

I've already addressed this on your StackOverflow question on this same topic: https://stackoverflow.com/questions/70977125/ There are several issues in the original code that you would need to fix, regardless of whether you use the code in Delphi or C++Builder. The code is not using module handles and Unicode buffers correctly, which clearly indicates that the code is old, predating Delphi's support for 64bit and Unicode environments. The code needs to be updated before you can then translate it correctly. That is because the code you are struggling with is translated incorrectly. Why are you translating this code AT ALL? You never answered that when I posted it on your StackOverflow question. You can use Delphi .pas files as-is in C++ Builder projects. The IDE will generate a .hpp file that you can then #include into your C++ code.

What exactly is not working? Please be more specific. Actually, I can't. I don't have to working IDE installed at the moment. Everything I wrote earlier was from memory only.

That is because you are not writing to the Memo while the reading loop is running. You are writing to the Memo only after the loop is finished. Change the code to write the current Buffer to the Memo after each successful read. And don't use the Memo.Text property to do that update, either. That will be very inefficient. A better way to append text to the end of a Memo is to use its SelText property instead, eg: procedure GetDosOutput(Output: TMemo; CommandLine: string; Work: string); var ... begin ... repeat WasOK := ReadFile(StdOutPipeRead, Buffer, 255, BytesRead, nil); if WasOK and (BytesRead > 0) then begin Buffer[BytesRead] := #0; Output.SelStart := Output.GetTextLen; Output.SelLength := 0; Output.SelText := Buffer; end; until (not WasOK) or (BytesRead = 0); ... end; procedure TForm7.Button1Click(Sender: TObject); begin GetDosOutput(Memo1, 'python mtk payload', ExtractFilePath(application.ExeName) + 'bin\'); end; If you don't want to pass in the TMemo directly, you could pass in a TStream instead, and then write a custom TStream descendant that overwrites the virtual Write() method to append to the Memo, eg: procedure GetDosOutput(Output: TStream; CommandLine: string; Work: string); var ... begin ... repeat WasOK := ReadFile(StdOutPipeRead, Buffer, 255, BytesRead, nil); if WasOK and (BytesRead > 0) then Output.WriteBuffer(Buffer, BytesRead); until (not WasOK) or (BytesRead = 0); ... end; type TMemoAppendStream = class(TStream) private FMemo: TMemo; public constructor Create(AMemo: TMemo); function Write(const Buffer; Count: Longint): Longint; override; end; constructor TMemoAppendStream.Create(AMemo: TMemo); begin inherited Create; FMemo := AMemo; end; function TMemoAppendStream.Write(const Buffer; Count: Longint): Longint; var BufferStr: AnsiString; begin Result := Count; SetString(BufferStr, PAnsiChar(@Buffer), Count); FMemo.SelStart := FMemo.GetTextLen; FMemo.SelLength := 0; FMemo.SelText := BufferStr; end; procedure TForm7.Button1Click(Sender: TObject); var Strm: TMemoAppendStream; begin Strm := TMemoAppendStream.Create(Memo1); try GetDosOutput(Strm, 'python mtk payload', ExtractFilePath(application.ExeName) + 'bin\'); finally Strm.Free; end; end;

Not really, no. Windows uses OLE for cross-process drag&drop. For backwards compatibility, if the receiving process does not implement IDropTarget, OLE synthesizes the legacy WM_DROPFILES window message for it. Yes, in-process drag&drop is the responsibility of the process to implement. There is no Win32 API support for it outside of OLE, so the process can basically use whatever it wants.

None of those leaks are related to IndyFormat(). However, you are clearly leaking a TIdHTTP object, which in turn leaks all of its internal objects, as well as the global GIdStack object in the IdStack.pas unit, because GIdStack is reference-counted and all Indy socket components increment its reference count when they are created and decrement it when they are destroyed. Fix the TIdHTTP leak, and the rest of the leaks in that screenshot will disappear. There are no known issues with memory leaks, no.

In Tokyo, IndyFormat() simply calls SysUtils.Format(), so any memory issue will have to be in the RTL itself.

I used to have a whole bunch of separate VMs for each IDE release going back many years, then I lost them all in a total system crash, and I didn't have backups of them at the time. But I did have backup copies of just the \Source and \Include folders for 5-XE3 (minus 7-2005), so at least I can refer to them when needed, I just can't compile for them anymore. Nice! Still, if you could get history working properly, that would make for a cool way to search for revisions between releases.

Neither. You keep saying "I get this" and show a call stack for a memory allocation. Why do you keep mentioning that, unless it is causing a problem for you? WHERE and WHEN are you getting that stack trace shown to you? I don't see any memory leak in this code, either. On the other hand, the stack trace you have shown is for a memory allocation made by the 1st call to IndyFormat() inside of TIdFormDataField.FormatHeader() while the HTTP post data is being prepared by TIdHTTP.Post(). But, FormatHeader() appends additional substrings (6, in your case) to the String returned by that 1st IndyFormat() call, so it doesn't make sense why you should be seeing a stack trace for only the 1st memory allocation, and not for other memory allocations (unless you just didn't show them?)

Fair enough. So, then I suggest Process Explorer to check open handles to the file at the time the problem occurs. Maybe something else is opening the file between the time you create it and the time you use it. Antivirus, perhaps? The alternative is to simply not use a file at all. As I demonstrated earlier, it is possible to give TIdMultipartFormStreamStream a TStream for posting. For instance, you could write your data to a TMemoryStream and then Post() that instead of a disk file. There is no point in performing the call to TIdHTTP.Post() in a loop that handles EFOpenError, because that exception will never happen. The file is not opened inside that loop, it is opened by the call to TIdMultipartFormDataStream.AddFile(). So, if the file can't be accessed by TIdMultipartFormDataStream, you are not catching that error. WHERE do you get that, exactly? That is a call stack for a memory allocation, so again I ask, are you trying to report a memory leak? That is a completely different issue than a file access error. Go up to the "Find" menu, choose "Find Handle or DLL...", and enter the name of your data file. You will get back a list of every open handle to the file, if any, including the name of the process that each handle belongs to. Correct.

Only if the app is written in Delphi or C++Builder to begin with...

First, you need to figure out where the file is actually open. Use a tool like SysInternals Process Explorer to see who has open handles to the file. If it turns out to be your app, then check your code to make sure you are closing all of your open handles to the file, and that if you need to open multiple handles then don't open them with conflicting permissions. You do know it is possible to get a file's size without actually opening a handle to the file, don't you? You can get the file's size from the filesystem's metadata for the file, such as with SysUtils.FindFirst(), rather than querying the size from the file itself. You shouldn't be using the file size as an indicator that the file is ready for use, though...

Then why mention it? Yes, it is. Yes. Though, behind the scenes, the OS may take some time to actually flush the data to the physical disk in the background, but its data cache will handle any requests for file data that hasn't been flushed yet. So, in that regard, if you KNOW the saving process has finished writing its data to the file before you call SaveTheSolvedSlice(), then you don't need the loop to wait to access the file. Just open the file and it will either success or fail, act accordingly: function TMain.SaveTheSolvedSlice(FileName: String): Boolean; var MimeType, Response, FullFilePath: String; jsonVal: TJSonValue; Stat, Size: String; SendParams: TIdMultiPartFormDataStream; Param: TIdFormDataField; //HTTPSendFile: TIdHTTP; begin Result := False; FullFilePath := 'tmp\out\'+FileName; if (not FileExists(FullFilePath)) then Exit; Response := 'NULL'; HTTPSendFile.Request.Clear; HTTPSendFile.ReadTimeout := 2000; HTTPSendFile.ConnectTimeout := 9000; SendParams := TIdMultiPartFormDataStream.Create; try MimeType := 'application/octet-stream'; //Params.AddFormField('_token', 'Myfb9OqYgDBwDws3zTL9QOs492XWfNtGLftUdNsH'); Param := SendParams.AddFile('slice', FullFilePath, MimeType); Param.ContentTransfer := 'binary'; StatusUpdate(Console, 'Attempting to post file ' + FullFilePath + ' (size: ' + Param.FieldStream.Size.ToString + ' bytes)'); Delay(10); Response := HTTPSendFile.Post(STORAGE_OUT_REPO, SendParams); Delay(5); if (Response <> 'NULL') then begin //{"Slice":{"Status":"ACCEPTED","FileName":"1_Some_Job_Filename.slc","Size":1812}} jsonVal := TJSonObject.ParseJSONValue(Response); try Stat := jsonVal.GetValue<String>('Slice.Status'); Size := jsonVal.GetValue<String>('Slice.Size'); finally jsonVal.Free; end; StatusUpdate(Console, 'Toasted Slice (size: ' + Size + ') ' + Stat.ToLower); Result := (Stat = 'ACCEPTED'); end; finally SendParams.Free; end; end; That being said, your ExportSliceToBinFile() is leaking MySliceFile, so it is leaving the file open. MyOPLToolPathFile.Free() should be MySliceFile.Free() instead. And it should be protected with a try..finally: function ExportSliceToBinFile(MyFileName: String): Boolean; var I: Integer; MySliceFile: TFileStream; begin try ForceDirectories('tmp\out'); MySliceFile := TFileStream.Create('tmp\out\'+MyFileName, fmCreate or fmOpenWrite); try for I := 1 to GlobalIndex do begin MySliceFile.WriteBuffer(GlobalSliceRecord[I], SizeOf(SLICE_RECORD)); end; finally MySliceFile.Free; end; Result := True; except Result := False; end; end;

Indicating what exactly? A memory leak? I would not do it that way. TIdMultipartFormDataStream.AddFile() opens the file for read-only access and denies further writing to the file. If you can't wait for the file to be fully written to disk before calling SaveTheSolvedSlice(), then I would suggest using a TFileStream in a loop to open the file for exclusive access until successful, and then pass the opened TFileStream to TIdMultipartFormDataStream.AddFormField(TStream), eg: function TMain.SaveTheSolvedSlice(FileName: String): Boolean; var MimeType, Response, FullFilePath: String; jsonVal: TJSonValue; Stat, Size: String; SendParams: TIdMultiPartFormDataStream; TOPLSliceFile: TStream; //HTTPSendFile: TIdHTTP; begin Result := False; FullFilePath := 'tmp\out\'+FileName; if (not FileExists(FullFilePath)) then Exit; Response := 'NULL'; HTTPSendFile.Request.Clear; HTTPSendFile.ReadTimeout := 2000; HTTPSendFile.ConnectTimeout := 9000; SendParams := TIdMultiPartFormDataStream.Create; try MimeType := 'application/octet-stream'; repeat try TOPLSliceFile := TFileStream.Create(FullFilePath, fmOpenRead or fmExclusive); except on E: EFOpenError do begin // I know, this is not the best option, but it is the only way to detect // a sharing violation error with TFileStream. The alternative is to call // CreateFile/FileOpen() and GetLastError() directly, and then create a // THandleStream from the result... if E.Message <> 'The process cannot access the file because it is being used by another process.' then raise; Delay(100); end; end; until False; try //Params.AddFormField('_token', 'Myfb9OqYgDBwDws3zTL9QOs492XWfNtGLftUdNsH'); SendParams.AddFormField('slice', MimeType, '', TOPLSliceFile, FileName).ContentTransfer := 'binary'; Delay(10); StatusUpdate(Console, 'Attempting to post file ' + FullFilePath+' (size: ' + TOPLSliceFile.Size.ToString + ' bytes)'); Delay(10); Response := HTTPSendFile.Post(STORAGE_OUT_REPO, SendParams); Delay(5); if (Response <> 'NULL') then begin //{"Slice":{"Status":"ACCEPTED","FileName":"1_Some_Job_Filename.slc","Size":1812}} jsonVal := TJSonObject.ParseJSONValue(Response); try Stat := jsonVal.GetValue<String>('Slice.Status'); Size := jsonVal.GetValue<String>('Slice.Size'); finally jsonVal.Free; end; StatusUpdate(Console, 'Toasted Slice (size: ' + Size + ') ' + Stat.ToLower); Result := (Stat = 'ACCEPTED'); end; finally TOPLSliceFile.Free; end; finally SendParams.Free; end; end;

Yeah, well, now the check will be even faster, since you don't need the string comparison anymore. isPutOrPatch := (ARequestInfo.CommandType in [THTTPCommandType.hcPUT, THTTPCommandType.hcPATCH]);

No, there is not. I create my own table manually, using a combination of Microsoft's Message Compiler and XN Resource Editor (to tweak the output from mc.exe) to produce a .res file that I then link into my project

Only because you were explicitly looking for CommandType=hcUnknown, which was not necessary to begin with. You could have simply done this instead: isPutOrPatch := (ARequestInfo.CommandType = THTTPCommandType.hcPUT) or TextIsSame(ARequestInfo.Command, 'PATCH') ; And that would have still worked fine with the CommandType change. Indy's release numbers were broken when Indy migrated from SVN to GitHub (issues #292 and #328), and that problem has not been addressed yet, but I do intend to ... some day, when I have time for it. However, the code is now live in Indy's master branch, and Lazarus' OPM has been updated to use this version (they have labeled it as version 10.6.2.4072). A future version of Delphi will pick up the change whenever Embarcadero decides to update their copy of Indy (usually in major IDE version releases).

I have been using TService for many years, and I have never experienced what you describe. Whenever I use LogMessage() (which, I admit is very rare, as I prefer my own logging), the passed string message is displayed as-is in the event log (prefixed with "The description for Event ID 0 ... cannot be found" if I don't include a proper message table resource that maps ID 0 to '%s'). No. The behavior of the ReportEvent() API has not changed over the years.

It is difficult to explain what needs to be changed exactly, when you have not shown the actual code that is erroring. However... Make sure your handler/override is using the general-purpose Char type, and not using AnsiChar directly. Prior to Delphi 2009, Char was an alias for AnsiChar, and as such it only supported 255 values max, which is also the max number of values that that a Set supports. Since Delphi 2009, Char is now an alias for WideChar, so it is too large to use in a Set. If you try, the Char value has to be truncated from 16 bits to 8 bits, which is what the warning is about. The CharInSet() function was introduced to hide that warning, but not the truncation. Prior to Delphi 2009, string was an alias for AnsiString. Now, it is an alias for UnicodeString. ShortString is a fixed-length ANSI string, always has been, and still is. Assigning a ShortString to an AnsiString is a loss-less operation, but assigning a ShortString to a UnicodeString can potentially cause data loss. So, the compiler now requires an explicit typecast whenever you assign an ANSI string to a Unicode string, or vice versa, indicating that you understand the risk. You really shouldn't be using ShortString for anything other than interacting with external legacy systems. This means your code is declaring function parameters that do not match what the compiler is expecting. This is most commonly seen in event handlers that are using the wrong parameter types.

UnicodeString, not WideString. They are two different string types.