TIdHTTPsServer, file transfer and network errors

[Mimiko 0]

I need to do some maintenance on an older project, which has been built using Delphi 10.3 32bit (some components just exist for this version). The Project is using TIdHTTPsServer to communicate and to transfer (upload) files. In CreatePostStream originally a TMemoryStream has been created, but it can't transfer more than approx. 500-600 MB. So I started to use a (modified) TFileStream instead (modified=extended to have LoadFromStream/SaveToStream and the like).
The filestream creates a unique temp file where the data goes. Within ServerCommandOther the data is written to a second filestream with the final filename (the final name isn't yet known on CreatePostStream, so I need to copy later). As long as the data is within a 500-600 MB range, all works well. If the uploaded data is larger, then the client gets a network timeout after a few seconds.

But the session thread is still active and the copy process is still running. Copying between PostStream and TargetStream is done via CopyFrom and so I placed some logging within its Read/Write loop. As soon as the copy process goes beyond the mentioned 500-600 MB, that loop seems to stop its process for several seconds, then it resumes copying, and this repeats randomly. Finally the file is copied with its full size, but the client's already gone.

I just don't understand why THTTPsServer closed the connection while the thread's still working. Is there something I can do to the session or the server to avoid the network timeout at the client? Or is there a problem with TFileStream regarding threads? Indy's reporting 10.6.3.11, so this isn't very old.

I tried setting all possible timeouts (ConnectTimeout, ReadTimeout, WriteTimeout on the session socket), enable/disable nagle, KeepAlive, ReuseSocket... all that, but to no success.

Is there something I didn't see or are there some dont's I didn't realize?

TIA
Michael

 

[Remy Lebeau 1619]

55 minutes ago, Mimiko said:

The filestream creates a unique temp file where the data goes. Within ServerCommandOther the data is written to a second filestream with the final filename (the final name isn't yet known on CreatePostStream, so I need to copy later).

Why not simply rename or move (not copy) the original temp file to the new filename, instead of copying its data to a new file?

55 minutes ago, Mimiko said:

As long as the data is within a 500-600 MB range, all works well. If the uploaded data is larger, then the client gets a network timeout after a few seconds. But the session thread is still active and the copy process is still running.

The only way I can see that happening is if you are not sending a response to the client until after the file data has been copied to the final file, and it's taking a long time to do that copy.  Clients are not going to wait around forever for a response after they send their data to the server, so you should try to send a response as soon as possible, ie when the data has been saved in the temp file, or once processing of that file has begun in the background.  If that is not an option, they you have no control over the client's timeout settings on the server side, so all you can do is optimize your processing to speed it up.  If your processing is going to take a long time, then return a response ASAP and then provide a separate API for the client to poll the status of the processing, ie a separate REST endpoint, or a WebSocket, etc.

55 minutes ago, Mimiko said:

Copying between PostStream and TargetStream is done via CopyFrom and so I placed some logging within its Read/Write loop. As soon as the copy process goes beyond the mentioned 500-600 MB, that loop seems to stop its process for several seconds, then it resumes copying, and this repeats randomly.

That has nothing to do with Indy, but with your OS and/or hard drive performance.  Maybe you are writing to bad sectors, or there are other I/Os in progress that slow down the OS's handling of your file, etc.   No way to diagnose that from your limited description.

 

Also, TStream.CopyFrom() was pretty basic in 10.3. It was re-written sometime after 10.3 with more buffering logic.  Also, consider using TBufferedFileStream instead of a plain TFileStream, at least.  TBufferedFileStream was introduced in Delphi 10.1. That way, more of the copying process is done in memory rather than directly on the hard drive.

55 minutes ago, Mimiko said:

I just don't understand why THTTPsServer closed the connection while the thread's still working.

It didn't.  The client timed out and closed the connection on its end.  If you are doing your file processing directly in your OnCommand handler then the server CAN'T close its end of the connection until you return control back to the server, ie by exiting the handler.

55 minutes ago, Mimiko said:

Is there something I can do to the session or the server to avoid the network timeout at the client?

Yes - don't perform long-running processes directly in your OnCommand handler that makes the client wait a long time for a response.

55 minutes ago, Mimiko said:

Or is there a problem with TFileStream regarding threads?

No.  Especially if different threads are working on different files.

55 minutes ago, Mimiko said:

I tried setting all possible timeouts (ConnectTimeout, ReadTimeout, WriteTimeout on the session socket), enable/disable nagle, KeepAlive, ReuseSocket... all that, but to no success.

There is no timeout you can set on the server side that will affect the client's own timeouts.  However, one thing you might try is TCP keepalives, not HTTP keepalives, ie via the AContext.Connection.Socket.Binding.SetKeepAliveValues() method.  But, that won't prevent the client from just giving up if no response arrives in a long time.

[Mimiko 0]

Remy,

thanks for your detailed reply. For your first question about rename/move instead of copying: The destination file can be elsewhere and is usually on another drive or even on another computer (network share). Unfortunately, the handling is fixed from the client side, so there's no option to implement an API or REST for copying the file. To understand it better, I placed some debug logging into my (non-production) code. So I can trace what's happening if a client's sending a file:

CreatePostStream, CreateSession, SessionStart, CommandOther (once!), DoneWithPostStream, SessionEnd. I see that CommandOther is called once, regardless of the file size, and that's why I placed the copy operation there. Or is it possible to put that into DoneWithPostStream?
However, I'll try to review the CopyFrom (as you suggested) and I'll try to use a TBufferedFileStream for those file operations. Maybe there's a way to improve handling by optimizing components and transfer, so I'm able to transfer larger files (even if not ANY size).

I'll be back with results soon. 

TIA

Michael