Dmitry Onoshko

I wasn't suggesting you not destroy your objects. But a destructor is not always the best place to perform complex cleanups, especially during app shutdown. For instance, you could use the Form's OnClose/Query event to detect the user's desired to exit the app, signal your code to start cleaning up (close sockets, cancel I/Os, etc), and disable the UI so the user can't generate more traffic in the meantime. When the cleanup is actually finished (all I/O completions are reported, etc), then exit the app. Ah, sorry, I must have misunderstood your idea. What I tried to address in my previous post is an idea of asking a socket to shutdown and letting it destroying itself automatically (without explicit destructor call). Back to your point, I feel requiring a particular method call before destroying the object like asking for trouble some time later. If one suddenly decides to call a destructor, it should clean all the stuff without any conditions. As for calling a destructor only at a particular moment in time, like before application termination, changing network app settings at runtime is a thing, and this implies actually shutting down sockets while app is still running, possibly with other sockets serving other ports/protocols/tasks but using the same IOCP and stuff. Disabling related UI parts while the restart takes place might be OK, though.

Are there any ready-to-use solutions for Delphi to display partially downloaded images? Like web browsers do over slow internet connections. I tried to load a few truncated versions of a PNG file into TImage (and, so, TPicture under the hood) and it fails on CRC check. Since this stuff is highly format-dependent (say, displaying a partially downloaded BMP is quite easy, for PNG it’s quite tricky unless it has multiple IDAT chunks, while for JPEG with its different versions this might be a real pain to implement) implementing such a feature looks like a separate project. So, I wonder if anyone knows libraries where it’s already done. (Of course, it’s not that critical for an application, and in most cases the image being downloaded can be replaced with a progress bar. I’m just curious.)

Well, not destroying an object implicitly feels like a code smell. At least, it requires more attention when reviewing code for possible leaks and stuff. So, yes, stopping an overlapped socket is almost east. Especially since the only way to guarantee pieces of TCP stream are processed in the right order is to have at most one outstanding Tx and Rx operation per socket. But then events come into play. Being notified of client disconnection is useful for bookkeeping, and this implies either synchronizing with the GUI thread (which is waiting for pending requests to finish in the destructor) or making an event that gets invoked from arbitrary thread. The second way might not be as bad, really, but felt too complicated back then. I remember also trying to make the wait for pending operations alertable (so Synchronize works) but that also feels somewhat wrong. Frankly speaking, the ease of event handling provided by TServerSocket and TClientSocket and the simplicity of code, classes and documentation (as compared to Indy, in fact) led to me making the wrong choice. Although, at least I made a minimum viable “product” (sort of) with them. Just to see how bad they are for the task.

Well, I gave it a try at the beginning of the project, with hand-written IOCP-based overlapped sockets. But the problem of properly shutting them down when the user just invokes the destructor from a GUI thread, plus lack of spare time for brave experiments made me switch to something asynchronous at hand till better times. Now I feel the better times will have to come sooner than expected.

Yep, I understand that, that’s why I wrote they should’ve used some value like DWORD(–1) or MAXINT, or MAXLONG, or something like that in the first place, since that’s quite a common, let me call it so, design pattern. If my memory serves, Raymond Chen called similar stuff sentinel values. Well, since for now I’m with TServerSocket and his friends, it’s the one who calls WSAStartup and initializes to v1. If I call it myself to be the first one, it might work but the classes might break, ’cause certain stuff has definitely changed between major WinSock versions. For instance, ScktComp uses Winapi.Winsock which defines SOMAXCONN as 5. Now if I’m the first to initialize WinSocks in the application, I get a backlog of 5 clients instead of whatever the OS could choose. Some stuff has also changed around Vista times in the treatment of SO_REUSEADDR. So, for like 10’000 clients we get 10’000 threads each taking quite a piece of resources, like memory and handles? And they start competing for the poor processor cores (let’s hope 8 or 16 if the end-user doesn’t try to use a PC instead of real server hardware)?

Well, maybe not so easy with sockets, but I remember hitting it when I was a student, writing a program that did some stuff in a separate thread and notified the UI by posting messages. Really easy. Yep, I know. But (1) what if even part of the expected clients connects simultaneouly and (2) why would thevalue change from 5 to $7FFFFFFF from WinSock1 to WinSock2? I understand the second value should have been used since WinSock1, but they probably had reasons to choose 5, and will it be treated as real maximum when WSAStastup initializes to version 1 is a big question for me. The point was to avoid using third-party libraries and to have asynchronous sockets. I’m still worried about the fact it uses blocking sockets, so I’m not sure what effect would that have with tens of thousands of clients. And with the need of DB access when serving the requests. I understand it doesn’t (shouldn’t?) use a thread-per-client approach, but…

I’ve just tried it and it seems I get no exception when trying to Open a TServerSocket on a busy port. In debugger I can see both bind and listen calls somehow succeeded. WinSock < 2.0 stuff, maybe? I’m somewhat worried about the maximum size of the Windows message queue. AFAIR, it was something around tens of thousands of messages by default (at least, in WinXP), and even if it has been increased since then, tens of thousands of clients are expected for the program, so (1) saving a few messages seems to be a good idea, (2) I start feeling the good old library is not that good for the task, (3) especially since SOMAXCONN is 5 there, (4) why on earth there’s no other good socket library included with Delphi and (5) OMG, I’ll probably have to implement IOCP-based sockets myself handling all the quirks of multithreading, but then again DB access is required for most of the messages that get received, so serialization is still a thing, and…

P.S. Turns out the values of AArgs matter: those were (124, 152, 10, 2). I tried (127, 0, 0, 0) and it worked. Then I tried (128, 0, 0, 0) and it failed with error saying about aArg1 this time. When I inspected AArgs in the debugger adding TVarData(AArgs[...]) to watches, I saw the aArgX array items were of type Byte inside the Variant. But somehow it seems FireDAC manages to ignore both the UNSIGNED part on the SQL side and the unsignedness of the variants on the client side.

I’m using FireDAC with MariaDB. Since I have quite a lot of stuff done with stored procedures, I wrote a simple function like this: function TMyDataModule.OpenProc(const AName: String; AArgs: array of Variant): TFDStoredProc; begin Result := TFDStoredProc.Create(nil); try Result.Connection := MyConnectionComponent; Result.Open(AName, AArgs); except Result.Free; raise; end; end; In my database I have a table: CREATE TABLE `Data` ( `ID` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT, `When` TIMESTAMP NOT NULL DEFAULT current_timestamp(), `MyID` BIGINT(20) UNSIGNED NOT NULL, `Bool1` TINYINT(1) NOT NULL, `FOURCC` CHAR(4) NOT NULL COLLATE 'utf8mb4_unicode_ci', `Comment` TINYTEXT NOT NULL COLLATE 'utf8mb4_unicode_ci', `Arg1` TINYINT(3) UNSIGNED NOT NULL, `Arg2` TINYINT(3) UNSIGNED NOT NULL, `Arg3` TINYINT(3) UNSIGNED NOT NULL, `Arg4` TINYINT(3) UNSIGNED NOT NULL, PRIMARY KEY (`ID`) USING BTREE ) COLLATE='utf8mb4_unicode_ci' ENGINE=InnoDB ; … and a stored procedure: CREATE DEFINER=`root`@`localhost` PROCEDURE `MyProc`( IN `aMyID` BIGINT, IN `aBool1` TINYINT(1), IN `aFOURCC` CHAR(4), IN `aComment` TINYTEXT, IN `aArg1` TINYINT UNSIGNED, IN `aArg2` TINYINT UNSIGNED, IN `aArg3` TINYINT UNSIGNED, IN `aArg4` TINYINT UNSIGNED ) LANGUAGE SQL NOT DETERMINISTIC CONTAINS SQL SQL SECURITY DEFINER COMMENT '' BEGIN INSERT INTO `Data`(`MyID`, `Bool1`, `FOURCC`, `Comment`, `Arg1`, `Arg2`, `Arg3`, `Arg4`) VALUES(aMyID, aBool1, aFOURCC, aComment, aArg1, aArg2, aArg3, aArg4); END When I try to invoke MyProc, I get this error: “Out of range value for column 'aArg2' at row 0”. In Delphi debugger I see AArgs containing perfectly valid values: a number, a boolean, a 4-character string, an empty string for aComment and four integers perfectly in the range of 0..255. But at OpenProc call it fails. I used Wireshark to see what data actually goes “onto the wire”: the byte values are those they should be. The same procedure with the same argument values runs just fine with HeidiSQL. But HeidiSQL seems to use simple text queries while FireDAC sends arguments in their binary representation. What could be the reason and fix to the problem? Thanks.

This is long story !, in short the answer is yes, loop is way better and will yield better throughput. I don’t really understand this part. My point was that to receive data the loop is not the way to go (well, not the only one) since that would generally lead to several posted messages, and when processing them — to a few useless recv calls if no data has arrived since then. But you seem to suggest using the loop anyway. This is already done. After all, at least for TCP, knowing the amount of data available in advance has no benefits for the code: the “protocol message” might arrive in pieces and/or concatenated with the rest of the previous and the beginning of the next one, so there still has to be some storage to be able to find the message boundaries. Unless, maybe, if the protocol used is simple enough to make the data stream being parseable with a simple finite automaton byte-by-byte.

Thanks, made a few corrections to take that into account. I also saw somewhere that receive notification message would be posted to the message queue whenever ReceiveBuf leaves some data unread, so no point making a loop for ReceiveBuf, right?

Well, in fact I do set the ErrorCode in the handler, ’cause the requirement is to provide custom user-friendly socket error handling, and it seems to be much easier to have a single point of failure instead of putting try…excepts every now and then. BTW, am I right that the whole point of checking for WSAEWOULDBLOCK vs some other error code is to avoid adding data to Tx buffer is the socket has already failed? And, if you don’t mind, another small TServerSocket-related question. It seems to only have an OnClientError event but no “OnError”-kind event for the listening socket. Which makes me wonder if there’s any better way to handle, say, listening port being already taken by another program than using SetErrorProc (which feels a bit messy, too).

Thanks, that really makes it much more clear, and since I already have custom wrappers around those TClientSocket and TServerSocket adding a Tx buffer of customizable size to the wrappers looks promising. But one thing is still unclear: SendBuf doesn’t return the WinSock error code and neither of the socket classes seem to have properties to access it. I can only see two possibilities here: (1) the OnError handler that is called from SendBuf if the error is not WSAEWOULDBLOCK and (2) calling WSAGetLastError again myself. The first option leads to complication of the OnError handler which now has to store the error code somewhere just in case anyone asks. The second option feels like jumping through levels: that would require adding WinSock library to uses clause and relying on the fact that the last operation to fail inside SendBuf is the send() I’m interested in. Is there any better option I can’t see?

I’ve recently got really worried about TCP send buffer overflows. I use TServerSocket and TClientSocket in asynchronous mode, and I’m going to send file data interleaved with smaller packets through them. What really scares me and doesn’t fit well in the whole picture is the case when SendBuf might return –1 (or anything else not equal to the size of the buffer being sent). I basically have two questions: 1) What is the proper way to detect a client that just doesn’t read the data from its socket (say. because it got hung)? 2) How should one handle the case when SendBuf says the buffer hasn’t been sent? In general I do understand I should try to SendBuf a little later, even better — in OnClientWrite event handler. To do that I’d have to use some sort of queue and put the data there. But then I think about the client that doesn’t read data: since there are shorter messages, not only file blocks, the queue might start growing and the logic gets really complicated. Any thoughts are appreciated.

type TCustomProtocol<TMessageType> = class type TMessageEvent = procedure(Sender: TCustomProtocol<TMessageType>; const AMessage: TMessageType) of object; private FOnMessage: TMessageEvent; ... end; TCoolProtocol = class(TCustomProtocol<TCoolMessage>) ... end; ... procedure TCoolServer.MessageReceived(Sender: TCustomProtocol<TCoolMessage>; const AMessage: TCoolMessage); var Proto: TCoolProtocol absolute Sender; begin // Handling goes here end; The first parameter here has a long type specification which actually contains two type names. Besides, the procedure header basically repeats itself: one TCoolMessage would be enough to understand what’s going on. Hmmm, I’ve just felt that the wider problem here is that there seems to be no way to define the event in such a way that Sender type is always the descendant that can extract the message. That would also allow the handler to get access to all the methods provided by the specific descendant. Anyway, the generic-based approach works, it just doesn’t seem nice enough.