Arnaud Bouchez

In fact, it is not allocation-free, it is heap-allocation free, since it needs to allocate its memory from the stack. And note that the stack size can be pretty limited... For instance, if the process is run within IIS, stack size is only 256KB (on 32-bit), which could be too small for using allocation-free collections everywhere. Under Linux, the default thread stack size can be pretty limited in comparison to Windows, too. What we do in practice in our servers code when we need such temporary storage, is to use a 4KB stack-allocated buffer, with optional heap allocation. See https://synopse.info/files/html/api-1.18/SynCommons.html#TSYNTEMPBUFFER There are some methods to pre-initialize the buffer with common patterns (zero bytes, increasing integers, random numbers).... But you need to use pointer arithmetic, whereas this article uses generics so may a little safer. 4KB is OS memory page size, so should already be there, and avoid to reach 256KB numbers.

Use a Delphi version < 2010 (where the enhanced RTTI has been introduced) or FPC/Lazarus, and to something even smaller, https://github.com/rofl0r/KOL or our https://github.com/synopse/LVCL

As for an embedded database, I would rather use SQlite3 or even FireBird, which has an embedded version. They are OpenSource, cross-platform, well maintained and have high feature set.

On server side, we log a lot of stuff... almost everything possible information. It is needed for post-mortem or post-bug forensic. Of course, our mORMot framework logs automatically all SOA calls, and all ORM and/or SQL process. And it also logs all exceptions. See https://synopse.info/files/html/Synopse mORMot Framework SAD 1.18.html#TITL_73 This was for low-level text logs. At higher level, we also log all high-level SOA calls, in form of local Sqlite3 databases. This is very efficient to make any kind of statistics, via some simple SQL SELECT. https://synopse.info/files/html/Synopse mORMot Framework SAD 1.18.html#TITLE_445

Here is what we usually do: Store logs in a dedicated folder, with proper read policy. -> especially under Unix, all background daemons run with a dedicated user, and the log files have chmod 640 to restrain read access. Always run the logs in Verbose mode, even on production, unless the execution environment is really constrained. (you never know what will happen, and detailed logs is a need for proper investigation, especially on production) -> you need a logging solution which is fast, and can log dozen MB/s with no performance penalty (e.g. our SynLog.pas unit writes in a background thread) Rotate logs to keep history as long as possible -> SynLog.pas has built-in SynLZ-compression of the logs for very fast and efficient rotation Never log passwords, or such sensitive information. -> password are always stored and persisted in salted + hashed form - but when transmitted as parameters, or as settings, they should be encrypted - even a naive encryption may be enough like with a TSynPersistentWithPassword class from SynCommons.pas -> more generally, sensitive information should be handled in memory with strong cryptography, e.g. via our cross-platform CryptDataForCurrentUser() from SynCrypto.pas Have a way to hide unexpected content, when log is automated. -> in mORMot.pas, you can register some sensitive types which parameters will be logged as "***" during SOA automated logging of calls. Ensure all those policies are always setup by default, and on all supported platforms -> today, we should ensure that Linux execution should be not weaker than Windows as a target (and it is easy to make executable safer under Linux) Optionally use a third-party SaaS solution to store your logs in the long term, with proper encryption and search abilities -> like https://livemon.com - server-side written in object pascal 🙂 Most of it is available out-of-the-box in http://mormot.net

Check https://www.delphitools.info/2013/10/30/efficient-string-building-in-delphi/ TStringBuilder has been enhanced in latest Delphi 10.3 IIRC. Our OpenSource `TTextWriter` class, which works on UTF-8 (which is very likely your eventual CSV file encoding), is very fast (it avoids most memory allocation), and has a `TTextWriter.CancelLastComma` method, as you expect. See https://synopse.info/files/html/api-1.18/SynCommons.html#TTEXTWRITER_CANCELLASTCOMMA

This is wrong if the refcount is 1: in that case (which happens when you just appended some texts to a string variable), SetLength() won't copy the string, just call ReallocMem() which is a no-op with FastMM4 over a minus one byte buffer, and put a #0 at the end. See how _UStrSetLength() is implemented in System.pas.

or even try procedure DeleteLastChar(var aText: string; aLastChar: char); inline; var len: {$ifdef FPC}PSizeInt{$else}PInteger{$endif}; begin len := pointer(aText); if len <> nil then begin dec(len); if PChar(pointer(aText))[len^ - 1] = aLastChar then begin PChar(pointer(aText))[len^ - 1] := #0; dec(len^); end; end; end; which has also the advantage of working with both Delphi and FPC. But all these sounds a bit like premature optimization to me... 😞

If you are SURE that the string has a reference count of 1, e.g. if it just has been allocated and not assigned/copied anywhere else, you can simply try: procedure DeleteLastChar(var aText: string; aLastChar: char); inline; begin if (aText <> '') and (aText[length(aText)] = aLastChar) then dec(PInteger(PAnsiChar(pointer(aText)) - SizeOf(integer))^); end; This will avoid a call the heap manager. Then do proper benchmark.

Square of distances is indeed the way I would have done it: faster (no branch, no call), and no complex math.

FastMM4, in FullDebugMode is able to track such pointer problems. I always use it, and also the FPC's Heaptrc unit (compile with -gl) which can find some other problems. I use pointers only when the code needs it, mainly for performance reasons. Otherwise, high-level structures like classes or dynamic arrays are just enough.

Why not directly make it `using` and replace the `begin` `end` with `{ }` ? :)

Where this information came from? If it is the case, I didn't see any performance enhancement at execution, at least from our mORMot regression tests, which run a lot of performance measurements. In fact, most loop unrolling could even be slower on modern CPUs - for instance, AES unrolled loop which was faster on P4 is slower than a tuned rolled loop. I doubt EMB did make proper compiler optimization in loop unrolling, and let the inlining performance problems remain.

I don't see any reason why it shouldn't.

FindFirst is definitively the slowest. It needs at least two calls (+ FindClose), and maintain some state in-between. GetFileAtributes() is the fastest under Windows, and fpaccess/euidaccess on POSIX systems. Note that a fallback to FileAge() is needed in case of sharing violation: function FileExists(const FileName: string): Boolean; {$IFDEF MSWINDOWS} // use GetFileAttributes: much faster than original, which uses FileAge=FindFirst var Attr: Integer; LastError: Cardinal; begin Attr := Integer(GetFileAttributesW(pointer(FileName))); if Attr <> -1 then Result := Attr and FILE_ATTRIBUTE_DIRECTORY = 0 else begin LastError := GetLastError; Result := (LastError <> ERROR_FILE_NOT_FOUND) and (LastError <> ERROR_PATH_NOT_FOUND) and (LastError <> ERROR_INVALID_NAME) and // (use FileAge to test SHARE_EXCLUSIVE files) ((LastError = ERROR_SHARING_VIOLATION) or (FileAge(FileName)<>-1)); end; end; {$ENDIF} {$IFDEF LINUX} begin Result := euidaccess(PChar(FileName), F_OK) = 0; end; {$ENDIF} (extracted from our Enhanced RTL source code) But the version currently included in Delphi 10.3 Rio SysUtils.pas is just as fast on Windows. On POSIX, it uses stats, which is slower than euidaccess().