I did not only use Task Manager, I also used the built-in memory profiler in Visual Studio.
I know that Windows keeps around some memory. I did the same test with a DLL written in C/C++ and the memory usage grows up to around ~300KB then further LoadLibrary/FreeLibrary calls seem to reuse previous memory and the overall memory usage does not continue to grow. But that is not the case for the Delphi DLL. After running it for half an hour or so, the memory usage was already at a few houndred MB and it showed no sign of stopping.
I tried using FastMM (ReportMemoryLeaksOnShutdown), but it did not show any leaks. Visual Studio memory profiler shows the allocations for each call but it only hints at "external code" and some memory address. According to the memory snapshots there are around ~5KB of memory allocations for each LoadLibrary call, that are not freed with FreeLibrary.
All I can say for sure is that if I remove "uses System.SysUtils" the problem is gone, which is obviously not a fix, since I want to use some of the functionality.