dummzeuch 1504 Posted September 23 (edited) The Win32 Delphi 2007 program below sets the PEFlag for IMAGE_FILE_LARGE_ADDRESS_AWARE, so I would expect it to have 3 GB of memory available. But it writes: TotalVirtual 2147352576 Bytes 2,000 GiB AvailVirtual 2093465600 Bytes 1,950 GiB So why does it say there are only 2 GB? The OS is Windows 10 pro 64 Bit with 16 GB of physical memory (from which a lot is available). I could have sworn that this has worked before. Edit: Duh! I forgot an 'or' between 'IMAGE_FILE_NET_RUN_FROM_SWAP' and 'IMAGE_FILE_LARGE_ADDRESS_AWARE' program Project1; {$APPTYPE CONSOLE} uses Windows, SysUtils; {$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP IMAGE_FILE_LARGE_ADDRESS_AWARE} procedure GetProcedureAddress(var P: Pointer; const ModuleName, ProcName: string); var ModuleHandle: HMODULE; begin if not Assigned(P) then begin ModuleHandle := GetModuleHandle(PChar(ModuleName)); if ModuleHandle = 0 then begin ModuleHandle := SafeLoadLibrary(PChar(ModuleName)); if ModuleHandle = 0 then raise Exception.CreateFmt('Library %s not found', [ModuleName]); end; P := GetProcAddress(ModuleHandle, PChar(ProcName)); if not Assigned(P) then raise Exception.CreateFmt('Function %s not found in library %s', [ProcName, ModuleName]); end; end; type _MEMORYSTATUSEX = packed record dwLength: DWORD; dwMemoryLoad: DWORD; ullTotalPhys: Int64; ullAvailPhys: Int64; ullTotalPageFile: Int64; ullAvailPageFile: Int64; ullTotalVirtual: Int64; ullAvailVirtual: Int64; ullAvailExtendedVirtual: Int64; end; {$EXTERNALSYM _MEMORYSTATUSEX} MEMORYSTATUSEX = _MEMORYSTATUSEX; {$EXTERNALSYM MEMORYSTATUSEX} LPMEMORYSTATUSEX = ^_MEMORYSTATUSEX; {$EXTERNALSYM LPMEMORYSTATUSEX} TMemoryStatusEx = _MEMORYSTATUSEX; type TGlobalMemoryStatusEx = function(out lpBuffer: TMemoryStatusEx): BOOL; stdcall; var _GlobalMemoryStatusEx: TGlobalMemoryStatusEx = nil; function GlobalMemoryStatusEx(out lpBuffer: TMemoryStatusEx): BOOL; stdcall; begin GetProcedureAddress(Pointer(@_GlobalMemoryStatusEx), kernel32, 'GlobalMemoryStatusEx'); Result := _GlobalMemoryStatusEx(lpBuffer); end; function GetTotalMemInfo: string; var MemStatusEx: TMemoryStatusEx; begin MemStatusEx.dwLength := SizeOf(MemStatusEx); if GlobalMemoryStatusEx(MemStatusEx) then begin Result := Format('TotalVirtual %d Bytes %.3f GiB', [MemStatusEx.ullTotalVirtual, MemStatusEx.ullTotalVirtual / 1024 / 1024 / 1024]); Result := Result + #13#10 + Format('AvailVirtual %d Bytes %.3f GiB', [MemStatusEx.ullAvailVirtual, MemStatusEx.ullAvailVirtual / 1024 / 1024 / 1024]); end else begin Result := 'GlobalMemoryStatusEx call failed'; end; end; begin try WriteLn(GetTotalMemInfo); Readln; except on E: Exception do WriteLn(E.ClassName, ': ', E.Message); end; end. Edited September 23 by dummzeuch Share this post Link to post
Kas Ob. 121 Posted September 23 25 minutes ago, dummzeuch said: {$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP IMAGE_FILE_LARGE_ADDRESS_AWARE} Should be {$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP or IMAGE_FILE_LARGE_ADDRESS_AWARE} You missed an "or" ! Share this post Link to post
Pat Foley 51 Posted September 24 My machine shows the same but shows 131.xxx when compiled as 64 I set the kernel32 to Kernelbase. My swapfile is hard set at 2917 or so. Share this post Link to post
FaFaFooey 6 Posted September 24 It looks like you solved your issue but since the topic is about large memory usage for a Delphi app I'd like to add that I had a very memory hungry app years ago and it kept crashing when usage reached a certain point. So I found some code about setting the working set size and my program never crashed again: var LMinWSS, LMaxWSS: NativeUInt; GetProcessWorkingSetSize(GetCurrentProcess, LMinWSS, LMaxWSS); SetProcessWorkingSetSize(GetCurrentProcess, LMinWSS, LMaxWSS * 2); Share this post Link to post
Anders Melander 1775 Posted September 24 31 minutes ago, FaFaFooey said: So I found some code about setting the working set size and my program never crashed again <Also known as throwing sh*t at a wall> 2 Share this post Link to post
FaFaFooey 6 Posted September 24 3 hours ago, Anders Melander said: <Also known as throwing sh*t at a wall> not really since my program was always fine until it hit the default working set because of the work load needed at that moment..microsoft has those apis for a reason. Share this post Link to post
Anders Melander 1775 Posted September 24 1 minute ago, FaFaFooey said: not really since my program was always fine until it hit the default working set because of the work load needed at that moment.. Full working set does not mean out of memory; We have virtual memory. Can you explain why increasing the working set solved your problem? 7 minutes ago, FaFaFooey said: microsoft has those apis for a reason. https://aviationhumor.net/russians-we-paid-whole-runway-we-use-whole-runway/ Share this post Link to post
dummzeuch 1504 Posted September 25 (edited) Just in case anybody is still reading: I found some code in my program that didn't work once the 2 GB limit was reached. "Fortunately" it was where a DLL was loaded into memory near the start of the execution so it never happened until I configured Windows to start allocating memory from top using the registry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management AllocationPreference= 0x100000 Then all of a sudden it crashed. I fixed this problem now (I hope). It was in dzlib library, unit u_dzResourceDllLoader, btw. so if somebody else is actually using that library, you might want to update. Edited September 25 by dummzeuch Share this post Link to post
Kas Ob. 121 Posted September 25 10 hours ago, FaFaFooey said: microsoft has those apis for a reason. Yes they have reasons, and this 100% doesn't include solving your mentioned problem, these APIs are about locking/reserving physical memory, to one end and one reason, minimizing page faults for one critical application, hence recover or reserve performance as much as possible, and by performance i mean the just to prevent losing performance due to high memory usage and paging to file, and that is it. 10 hours ago, FaFaFooey said: not really since my program was always fine until it hit the default working set because of the work load needed at that moment I can't imagine how this can help your application not crashing, but i can few scenarios where it delay the crash enough that you don't notice, this does not fix the problem, and there is bug(s) somewhere, it might be 1) Your bug very similar to the @dummzeuch mentioned, it is coming from casting pointer to signed integer then perform some operations on that integer, like shuffling it to other variables, or do some arithmetic operation with it, back in days 32bit application had 2GB and there wasn't a problem, i mean there wasn't crashing or access violations (because the problem is there waiting), then comes the boot switch /3GB that allowed/added one extra GB to the memory rendering all the abusing of the pointer as access violation, but this didn't happen or noticed right away as it will happen arbitrary, and only when your application utilized so much memory, in newer Windows version, the OS started to make this allocation addresses more arbitrary, hence these bugs started to show even if your application didn't utilize a lot of memory, anyways, these access violation and miss use of pointers, could be easily found and fixed by use the FastMM4 with MEM_TOP_DOWN, which i remember is default, hence will fixed right away, without you might depend on luck and high memory usage to capture. 2) Your application in fact using high volume of memory but it does abuse threading and miss using timing, if you have one Sleep because you are assuming an operation will take 5ms and you gave it something like 500ms, that should be fine, right ? NO ! , you are wrongly assuming this can be enough, because when when paging hit, the operation will block, by operation i mean a single CPU instruction will trigger OS VMM intervention due to hard page fault, and this thread will block until VMM free some physical memory, not really free (!!) but stop another thread/process and then write its memory from the physical to the page file, then load you paged memory, then continue your thread, by continue i mean literally re-execute that single instruction, the one stopped the earth planet form rotation. These bugs coming from miss using timing where very frequent back in days with HDD, with SSD it is faster and wrong logic might pass unchallenged, yet with heavy software running on OS like Windows Defender that literally capable to utilize the disk operation at tens of thousands operations per second when it does scan, these bugs come to light again. So i hope that was readable and clear, and of course i hope helps identifying your bugs, but one thing is for sure denying physical memory by manipulating working set, is not good sign at all, let your application be free of bugs and let it take small extra time due paging, and let the OS perform as intended. Share this post Link to post
David Heffernan 2345 Posted September 25 11 hours ago, FaFaFooey said: microsoft has those apis for a reason It's just that the reason isn't to do what you did Share this post Link to post
David Heffernan 2345 Posted September 25 1 hour ago, dummzeuch said: Just in case anybody is still reading: I found some code in my program that didn't work once the 2 GB limit was reached. "Fortunately" it was where a DLL was loaded into memory near the start of the execution so it never happened until I configured Windows to start allocating memory from to using the registry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management AllocationPreference= 0x100000 Then all of a sudden it crashed. I fixed this problem now (I hope). It was in dzlib library, unit u_dzResourceDllLoader, btw. so if somebody else is actually using that library, you might want to update. Top down memory allocation, which is what I presume this setting is, was how I found most pointer truncation bugs back in the day. It also broke just about every anti virus software that we tried, until I settled on the MS tool! Share this post Link to post
dummzeuch 1504 Posted September 25 5 minutes ago, David Heffernan said: Top down memory allocation, which is what I presume this setting is I got it from some random 😉 post on StackOverflow (I could not find it on any Microsoft site) Share this post Link to post