Only 2 GB available despite IMAGE_FILE_LARGE_ADDRESS_AWARE

[dummzeuch 1499]

The Win32 Delphi 2007 program below sets the PEFlag for IMAGE_FILE_LARGE_ADDRESS_AWARE, so I would expect it to have 3 GB of memory available.

But it writes:

TotalVirtual 2147352576 Bytes 2,000 GiB
AvailVirtual 2093465600 Bytes 1,950 GiB

So why does it say there are only 2 GB?

The OS is Windows 10 pro 64 Bit with 16 GB of physical memory (from which a lot is available).

I could have sworn that this has worked before.

 

Edit: Duh! I forgot an 'or' between 'IMAGE_FILE_NET_RUN_FROM_SWAP' and 'IMAGE_FILE_LARGE_ADDRESS_AWARE'

program Project1;

{$APPTYPE CONSOLE}

uses
  Windows,
  SysUtils;

{$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP IMAGE_FILE_LARGE_ADDRESS_AWARE}

procedure GetProcedureAddress(var P: Pointer; const ModuleName, ProcName: string);
var
  ModuleHandle: HMODULE;
begin
  if not Assigned(P) then begin
    ModuleHandle := GetModuleHandle(PChar(ModuleName));
    if ModuleHandle = 0 then begin
      ModuleHandle := SafeLoadLibrary(PChar(ModuleName));
      if ModuleHandle = 0 then
        raise Exception.CreateFmt('Library %s not found', [ModuleName]);
    end;
    P := GetProcAddress(ModuleHandle, PChar(ProcName));
    if not Assigned(P) then
      raise Exception.CreateFmt('Function %s not found in library %s', [ProcName, ModuleName]);
  end;
end;

type
  _MEMORYSTATUSEX = packed record
    dwLength: DWORD;
    dwMemoryLoad: DWORD;
    ullTotalPhys: Int64;
    ullAvailPhys: Int64;
    ullTotalPageFile: Int64;
    ullAvailPageFile: Int64;
    ullTotalVirtual: Int64;
    ullAvailVirtual: Int64;
    ullAvailExtendedVirtual: Int64;
  end;
{$EXTERNALSYM _MEMORYSTATUSEX}

  MEMORYSTATUSEX = _MEMORYSTATUSEX;
{$EXTERNALSYM MEMORYSTATUSEX}
  LPMEMORYSTATUSEX = ^_MEMORYSTATUSEX;
{$EXTERNALSYM LPMEMORYSTATUSEX}

  TMemoryStatusEx = _MEMORYSTATUSEX;

type
  TGlobalMemoryStatusEx = function(out lpBuffer: TMemoryStatusEx): BOOL; stdcall;

var
  _GlobalMemoryStatusEx: TGlobalMemoryStatusEx = nil;

function GlobalMemoryStatusEx(out lpBuffer: TMemoryStatusEx): BOOL; stdcall;
begin
  GetProcedureAddress(Pointer(@_GlobalMemoryStatusEx), kernel32, 'GlobalMemoryStatusEx');
  Result := _GlobalMemoryStatusEx(lpBuffer);
end;

function GetTotalMemInfo: string;
var
  MemStatusEx: TMemoryStatusEx;
begin
  MemStatusEx.dwLength := SizeOf(MemStatusEx);
  if GlobalMemoryStatusEx(MemStatusEx) then begin
    Result := Format('TotalVirtual %d Bytes %.3f GiB',
      [MemStatusEx.ullTotalVirtual, MemStatusEx.ullTotalVirtual / 1024 / 1024 / 1024]);
    Result := Result + #13#10
      + Format('AvailVirtual %d Bytes %.3f GiB',
      [MemStatusEx.ullAvailVirtual, MemStatusEx.ullAvailVirtual / 1024 / 1024 / 1024]);
  end else begin
    Result := 'GlobalMemoryStatusEx call failed';
  end;
end;

begin
  try
    WriteLn(GetTotalMemInfo);
    Readln;
  except
    on E: Exception do
      WriteLn(E.ClassName, ': ', E.Message);
  end;
end.

 

Edited September 23 by dummzeuch

[Kas Ob. 121]

25 minutes ago, dummzeuch said:

{$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP IMAGE_FILE_LARGE_ADDRESS_AWARE}

Should be 

{$SETPEFLAGS IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP or IMAGE_FILE_NET_RUN_FROM_SWAP or IMAGE_FILE_LARGE_ADDRESS_AWARE}

You missed an "or" !

[Pat Foley 51]

My machine shows the same but shows 131.xxx when compiled as 64 I set the kernel32 to Kernelbase. My swapfile is hard set at 2917 or so.  

[FaFaFooey 6]

It looks like you solved your issue but since the topic is about large memory usage for a Delphi app I'd like to add that I had a very memory hungry app years ago and it kept crashing when usage reached a certain point. So I found some code about setting the working set size and my program never crashed again:

 

var
  LMinWSS, LMaxWSS: NativeUInt;


  GetProcessWorkingSetSize(GetCurrentProcess, LMinWSS, LMaxWSS);
  SetProcessWorkingSetSize(GetCurrentProcess, LMinWSS, LMaxWSS * 2);

 

[Anders Melander 1773]

31 minutes ago, FaFaFooey said:

So I found some code about setting the working set size and my program never crashed again

<Also known as throwing sh*t at a wall>

[FaFaFooey 6]

3 hours ago, Anders Melander said:

<Also known as throwing sh*t at a wall>

not really since my program was always fine until it hit the default working set because of the work load needed at that moment..microsoft has those apis for a reason.

[Anders Melander 1773]

1 minute ago, FaFaFooey said:

not really since my program was always fine until it hit the default working set because of the work load needed at that moment..

Full working set does not mean out of memory; We have virtual memory.

 

Can you explain why increasing the working set solved your problem?

 

7 minutes ago, FaFaFooey said:

microsoft has those apis for a reason.

https://aviationhumor.net/russians-we-paid-whole-runway-we-use-whole-runway/

[dummzeuch 1499]

Just in case anybody is still reading: I found some code in my program that didn't work once the 2 GB limit was reached. "Fortunately" it was where a DLL was loaded into memory near the start of the execution so it never happened until I configured Windows to start allocating memory from top using the registry:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
AllocationPreference= 0x100000 

Then all of a sudden it crashed.

 

I fixed this problem now (I hope).

It was in dzlib library, unit u_dzResourceDllLoader, btw. so if somebody else is actually using that library, you might want to update.

Edited September 25 by dummzeuch

[Kas Ob. 121]

10 hours ago, FaFaFooey said:

microsoft has those apis for a reason.

Yes they have reasons, and this 100% doesn't include solving your mentioned problem, these APIs are about locking/reserving physical memory, to one end and one reason, minimizing page faults for one critical application, hence recover or reserve performance as much as possible, and by performance i mean the just to prevent losing performance due to high memory usage and paging to file, and that is it.

 

10 hours ago, FaFaFooey said:

not really since my program was always fine until it hit the default working set because of the work load needed at that moment

I can't imagine how this can help your application not crashing, but i can few scenarios where it delay the crash enough that you don't notice, this does not fix the problem, and there is bug(s) somewhere, it might be 

1) Your bug very similar to the @dummzeuch mentioned, it is coming from casting pointer to signed integer then perform some operations on that integer, like shuffling it to other variables, or do some arithmetic operation with it, back in days 32bit application had 2GB and there wasn't a problem, i mean there wasn't crashing or access violations (because the problem is there waiting), then comes the boot switch /3GB that allowed/added one extra GB to the memory rendering all the abusing of the pointer as access violation, but this didn't happen or noticed right away as it will happen arbitrary, and only when your application utilized so much memory, in newer Windows version, the OS started to make this allocation addresses more arbitrary, hence these bugs started to show even if your application didn't utilize a lot of memory, anyways, these access violation and miss use of pointers, could be easily found and fixed by use the FastMM4 with MEM_TOP_DOWN, which i remember is default, hence will fixed right away, without you might depend on luck and high memory usage to capture.

2) Your application in fact using high volume of memory but it does abuse threading and miss using timing, if you have one Sleep because you are assuming an operation will take 5ms and you gave it something like 500ms, that should be fine, right ? NO ! , you are wrongly assuming this can be enough, because when when paging hit, the operation will block, by operation i mean a single CPU instruction will trigger OS VMM intervention due to hard page fault, and this thread will block until VMM free some physical memory, not really free (!!) but stop another thread/process and then write its memory from the physical to the page file, then load you paged memory, then continue your thread, by continue i mean literally re-execute that single instruction, the one stopped the earth planet form rotation.

These bugs coming from miss using timing where very frequent back in days with HDD, with SSD it is faster and wrong logic might pass unchallenged, yet with heavy software running on OS like Windows Defender that literally capable to utilize the disk operation at tens of thousands operations per second when it does scan, these bugs come to light again.

 

So i hope that was readable and clear, and of course i hope helps identifying your bugs, but one thing is for sure denying physical memory by manipulating working set, is not good sign at all, let your application be free of bugs and let it take small extra time due paging, and let the OS perform as intended.

[David Heffernan 2345]

11 hours ago, FaFaFooey said:

microsoft has those apis for a reason

It's just that the reason isn't to do what you did

[David Heffernan 2345]

1 hour ago, dummzeuch said:

Just in case anybody is still reading: I found some code in my program that didn't work once the 2 GB limit was reached. "Fortunately" it was where a DLL was loaded into memory near the start of the execution so it never happened until I configured Windows to start allocating memory from to using the registry:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
AllocationPreference= 0x100000 

Then all of a sudden it crashed.

 

I fixed this problem now (I hope).

It was in dzlib library, unit u_dzResourceDllLoader, btw. so if somebody else is actually using that library, you might want to update.

Top down memory allocation, which is what I presume this setting is, was how I found most pointer truncation bugs back in the day. It also broke just about every anti virus software that we tried, until I settled on the MS tool!

[dummzeuch 1499]

5 minutes ago, David Heffernan said:

Top down memory allocation, which is what I presume this setting is

I got it from some random 😉 post on StackOverflow (I could not find it on any Microsoft site)