TFileStream for memory mapped files

[AlexBelo 12]

Hi all.

 

Do anybody know any ready for use class for handling memory mapped files in TFileStream manner

(r/w operations with emulation of continuous data stream)?

 

TIA.

--

Alex

[Anders Melander 1782]

A search on my HD for TMemoryMappedStream found a few different implementations. The attached one, by Dmitry Streblechenko, looks to be the cleanest and it has no dependencies.

Quote

      Dmitry Streblechenko
      Department of Physics and Astronomy
      Arizona State University
      Box 871504
      Tempe, AZ 85287-1504

      E-mail: dmitrys@asu.edu

 class TMemoryMappedFileStream
 Encapsulation of the Windows Memory Mapped object.

 Use, modify or distribute (do not sell) as you please
 as long as you give me a credit.

 Allows for
        1) simple interprocess communications via the named
           mapped objects transparently accessible either
           as the TStream decendents or as arrays using
           Memory property
        2) Access to any files up to 4GB size as the memory arrays
           using Memory property

 Usage:
        Just like any TStream descendent ( Read(), Write(),
        Size, Position, etc.)

 Additional properties and methods:
   property Memory   :pointer; (read only) - use like any
                               other pointer, DO NOT use in the
                               memory de/re/allocation routines.
   property FileName :string; (read only) - name of the
                              file mapped to memory
   property MapName  :string; (read only) - name of the memory
                              mapped object (has nothing to do with
                              the FileName).
   property MemoryDelta:longint; (read/write) - increment with which
                                 object's allocated memory will grow

 Hints:

  To create new mapped file object, use constructor
    Create(FileName, MapName,Mode)
  FileName - name of the file to map, can be '' if you want to use
             Windows swap file (useful for interprocess communications)
  MapName - name of the mapped object to create, can be '' if
            you do not want to open it using different mapped file object
  Mode - file open/creat mode, see TFileStream.Create help

  To use existing mapped file object, use constructor
     Open(MapName, Mode)
  MapName - name of the existing mapped file object (see above),
            creator can reside in a different process.
  Mode - Mapped object open mode, see TFileStream.Create help.
         DO NOT use fmCreate mode, exception will be thrown!
         All other modes are Ok.

Example - count all letters 'a' in a file
         type hugearray=array[0..$FFFFFFFF-1] of char;
         var mmf: TMemoryMappedFileStream;
             HugeFileName:string;
             i,counter:longint;
         begin
         HugeFileName:=...;
         mmf:=TMemoryMappedFileStream.Create(HugeFileName,'',fmOpenRead);
         counter:=0;
         for i:=0 to mmf.Size-1 do
          if hugearray(mmf.memory^)='a' then inc(counter);

 

 

dstreams.zip

[Fr0sT.Brutal 900]

https://sourceforge.net/projects/tpipro/

in IpStrms.pas

[AlexBelo 12]

Thank you for your responce.

 

AFAICS both sources do not automatically process boundaries of currently mapped view during r/w operations . . . 

I.e. if start of r/w data block is in current view but its end is not then operation will fail.

 

But I wish to r/w stream without thinking about boundaries of current view; Read/Write methods should move view (of predefined size) automatically.

Of course, I can modify Read/Write methods myself to achieve necessary functionality.

[Anders Melander 1782]

A memory mapped file can not be expanded. By design.

 

Of course you can emulate expansion by closing the current view, expanding the file and then mapping the file again, but that is not a safe operation as any existing pointers to the old view will now be invalid.

[AlexBelo 12]

I don't want to expand file but operate with big file via mapped view of reasonable size; mapping view of entire file is impossible.

 

Take a look: 

MapViewOfFile function

https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile

 

Mapping a file makes the specified portion of a file visible in the address space of the calling process.

For files that are larger than the address space, you can only map a small portion of the file data at one time. When the first view is complete, you can unmap it and map a new view.

[Fr0sT.Brutal 900]

1 hour ago, AlexBelo said:

I can modify Read/Write methods myself to achieve necessary functionality

Do it 😉 considering that 32bit will likely become obsolete in the near future, I doubt anyone will implement the functionality you're telling about. From the 1st glance, it shouldn't be too hard to implement

[AlexBelo 12]

I still use RAD2007, it is 32bit.

And yes - it is not hard to implement "walking" view. (But we always have a hope that someone already did it before. 🙂)

 

Mapping view of big size (entire file as subcase) takes a lot of memory (no miracle here) so it is not "for free" even in 64bit OS.

[David Heffernan 2345]

5 hours ago, AlexBelo said:

Hi all.

 

Do anybody know any ready for use class for handling memory mapped files in TFileStream manner

(r/w operations with emulation of continuous data stream)?

 

TIA.

--

Alex

Why would you wrap a memory map with a stream? You use a memory map when you want to access a file using memory operations. But the stream interface reads chunks of the file in a caller provided buffer. If you want a stream interface I don't see the point of a memory map. 

[AlexBelo 12]

You are right in general when you say about direct manipulations with data in created view.

But my main motivation here is standartization of data access with descendant of TStream.

 

[Fr0sT.Brutal 900]

1 hour ago, AlexBelo said:

But my main motivation here is standartization of data access with descendant of TStream.

TFileStream ;D

 

3 hours ago, AlexBelo said:

Mapping view of big size (entire file as subcase) takes a lot of memory (no miracle here) so it is not "for free" even in 64bit OS.

Hmm, from what I know mapping doesn't read-and-allocate the contents of the mapped file. Contrary, it allows working with the contents like it was allocated in memory

Edited May 31, 2021 by Fr0sT.Brutal

[AlexBelo 12]

13 hours ago, Fr0sT.Brutal said:

TFileStream ;D

Hmm... Good option. 🙂

 

13 hours ago, Fr0sT.Brutal said:

from what I know mapping doesn't read-and-allocate the contents of the mapped file. Contrary, it allows working with the contents like it was allocated in memory

As documentation states:

MapViewOfFile function (memoryapi.h)

Maps a view of a file mapping into the address space of a calling process.

 

So mapped view is in "my" address space.

AFAIUI, OS gives me (and others who use the same mapped file) access to some "external" pages in memory but I see these pages as array allocated in "my" memory. And big view will take big continuous piece of virtual memory in my process.

 

About TFileStream. As documentation states:

 

... file views derived from any file mapping object that is backed by the same file are coherent or identical at a specific time. Coherency is guaranteed for views within a process and for views that are mapped by different processes.

 

(Not surprising: data resides in the same pages in memory.)

AFAIUI "normal" files don't have such property.

Edited June 1, 2021 by AlexBelo

[Fr0sT.Brutal 900]

8 hours ago, AlexBelo said:

So mapped view is in "my" address space.

AFAIUI, OS gives me (and others who use the same mapped file) access to some "external" pages in memory but I see these pages as array allocated in "my" memory. And big view will take big continuous piece of virtual memory in my process.

Yep, after mapping a view your app in fact has less maximum address space available. However, this only has meaning with x32 apps. In x64 apps you have 18M Terabytes available so one little mapped view won't hurt much 🙂

8 hours ago, AlexBelo said:

About TFileStream. As documentation states:

 

... file views derived from any file mapping object that is backed by the same file are coherent or identical at a specific time. Coherency is guaranteed for views within a process and for views that are mapped by different processes.

Soooo...?