David Heffernan

It's odd that the emba tool won't work directly with platform standard import libs. You'd think they'd find a way to do that.

All the details we'd need to say anything about your program.

I have never tried to allocate such enormous amounts of memory using Delphi's heap manager (Fastmm4), I really don't know how it behaves if you try to allocate one huge chunk that is bigger than what the machine has physically. These huge blocks won't be allocated by fastmn. They will be passed through to VirtualAlloc. But we aren't talking about a chunk of memory greater than what the machine has physically anyway. I'm rather sceptical that MMF would ever be appropriate here.

Doesn't memory already handle all of this? I mean, the paging system does all of this already surely?

They control the code for all processes involved. That's the crucial difference.

Why would a memory mapped file be remotely helpful here?

No. I have classes that have similar methods. For example, my reader class can be used with while/readln pattern, or a for in loop.

You can add buffers wherever you like. I have line based text file readers and writers that buffer actual file access.

Read the code of an open source project that does this and work out what they do

You don't need to use these apis to have buffers

One was mentioned above. But there are lots. This looks solid https://github.com/vilya/miniply

Does the C library handle binary format?

Clearly it's a procedure not a function though.

Hyoerthreading is probably useless. I've never found a task which benefits from it. I guess they must exist though, or is it just pure marketing? Modern AMD chipsets seems to have remarkable scaling though.

Depending on what you do when you query these bits you may find that the memory is the bottleneck and threading won't help. Did you do any benchmarking yet? Also if performance matters then delphi is invariably not the answer.

This looks like a documentation error. It obviously doesn't make sense for the result to be passed back through a var param and a function return value. I think it should be like so: procedure BlockWrite(var F: File; const Buf; Count: Integer; var Written: Integer); overload; procedure BlockWrite(var F: File; const Buf; Count: Integer); overload; procedure BlockWrite(var F: File; const Buf; Count: Integer; var Written: Integer); overload; procedure BlockWrite(var F: File; const Buf; Count: Integer); overload; Your code gives the same error in Delphi 6, so I don't think anything has changed. Have a look at this section of Ray Lischner's book which says the same: https://www.oreilly.com/library/view/delphi-in-a/1565926595/re28.html

Does the behaviour change then depending on whether or not IO errors are enabled?

unit Bitset; interface uses SysUtils, Math; type TBitSet = record private FBitCount: Int64; FSets: array of set of 0..255; class function SetCount(BitCount: Int64): Int64; static; procedure MakeUnique; procedure GetSetIndexAndBitIndex(Bit: Int64; out SetIndex: Int64; out BitIndex: Integer); function GetIsEmpty: Boolean; procedure SetBitCount(Value: Int64); function GetSize: Int64; public class operator In(const Bit: Int64; const BitSet: TBitSet): Boolean; class operator Equal(const bs1, bs2: TBitSet): Boolean; class operator NotEqual(const bs1, bs2: TBitSet): Boolean; property BitCount: Int64 read FBitCount write SetBitCount; property Size: Int64 read GetSize; property IsEmpty: Boolean read GetIsEmpty; procedure Clear; procedure IncludeAll; procedure Include(const Bit: Int64); procedure Exclude(const Bit: Int64); end; implementation { TBitSet } procedure TBitSet.MakeUnique; begin // this is used to implement copy-on-write so that the type behaves like a value SetLength(FSets, Length(FSets)); end; procedure TBitSet.GetSetIndexAndBitIndex(Bit: Int64; out SetIndex: Int64; out BitIndex: Integer); begin Assert(InRange(Bit, 0, FBitCount-1)); SetIndex := Bit shr 8; // shr 8 = div 256 BitIndex := Bit and 255; // and 255 = mod 256 end; function TBitSet.GetIsEmpty: Boolean; var i: Int64; begin for i := 0 to High(FSets) do begin if FSets[i]<>[] then begin Result := False; Exit; end; end; Result := True; end; procedure TBitSet.SetBitCount(Value: Int64); var Bit, BitIndex: Integer; SetIndex: Int64; begin if (Value<>FBitCount) or not Assigned(FSets) then begin Assert(Value>=0); FBitCount := Value; SetLength(FSets, SetCount(Value)); if Value>0 then begin (* Ensure that unused bits are cleared, necessary give the CompareMem call in Equal. This also means that state does not persist when we decrease and then increase BitCount. For instance, consider this code: var bs: TBitSet; ... bs.BitCount := 2; bs.Include(1); bs.BitCount := 1; bs.BitCount := 2; Assert(not (1 in bs)); *) GetSetIndexAndBitIndex(Value - 1, SetIndex, BitIndex); for Bit := BitIndex + 1 to 255 do begin System.Exclude(FSets[SetIndex], Bit); end; end; end; end; function TBitSet.GetSize: Int64; begin Result := Length(FSets)*SizeOf(FSets[0]); end; class function TBitSet.SetCount(BitCount: Int64): Int64; begin Result := (BitCount + 255) shr 8; // shr 8 = div 256 end; class operator TBitSet.In(const Bit: Int64; const BitSet: TBitSet): Boolean; var SetIndex: Int64; BitIndex: Integer; begin BitSet.GetSetIndexAndBitIndex(Bit, SetIndex, BitIndex); Result := BitIndex in BitSet.FSets[SetIndex]; end; class operator TBitSet.Equal(const bs1, bs2: TBitSet): Boolean; begin Result := (bs1.FBitCount=bs2.FBitCount) and CompareMem(Pointer(bs1.FSets), Pointer(bs2.FSets), bs1.Size); end; class operator TBitSet.NotEqual(const bs1, bs2: TBitSet): Boolean; begin Result := not (bs1=bs2); end; procedure TBitSet.Clear; var i: Int64; begin MakeUnique; for i := 0 to High(FSets) do begin FSets[i] := []; end; end; procedure TBitSet.IncludeAll; var i: Int64; begin for i := 0 to BitCount-1 do begin Include(i); end; end; procedure TBitSet.Include(const Bit: Int64); var SetIndex: Int64; BitIndex: Integer; begin MakeUnique; GetSetIndexAndBitIndex(Bit, SetIndex, BitIndex); System.Include(FSets[SetIndex], BitIndex); end; procedure TBitSet.Exclude(const Bit: Int64); var SetIndex: Int64; BitIndex: Integer; begin MakeUnique; GetSetIndexAndBitIndex(Bit, SetIndex, BitIndex); System.Exclude(FSets[SetIndex], BitIndex); end; end. This is based on code of mine that has is limited to integer bit count. I've not tested it extended to Int64, but I'm sure anyone that wanted to use random code like this would test.

Why would you not expect to get a 32GB block when the available address space is 128TB?

What about the cost of moving data between gpu and main memory? Benefits of gpu are in highly parallel computation. Where is that in this scenario. Just saying GPU doesn't make something fast or efficient.

How could this be more efficient than a contiguous array?

Show complete but minimal code please

No. You use a hashed collection. Like a dictionary, but one without a value, which is known as a set. But at 5% full then I don't think it will get you any benefit, because of the overhead of the hashed collection.

Expect it to be discontinued at short notice when Google get bored of it

How many such instances of this type do you need in memory at any time? And what's the expected number of bits that are set at any time? Do you really need to store all bits, both 0 and 1. Can't you just stor the 1s and infer the 0s from the fact that they aren't stored as 1s?