A.M. Hoornweg

Thank you very much!

@Anders Melander that would be very kind of you! I'd like to take a closer look at it at the very least !

@Anders MelanderThis recording application stores processed data records at 10 Hz, which is slow by any metric, but it performs sampling and aggregation at a much higher frequency. This is a "finished" application, tried and tested,  and we'd like to avoid breaking anything because its reliability is vital to our business. 

But since the beginning of the Covid 19 pandemic my entire department works from home and we have the need to access that data.  So the idea was to make a tiny modification to that application, to give it a circular buffer in RAM that is accessible from an outside process and to dimension that buffer big enough to contain an hour of data. 

We would then write an independent application that acts as a TCP server, allowing us to stream large chunks of data from the buffer.  Not disturbing the data acquisition itself is an absolute necessity, hence my question about a lean locking mechanism. It is absolutely no problem if the consumer must wait a few ms, but the producer should be as undisturbed as possible. 

And of course I meant the word "pointer" in a generic sense, not as a logical address. The buffer would get a 4 kb header with a version number and some properly aligned control variables. All "pointers" in there will just be record numbers.

@Anders Melander  in this case a signal to the consumer thread that data is available isn't necessary.    The producer writes into the ring buffer @ 10Hz, the consumer polls the ring buffer every few seconds and pulls whatever was put in there. A synchronization object is only needed for atomicity of the pointers and counters.

Maybe a crazy idea, but since this block of memory is in a shared address space of two applications, would it be possible to place a "critical section" object in there?   

I'm looking for a lightweight method to serialize access (something more lightweight than a mutex). One process will be writing the ring buffer and another will be reading it.

Thank you all! I'm giving it a try.

Hello all,

I need to map a block of memory (a ring buffer) into the address space of multiple processes for shared access. The Windows API offers a technology called memory-mapped files for this purpose.

My question: does the "Interlockedxxxx" API work reliably across process boundaries ?  The windows API documentation does not state that explicitly.  My hope is that these commands translate into process-agnostic cpu opcodes.

The disadvantage of all these broadcast methods is that they don't work across segmented LAN's.   And then there's the can of worms called IPv6.

The "clean" way would be to use Bonjour a.k.a.  ZeroConf.

https://en.wikipedia.org/wiki/Zero-configuration_networking

It is a somewhat "standardized" way of announcing all sorts of services using UDP broadcasts.  Companies like Apple use it for discovering wireless devices such as printers.  A zeroconf  implementation can be found in the Remobjects suite ( https://docs.remotingsdk.com/Clients/Concepts/ROZeroConf/ )  but after some googling I found an open source implementation as well ( https://github.com/deltics/delphi.libs/tree/master/bonjour ).

On 8/21/2019 at 9:14 AM, Guest said:

For backups it is better to use split files.

That's true if your backup target drive is fat32, which doesn't support files > 4GB. 

Otherwise I would be more comfortable with one big disk file (which I use).

I run Delphi /2007, XE and Sidney) inside a VMWare VM since "forever" and I'm very happy with the speed. One setting that made a lot of difference (speed wise) for me is "disable memory page trimming". In my case, the virtual disk image is on a SSD and all hard disks are configured as "independent".  

I have a bad feeling about this.

When I perform a search (using Agent Ransack) for the string "Integer("  in the subdirectory where I keep my third-party libraries, I see lots and lots of places where pointers are being cast into signed 32-bit integers.  I see this especially in old libraries originally written by TurboPower, such as Abbrevia and Async Professional.  For example, the version of Abbrevia that can currently be downloaded using Getit for Delphi 10.4.1 contains the following line in unit AbCabTyp.pas (line 460):

  Result := Integer(TFileStream.Create(Archive.FIIPName, fmCreate))

This is just an example (my application does not use that particular unit) but these libraries are riddled with such gems.   Am I neurotic if I distrust such libraries in 32-bit code compiled with the large address space awareness flag ? 

I've got it working now, thanks!

22 minutes ago, Jacek Laskowski said:

https://www.thedelphigeek.com/2019/04/fastmm4-large-memory.html

Thank you!

I am already using that version of FastMM4 with the default options (I just checked, options fulldebugmode and AlwaysAllocateTopDown are set). 

But when I debug my program, compiled with {$SetPEFlags $0020}, it appears that allocated objects such as my main form still have addresses way below 2 GB.  Could it be that the {AlwaysAllocateTopDown} option isn't working in FastMM4 ?

Hello all,

I have a 32-bit application written in Delphi XE that I would like to compile with the PE flag "IMAGE_FILE_LARGE_ADDRESS_AWARE", because I'm occasionally nearing the 2 GB RAM limit.  I can't compile this application in 64 bit mode due to dependencies on some 32-bit DLLs.

My question:

in order to test and debug this in a reasonable time frame, is there any way I can quickly "provoke" any possible lurking bugs (such as pointer math that may still use signed integers somewhere)? For example, is there any way of telling the fastMM4 memory manager to start allocating heap addresses top-down instead of bottoms-up, so that all objects are allocated "above" 2GB ?

16 hours ago, Stefan Glienke said:

Those numbers make me assume that you ran in Debug config (i.e. without $O+).

No!  But your reply made me double-check.

I  ran the original test in a "tbutton.onclick" event instead of FormCreate, expecting that it would make no difference. It's just my way of doing things.

Weird enough, it DOES make a difference. Inside FormCreate() the numbers are 495 and 241 ms. 

Why the heck does stuff run slower in an OnClick() event?

On my Delphi 10.4.1, both loops run equally fast (1214 and 1213 ms).  This is on a core i9 notebook processor.

What throughput are we talking about here anyway, in kilobytes per second? 

I think good design is much more important than premature optimizations.  

6 minutes ago, Fr0sT.Brutal said:

Yes, faced the same issue. Despite its name, this type is kinda tricky. I had to use SetCodePage(S, $FFFF, False); in my function where I create variabe of this type.

You can do that on any Ansistring. RawByteString does not add any value here.

Also, it is a can of worms.  If you do something like c:=a+b, what's the code page of the result?

I must really warn for RawByteString. 

The documentation says that you should not instantiate variables of type RawByteString, it is meant to be used as a parameter or result type of a function.

I have tried, and if you declare a variable of type Rawbytestring and put data into it, it will behave just like Ansistring and has the same code page issues:

procedure TForm2.Button2Click(Sender: TObject);
var raw:RawByteString;
begin
     setlength(raw,100);
     Showmessage(format('The code page of this rawbytestring is %d',[stringcodepage(raw)]));
end;

 On my system this rawbytestring has code page 1252! 

9 minutes ago, Anders Melander said:

The only general advice I can give is: Convenience or Performance? Pick one.

 

I won't get into this discussion about which of the many different data types can solve your problem best, because your requirements seems rather fluid and it's impossible to give any targeted advice, when we don't know anything about the specifics.

I'm puzzled why one would even be asking a question like this other than to pass time. If the implementation details are important you shouldn't rely on something someone said on a forum anyway. Try different solutions and benchmark them. If none of the solutions are good enough then there will be something to discuss.

 

Disclaimer: I've been up all night and I'm on my eight cup of coffee.

I sympathize with the OP, I have the same problem often, data streams containing a mixture of text and binary.   

- Many communication components for RS232 (such as good-old Async Pro) have receive events which pass a string. So far I haven't seen any that has the decency to pass a tBytes. 

- With modems (both the old-fashion ones and modern cellular 4G ones) one communicates in the "Hayes" protocol which is text-based until the connection is established, then the data stream becomes binary.

4 minutes ago, Kryvich said:

@A.M. Hoornweg It depends on Windows ANSI codepage. I have CP-1251.

The compiler also issues a lot of warnings:

 

Ansistring28591.dpr

Yes of course the compiler warns. Ansistrings (with the exception of utf8string) can contain only a subset of Unicode so the compiler will warn if you convert between the types.

But the code proves without any doubt that "Binarystring" can be losslessly assigned to Unicodestring and back again and that the ordinal values stay the same.  So for storing bytes (which contain only values 0-255 anyway) that is perfectly safe to use.   

Just don't assign them to any other flavor of Ansistring or things get corrupted.

42 minutes ago, Kryvich said:

We need a real RawByteString without ANY implicit conversions. Just show a compiler error when trying to implicitly convert a string. Or improved TBytes with copy-on-write and all manipulation routines.

 Ansistring (28591) does that, if you convert it to unicode then the ordinal values of the widechars are identical to the ordinal values of the ansichars.

Type Binarystring = type Ansistring (28591);


procedure TForm2.Button1Click(Sender: TObject);
VAR Original, aCopy:Binarystring;
    Ansi:Ansistring;
    Raw:Rawbytestring;
    changed:Boolean;
    Uni:Unicodestring; i:integer;
begin
     SetLength(Original,256);
     for i:=1 to 256 do
         Original[i]:=ansichar(i-1);

     Uni:=Original; //copy to unicode, see if ordinal values are the same

     changed:=False;
     for i:=1 to 256 do
       if Byte(Original[i]) <> WORD(uni[i]) then
           changed:=true;

   If not changed then
      showmessage('It appears that the Widechar ordinal values are THE SAME as the Ansichar values');

    aCopy:=Uni;
    IF aCopy=Original then
         showmessage('It appears that converting back to binarystring is SAFE');

   Ansi:=Uni;
    IF Original<>Ansi then
         showmessage('It appears that converting back to ANSIstring is UNSAFE');

   Raw:=Uni;
    IF Original<>Raw then
         showmessage('It appears that converting back to RAWBYTEstring is UNSAFE');
end;

6 minutes ago, Kryvich said:

@A.M. Hoornweg Why duplicate what the standard library already does for strings, including byte manipulation and copy-on-write?

Because the data is not strings.  But the in-memory representation of a dynamic array is rather similar so the functionality should be straightforward to duplicate.  And thanks to generics, it's probably possible to design the base class to be universal, IIRC a tbytes is just something like "tarray<byte>" (but I may be wrong here).

3 minutes ago, Rollo62 said:

Yes I have that, but the strings are so much more elegant.
I would whish to have better TBytes and dyn. array support like that in Delphi.

Lookup "operator overloading". 

It should be possible to do something like newbytes:=something+somethingelse+'Hello world';    

But adding text to tBytes or searching a text inside a tBytes would mean code page handling.  You could consider defining a Binarystring type (type binarystring=type Ansistring (28591)) because that's least likely to get messed up.