A.M. Hoornweg

@Anders Melander  in this case a signal to the consumer thread that data is available isn't necessary.    The producer writes into the ring buffer @ 10Hz, the consumer polls the ring buffer every few seconds and pulls whatever was put in there. A synchronization object is only needed for atomicity of the pointers and counters.

Maybe a crazy idea, but since this block of memory is in a shared address space of two applications, would it be possible to place a "critical section" object in there?   

I'm looking for a lightweight method to serialize access (something more lightweight than a mutex). One process will be writing the ring buffer and another will be reading it.

Thank you all! I'm giving it a try.

Hello all,

I need to map a block of memory (a ring buffer) into the address space of multiple processes for shared access. The Windows API offers a technology called memory-mapped files for this purpose.

My question: does the "Interlockedxxxx" API work reliably across process boundaries ?  The windows API documentation does not state that explicitly.  My hope is that these commands translate into process-agnostic cpu opcodes.

The disadvantage of all these broadcast methods is that they don't work across segmented LAN's.   And then there's the can of worms called IPv6.

The "clean" way would be to use Bonjour a.k.a.  ZeroConf.

https://en.wikipedia.org/wiki/Zero-configuration_networking

It is a somewhat "standardized" way of announcing all sorts of services using UDP broadcasts.  Companies like Apple use it for discovering wireless devices such as printers.  A zeroconf  implementation can be found in the Remobjects suite ( https://docs.remotingsdk.com/Clients/Concepts/ROZeroConf/ )  but after some googling I found an open source implementation as well ( https://github.com/deltics/delphi.libs/tree/master/bonjour ).

On 8/21/2019 at 9:14 AM, Guest said:

For backups it is better to use split files.

That's true if your backup target drive is fat32, which doesn't support files > 4GB. 

Otherwise I would be more comfortable with one big disk file (which I use).

I run Delphi /2007, XE and Sidney) inside a VMWare VM since "forever" and I'm very happy with the speed. One setting that made a lot of difference (speed wise) for me is "disable memory page trimming". In my case, the virtual disk image is on a SSD and all hard disks are configured as "independent".  

I have a bad feeling about this.

When I perform a search (using Agent Ransack) for the string "Integer("  in the subdirectory where I keep my third-party libraries, I see lots and lots of places where pointers are being cast into signed 32-bit integers.  I see this especially in old libraries originally written by TurboPower, such as Abbrevia and Async Professional.  For example, the version of Abbrevia that can currently be downloaded using Getit for Delphi 10.4.1 contains the following line in unit AbCabTyp.pas (line 460):

  Result := Integer(TFileStream.Create(Archive.FIIPName, fmCreate))

This is just an example (my application does not use that particular unit) but these libraries are riddled with such gems.   Am I neurotic if I distrust such libraries in 32-bit code compiled with the large address space awareness flag ? 

I've got it working now, thanks!

22 minutes ago, Jacek Laskowski said:

https://www.thedelphigeek.com/2019/04/fastmm4-large-memory.html

Thank you!

I am already using that version of FastMM4 with the default options (I just checked, options fulldebugmode and AlwaysAllocateTopDown are set). 

But when I debug my program, compiled with {$SetPEFlags $0020}, it appears that allocated objects such as my main form still have addresses way below 2 GB.  Could it be that the {AlwaysAllocateTopDown} option isn't working in FastMM4 ?

Hello all,

I have a 32-bit application written in Delphi XE that I would like to compile with the PE flag "IMAGE_FILE_LARGE_ADDRESS_AWARE", because I'm occasionally nearing the 2 GB RAM limit.  I can't compile this application in 64 bit mode due to dependencies on some 32-bit DLLs.

My question:

in order to test and debug this in a reasonable time frame, is there any way I can quickly "provoke" any possible lurking bugs (such as pointer math that may still use signed integers somewhere)? For example, is there any way of telling the fastMM4 memory manager to start allocating heap addresses top-down instead of bottoms-up, so that all objects are allocated "above" 2GB ?

16 hours ago, Stefan Glienke said:

Those numbers make me assume that you ran in Debug config (i.e. without $O+).

No!  But your reply made me double-check.

I  ran the original test in a "tbutton.onclick" event instead of FormCreate, expecting that it would make no difference. It's just my way of doing things.

Weird enough, it DOES make a difference. Inside FormCreate() the numbers are 495 and 241 ms. 

Why the heck does stuff run slower in an OnClick() event?

On my Delphi 10.4.1, both loops run equally fast (1214 and 1213 ms).  This is on a core i9 notebook processor.

What throughput are we talking about here anyway, in kilobytes per second? 

I think good design is much more important than premature optimizations.  

6 minutes ago, Fr0sT.Brutal said:

Yes, faced the same issue. Despite its name, this type is kinda tricky. I had to use SetCodePage(S, $FFFF, False); in my function where I create variabe of this type.

You can do that on any Ansistring. RawByteString does not add any value here.

Also, it is a can of worms.  If you do something like c:=a+b, what's the code page of the result?

I must really warn for RawByteString. 

The documentation says that you should not instantiate variables of type RawByteString, it is meant to be used as a parameter or result type of a function.

I have tried, and if you declare a variable of type Rawbytestring and put data into it, it will behave just like Ansistring and has the same code page issues:

procedure TForm2.Button2Click(Sender: TObject);
var raw:RawByteString;
begin
     setlength(raw,100);
     Showmessage(format('The code page of this rawbytestring is %d',[stringcodepage(raw)]));
end;

 On my system this rawbytestring has code page 1252! 

9 minutes ago, Anders Melander said:

The only general advice I can give is: Convenience or Performance? Pick one.

 

I won't get into this discussion about which of the many different data types can solve your problem best, because your requirements seems rather fluid and it's impossible to give any targeted advice, when we don't know anything about the specifics.

I'm puzzled why one would even be asking a question like this other than to pass time. If the implementation details are important you shouldn't rely on something someone said on a forum anyway. Try different solutions and benchmark them. If none of the solutions are good enough then there will be something to discuss.

 

Disclaimer: I've been up all night and I'm on my eight cup of coffee.

I sympathize with the OP, I have the same problem often, data streams containing a mixture of text and binary.   

- Many communication components for RS232 (such as good-old Async Pro) have receive events which pass a string. So far I haven't seen any that has the decency to pass a tBytes. 

- With modems (both the old-fashion ones and modern cellular 4G ones) one communicates in the "Hayes" protocol which is text-based until the connection is established, then the data stream becomes binary.

4 minutes ago, Kryvich said:

@A.M. Hoornweg It depends on Windows ANSI codepage. I have CP-1251.

The compiler also issues a lot of warnings:

 

Ansistring28591.dpr

Yes of course the compiler warns. Ansistrings (with the exception of utf8string) can contain only a subset of Unicode so the compiler will warn if you convert between the types.

But the code proves without any doubt that "Binarystring" can be losslessly assigned to Unicodestring and back again and that the ordinal values stay the same.  So for storing bytes (which contain only values 0-255 anyway) that is perfectly safe to use.   

Just don't assign them to any other flavor of Ansistring or things get corrupted.

42 minutes ago, Kryvich said:

We need a real RawByteString without ANY implicit conversions. Just show a compiler error when trying to implicitly convert a string. Or improved TBytes with copy-on-write and all manipulation routines.

 Ansistring (28591) does that, if you convert it to unicode then the ordinal values of the widechars are identical to the ordinal values of the ansichars.

Type Binarystring = type Ansistring (28591);


procedure TForm2.Button1Click(Sender: TObject);
VAR Original, aCopy:Binarystring;
    Ansi:Ansistring;
    Raw:Rawbytestring;
    changed:Boolean;
    Uni:Unicodestring; i:integer;
begin
     SetLength(Original,256);
     for i:=1 to 256 do
         Original[i]:=ansichar(i-1);

     Uni:=Original; //copy to unicode, see if ordinal values are the same

     changed:=False;
     for i:=1 to 256 do
       if Byte(Original[i]) <> WORD(uni[i]) then
           changed:=true;

   If not changed then
      showmessage('It appears that the Widechar ordinal values are THE SAME as the Ansichar values');

    aCopy:=Uni;
    IF aCopy=Original then
         showmessage('It appears that converting back to binarystring is SAFE');

   Ansi:=Uni;
    IF Original<>Ansi then
         showmessage('It appears that converting back to ANSIstring is UNSAFE');

   Raw:=Uni;
    IF Original<>Raw then
         showmessage('It appears that converting back to RAWBYTEstring is UNSAFE');
end;

6 minutes ago, Kryvich said:

@A.M. Hoornweg Why duplicate what the standard library already does for strings, including byte manipulation and copy-on-write?

Because the data is not strings.  But the in-memory representation of a dynamic array is rather similar so the functionality should be straightforward to duplicate.  And thanks to generics, it's probably possible to design the base class to be universal, IIRC a tbytes is just something like "tarray<byte>" (but I may be wrong here).

3 minutes ago, Rollo62 said:

Yes I have that, but the strings are so much more elegant.
I would whish to have better TBytes and dyn. array support like that in Delphi.

Lookup "operator overloading". 

It should be possible to do something like newbytes:=something+somethingelse+'Hello world';    

But adding text to tBytes or searching a text inside a tBytes would mean code page handling.  You could consider defining a Binarystring type (type binarystring=type Ansistring (28591)) because that's least likely to get messed up.

Strings should not be used for byte manipulation (which does not contradict an earlier post of mine, explaining how to get legacy code libraries working if they do this).

Rather, write a new class or class helper that enhances tbytes and offers the functionality that makes strings so practical (insert, delete, append, concatenate, pos). Maybe even some new classes "tBytelist" and "tByteBuilder" as an analog to tStringlist and tStringbuilder.

On 9/22/2020 at 8:46 PM, c0d3r said:

Try deleting one component on an ancient form, causing all sorts of AVs on inherited forms. Turns out you have to open each inherited forms at design time after the deletion, to answer a dump question "if the reference is deleted or renamed"..., then save them.  What were these $&&*%$$ EMBR guys thinking?  The worse part was that some of inherited forms were OK no AVs and some of inherited forms weren't OK and AVs.

 

Imagine if you have hundreds inherited forms....

When I want to safely delete a component on an ancestor form, I do it outside of Delphi.

I use a search tool (Agent Ransack) to find out if the component appears in any descendant forms.   I simply search the folder for all *.dfm files which contain the text  "inherited mycomponentname".  Then I simply drag &drop the resulting list of file names onto an empty Notepad++ window and perform the deletion manually. 

When this is done, it is safe to delete the component on the ancestor form.

2 minutes ago, Anders Melander said:

That is not the purpose of the checksum. The purpose is to guarantee that the output is correct.

Which brings us back to the solution I proposed originally. Ensure that the compressed data is valid (with another checksum) before trying to feed it to the decompressor.