Cast an array of numbers to array of bytes without re-allocating

[Der schöne Günther 316]

I have some API that allocates and returns a TArray<Word>.

I would then like to process this array with a library that operates exclusively on TArray<Byte>.

 

How can I cast my_word_array to my_byte_array without re-allocating space for a new array and then copying the entire content over? 

 

My only idea so far was a clunky workaround using the absolute directive and manually overwriting the array's length indicator in memory:

procedure changeLengthIndicator(const firstElement; const newLength: NativeInt);
var
    ptr: PNativeInt;
begin
    // Source regarding memory layout:
    // https://docwiki.embarcadero.com/RADStudio/Alexandria/en/Internal_Data_Formats_(Delphi)#Dynamic_Array_Types

    ptr := Addr(firstElement);
    Dec(ptr); //set pointer to length indicator
    ptr^ := newLength;
end;

procedure p();
var
    words: TArray<Word>;
    bytes: TArray<Byte> absolute words;
    byteCount: NativeInt;
begin
    words := [$0FF0, $FEFE, $0000, $0001, $ABAB];

    byteCount := Length(words) * SizeOf(Word);
    changeLengthIndicator(bytes[0], byteCount);

    processBytes(bytes);
end;

Is there a better option?

Unfortunately, there is no making sure no one will use the words array after the call to changeLengthIndicator(..).

Edited April 5, 2022 by Der schöne Günther

[Attila Kovacs 629]

What does setlength do with the casted memory?-

[Der schöne Günther 316]

It will overwrite the second half of the existing data with zeroes because it considers it "newly allocated"

Quote

Existing elements in the array are preserved and newly allocated space is set to 0 or nil.

System.SetLength - RAD Studio API Documentation (embarcadero.com)

Edited April 5, 2022 by Der schöne Günther

[Attila Kovacs 629]

Ok, what does then

SetLength(words, Length(Words)*2);

SetLength(TArray<Byte>(words), Length(Words) div 2);

with the array? 😉

 

Btw, it will also reallocate and copy the memory if needed so forget SetLength().

Does your version leak memory?

 

[Der schöne Günther 316]

47 minutes ago, Attila Kovacs said:

Does your version leak memory?

No, using the absolute variable does not increase the array's reference count.

 

See this snippet: Project1.dpr · GitHub

[Steven Kamradt 20]

I see nothing wrong with using absolute in this case, however you could make it a tiny bit stronger by placing it all in a class and hiding the storage.  Maybe use TList as a starting point, but add accessor methods to fetch/append/insert/delete for each type. Keep a state variable so you know what mode its in and only have to switch modes when you change.  Also, keep counts for each supported type so you can set the length appropriately.  The list would always be an array of integers behind the scenes, but it then could contain an odd number of bytes or words.  A ForEachByte or ForEachWord routine could be coded to use a stack to set and reset the state variable if necessary.

 

I have done something similar with absolute when I needed to compare large blocks of data.  An integer compares or moves at just short of 4x as fast as looping through 4 bytes.  Disclaimer: This doesn't mean you should use this technique because you can. Its good practice to always measure to find the real bottlenecks and not add complexity to optimize when the net result is still nearly identical.  

[Der schöne Günther 316]

54 minutes ago, Steven Kamradt said:

Keep a state variable so you know what mode its in and only have to switch modes when you change.

That's actually an interesting idea. It will store if it's in "byte" or "word" mode, and if you're trying to get the "wrong" data, it will instead throw a runtime exception. That's certainly better than what I have right now.

[Fr0sT.Brutal 900]

Just remember such hacks could break if compiled with RTL that has another layout. Besides this, the solution is quite OK. However, if at least one of 3rd parties you use accepted raw pointers, the issue wouldn't exist at all.

Edited April 7, 2022 by Fr0sT.Brutal

[Der schöne Günther 316]

1 hour ago, Fr0sT.Brutal said:

such hacks could break if compiled with RTL that has another layout

Of course. That's what unit tests are for 😎

 

1 hour ago, Fr0sT.Brutal said:

if at least one of 3rd parties you use accepted raw pointers, the issue wouldn't exist at all

True. Or, even better and less c-like, IEnumerable<Byte>

[Fr0sT.Brutal 900]

15 hours ago, Der schöne Günther said:

Of course. That's what unit tests are for 😎

Fair point

15 hours ago, Der schöne Günther said:

Or, even better and less c-like, IEnumerable<Byte>

I'm a bit old-school. Pointer+length work with almost any language, interfaces are mess (especially Delphi ones)