İs possible same pointer size for Win32/Win64?

[kosovali 0]

Hi,

 

I have 1000+ records.  This records contains pointer types. When records streamed on Win32 /Win64 has a different sizes. I want to use this records with win32/win64. 

In Win32 Pointer size is 4, can be this size 8 bytes?

Or any other solutions? 

 

type
    ptr_rec1=^rec1;
    rec1=record
     i1:integer;
     p1:Pointer;
    end;
   
    ptr_rec2=record
      rec:ptr_rec1;
      i:integer;
      p2:Pointer;
    end;
    
    SizeOf(ptr_rec2); //Different sizes for Win32/Win64

 

Edited January 5 by kosovali

[Der schöne Günther 316]

Isn't that like ... the whole point of Win32/Win64? 🤔

 

Recommended read:

Embarcadero: 64-bit Windows Data Types Compared to 32-bit Windows Data Types

Edited January 5 by Der schöne Günther

[Lars Fosdal 1792]

57 minutes ago, kosovali said:

When records streamed on Win32 /Win64 has a different sizes

Why would you stream the pointers? They only have any meaning inside each application as a reference to a memory position.

[David Heffernan 2347]

1 hour ago, kosovali said:

Or any other solutions? 

But what is the problem? You can't solve something until you know what the problem is.

 

As Lars says, streaming pointers seems, er, pointless.

[kosovali 0]

16 minutes ago, Lars Fosdal said:

Why would you stream the pointers? They only have any meaning inside each application as a reference to a memory position.

Pointer fields updated in runtime.

[kosovali 0]

1 minute ago, David Heffernan said:

But what is the problem? You can't solve something until you know what the problem is.

 

As Lars says, streaming pointers seems, er, pointless.

Pointer fields updated in runtime. 30 years old codebase. Not created now.  It is developed from pascal ide.

[Kas Ob. 121]

13 minutes ago, kosovali said:

Pointer fields updated in runtime. 30 years old codebase. Not created now.  It is developed from pascal ide.

This means the size of memory being used is very limited, or somehow limited and contained.

In this case i would suggest to start to adjust the old code first to maintain huge block of memory enough for these structures, then replace all the pointers with relative address pointers, by that i mean all will point to SOME_ADDR + BASE_ADDRESS, as such the address pointers could be only 32bit, while BASE_ADDRESS is irrelevant and will be applied once by the code after allocating the memory (aka after loading or transferring the whole block).

This will allow you to compress the used memory, but under condition that you don't use the default memory manager and allocate these record/structures on you own.

But again, before streamed/serialized you could transfer the structures and replace all the pointers with relative ones to the base, which be default could be 0. 

[DelphiUdIT 178]

1 hour ago, kosovali said:

In Win32 Pointer size is 4, can be this size 8 bytes?

There is not only a problem of the length of the "pointers" but also of the arrangement of the data within the record.
The compiler "fills" the record structure with zero bytes to align the data to the defined alignment (which can be the standard one or one defined in that section of code).
Therefore, the transmission of that data (whatever it is) is still at risk even if it were performed within programs made with Delphi. A change of alignment (even the standard one) for example between different platforms or different compiler releases could lead to different results.

[Lars Fosdal 1792]

Does the stream consist only of ptr_rec2 elements? Is it simply a double linked list?

Is the element sequence in the stream of importance?

Are the pointers relative positions in the stream or are they original memory positions in the old win32 app and discarded/replaced when loading in the receiving app?

Do you need to keep it in the same linked list format in the new 64-bit app?

 

Both ptr_rec1 and pointer are 8 bytes in 64-bit - so to read the stream in 64-bit, you would have to replace the pointer types with a 32-bit variable

type
    Fake32bitPtr = UInt32;
    ptr_rec1=Fake32bitPtr;
    rec1=record
     i1:integer;
     p1:Fake32bitPtr;
    end;
   
    ActualPtr_rec2 = ^ptr_rec2;
    ptr_rec2=record
      rec:ptr_rec1;
      i:integer;
      p2:Fake32bitPtr;
    end;

If the stream is not incredibly large, you could create a shadow structure that basically is an array of ActualPtr_rec2 and point to the ptr_rec2 elements without caring about the pointers?

 

 

 

 

 

[kosovali 0]

14 minutes ago, DelphiUdIT said:

There is not only a problem of the length of the "pointers" but also of the arrangement of the data within the record.
The compiler "fills" the record structure with zero bytes to align the data to the defined alignment (which can be the standard one or one defined in that section of code).
Therefore, the transmission of that data (whatever it is) is still at risk even if it were performed within programs made with Delphi. A change of alignment (even the standard one) for example between different platforms or different compiler releases could lead to different results.

Used packed record.

[Cristian Peța 103]

Do you have thousand record types? Then you have some work...

 

If there are not so many record types and you need to use both 32 and 64 bit versions then I would make for every record an old version that has Cardinal instead of pointer just for reading from and writing to the stream.

And two procedures to copy data between old and new record.

The old record with Cardinal will be used only to stream data.

 

If you want to do more then use old records with Cardinal instead of pointer only for reading. And save data in a new format like JSON.

Edited January 5 by Cristian Peța

[kosovali 0]

26 minutes ago, Lars Fosdal said:

Does the stream consist only of ptr_rec2 elements? Is it simply a double linked list?

Is the element sequence in the stream of importance?

Are the pointers relative positions in the stream or are they original memory positions in the old win32 app and discarded/replaced when loading in the receiving app?

Do you need to keep it in the same linked list format in the new 64-bit app?

 

Both ptr_rec1 and pointer are 8 bytes in 64-bit - so to read the stream in 64-bit, you would have to replace the pointer types with a 32-bit variable

type
    Fake32bitPtr = UInt32;
    ptr_rec1=Fake32bitPtr;
    rec1=record
     i1:integer;
     p1:Fake32bitPtr;
    end;
   
    ActualPtr_rec2 = ^ptr_rec2;
    ptr_rec2=record
      rec:ptr_rec1;
      i:integer;
      p2:Fake32bitPtr;
    end;

If the stream is not incredibly large, you could create a shadow structure that basically is an array of ActualPtr_rec2 and point to the ptr_rec2 elements without caring about the pointers?

 

 

Records contains; Record Pointer, Double Link List Pointer. etc.

Element sequence is important.

Pointer values not important for streaming. Only used runtime. 

We want to same format in the 64 bit app.

Fake32bitPtr can not be used with GetMem. Error: "Incompatible Types".

I can change -minimal- record fields for 64 bit compatibility. 

 

Code changed for Win32/Win64 com.

type
    Fake64bitPtr = UInt64;
    ptr_rec1=Fake64bitPtr;
    rec1=record
     i1:integer;
     p1:Fake64bitPtr;
    end;
   
    ActualPtr_rec2 = ^ptr_rec2;
    ptr_rec2=record
      rec:ptr_rec1;
      i:integer;
      p2:Fake64bitPtr;
    end;
    
    

But GetMem raised error.

[Lars Fosdal 1792]

I think you missed the point of my suggested changes. The point was only to be able to read the 32-bit stream.

I think you will need to rewrite the code that loads the data from the old-32 bit structures.

I don't see a clean easy way to avoid that, f.x. by using my example record struct to read the old stream, and then move it to a 64-bit version with actual pointers..

 

You say Element sequence is important - is the order of the loaded double linked list supposed to be different from the streamed order?

 

I have a feeling that your records are more complex than your example, and I certainly don't understand the reason for the untyped pointer.

Are the actual records fixed in size or do they vary in size?

 

Are you writing new 32-bit and 64-bit apps, so this is not a transition thing - but a need for a future shared format?

In that case, I would rethink the structure and replace the pointers with stream offsets.

[Remy Lebeau 1403]

You should NEVER stream pointers from one process' address space into another process' address space, unless both processes are running concurrently and one process needs to directly access the other process's memory via the ReadProcessMemory() and/or WriteProcessMemory() APIs (using shared memory would be better, though).

 

Otherwise, just don't do it!  Stream offsets instead.  And in the example given, a linked list can certainly be streamed using offsets instead of pointers.  The actual pointers would only be meaningful in the memory of the process that originally creates the list, and in the process that loads the stream into a new list in it own memory.  Pointers are fairly meaningless when passed around from one process to another.

[Kas Ob. 121]

10 hours ago, Remy Lebeau said:

Stream offsets instead

Thank you !

That is the word i was looking for.

[David Heffernan 2347]

Streaming offsets doesn't seem helpful. The user isn't interested in streaming addresses. The addresses are initialised at runtime. They are only streamed because the code is, er, sloppy, and blits raw records rather than serialising. 

[Lars Fosdal 1792]

Offsets may be helpful if the streamed order is not the same as the desired order, such as for a tree structure. Personally, I'd opt for identities and rebuild the structure after loading the stream. But, since OP doesn't share sufficient info about content and structure, this is all speculation.

[David Heffernan 2347]

2 hours ago, Lars Fosdal said:

Offsets may be helpful if the streamed order is not the same as the desired order, such as for a tree structure. Personally, I'd opt for identities and rebuild the structure after loading the stream. But, since OP doesn't share sufficient info about content and structure, this is all speculation.

What would offsets be helpful for given that the data streamed in those fields is never used

[darnocian 87]

2 hours ago, David Heffernan said:

What would offsets be helpful for given that the data streamed in those fields is never used

Say in serialisation, you construct offsets relative to a certain address (say the start of the first record), and in deserialisation in another process, you convert back to pointers again. The key to it being useful is the layout and relative addressing.

[David Heffernan 2347]

12 hours ago, darnocian said:

Say in serialisation, you construct offsets relative to a certain address (say the start of the first record), and in deserialisation in another process, you convert back to pointers again. The key to it being useful is the layout and relative addressing.

I'm not talking about generalities. I'm talking about this specific topic. The OP already said that the pointer values that are streamed are ignored.

 

The problem at hand is that the code does naive blitting of internal records, and these records have different layouts for different targets.

[Stefan Glienke 2009]

Just store Pointer as UInt64 all the time