Accessing the memory of a Python List

[fjames 0]

I was wondering if it was possible to get data out of Delphi and into Python without causing a copy of the data to occur, and vice versa.

 

For instance if I have an array of Doubles, is there a way to get those doubles into a Python list without copying them?

I see in PythonEngine, there is a procedure, TPythonEngine.ArrayToPyList which calls VarRecAsPyObject for each item in the list,

and that will create a new python object for each item, calling PyFloat_FromDouble.

 

Similar for the opposite direction, its going to need to call PyFloat_AsDouble to turn a Python List into an array of Doubles.

 

Is there a more efficient manner, where I can give the pointer of my array to Python?

 

My Goal for doing this, is that I can then hook up PyObject_GetBuffer on TPythonInterface and get the buffer from my Python List

and then I can pass that Buffer off to Numpy, rather than copying the values. 

[David Heffernan 2345]

If you want to work with numpy, and minimise copying, then you create a numpy ndarray array object and use the buffer protocol to gain access to the object's internal buffer. Do all your work in Delphi using that internal buffer. 

[fjames 0]

Thank you for your reply!

 

I guess I should rephrase my question then.

 

How would you get Delphi values (for example an array of Double) into a Python Buffer Object, does this involve just changing the 'void *buf' on the PyBuffer?.

 

9 hours ago, David Heffernan said:

Do all your work in Delphi using that internal buffer. 

This is what I want to be doing. I want to point a Buffer to the memory of a Delphi array, then use the values in that Buffer within Python.

Similarly, I want to be able to create a list (or ndarray) in Python and then by accessing the buffer of this object in Delphi, pull out the values into a Delphi array without copying them.

 

This is what I want to know if is possible. Is there a way to give the Buffer the memory address of the Values in an array so that the buffer uses that memory and now consists of those values?

 

If am I understanding you correctly, you are saying that once I get the internal buffer, I can modify it to point to memory I allocated in Delphi, then in python do something, and then again access the buffer in Delphi to get access to the memory with updated values?

[David Heffernan 2345]

Something can't be both a Delphi array and a numpy array. You should do what in said. Work with a numpy array's buffer. You'll access that as a pointer in Delphi. Use pointer math. 

[Dave Novo 51]

Hi David,

 

Thanks for the responses. The issue is that we already have a few 100 Mb TArray<Double> already allocated on the Delphi side.  We want the user to be able to access that data on the python side without making a copy. ideally, somehow tell the python structure to use our pointer (that we allocated in Delphi) as the source of the data for the numpy array (or matrix). Of course, we would have to tell the user not to modify, or reallocate the contents of that memory, it should be considered read only for them.  Is that even possible? or is our only choice to create numpy ndarray array object and copy our already allocated memory into the buffer exposed by the ndarray object?

[David Heffernan 2345]

I don't think there's an easy way to get numpy to use your raw array memory. My initial suggestion, I suspect, is the only tractable way to do this without copying. 

[Fr0sT.Brutal 900]

You of course can feed Python with the pointer to your Delphi array but what you can do with it depends on what you want Python to do with it. Iа Python has raw arrays, you're lucky.

[David Heffernan 2345]

2 hours ago, Fr0sT.Brutal said:

You of course can feed Python with the pointer to your Delphi array but what you can do with it depends on what you want Python to do with it.

What @fjames wants to do in Python is to use numpy without copying data between Delphi and Python. Are you aware of a way to do this with numpy that I have missed?

[Dave Novo 51]

@Fr0sT.Brutal - do you have an example of how you can feed python with the pointer to the delphi array and then use this data in Python?

[David Heffernan 2345]

2 hours ago, Dave Novo said:

@Fr0sT.Brutal - do you have an example of how you can feed python with the pointer to the delphi array and then use this data in Python?

You want to do more than this don't you. You want to use numpy methods with this shared data. Is that correct? 

[fjames 0]

1 hour ago, David Heffernan said:

@David Heffernan - You want to do more than this don't you. You want to use numpy methods with this shared data. Is that correct? 

Yes, the goal is to be able to operate on this shared data in Python. As well as allocate memory within Python and share that with Delphi.

 

I wrote up a very basic snippet of what I was thinking to hopefully aide in clarifying the goal.

// In Delphi
var
  myArray: array of Integer;
  numpyArray: PPyObject;

// I have an array allocated in Delphi  
// myArray := [1,2,3,4,5,6,7,8,9,10];
numpyArray := CreateNumpyArrayFromArray(myArray); // Trying to create this method

PythonModule.SetVar('InputArray', numpyArray);
PythonEngine.Py_DecRef(numpyArray);

# In Python
import numpy as np
import DelphiModule

# Where this is a numpy array using values from Delphi
inputData = DelphiModule.InputArray

# I use this data to create new data 
# (should be able to use inputData like any 
# other ndArray, just cannot modify)
ResultData = np.square(inputData)

// Back In Delphi
numpyArray := MainModule.GetVar('ResultData');
newArray := GetArrayFromNumpyArray(numpyArray);
// With the goal being that I dont need to copy
// the values from this array, but can now just use them
// newArray = [1,4,9,16,25,36,49,64,81,100]

With the idea being that these arrays are going to be very large and I would like to share multiple, which is why I am trying to avoid copying back and forth.

Edited November 3, 2020 by fjames

[David Heffernan 2345]

It seems odd to me that you won't use the solution that I outlined above which is known to work.

 

But if you won't entertain that then you are probably asking in the wrong place. Because what you are asking is really a numpy question. I'd ask the question on SO and tag it python and numpy. Doesn't matter at all that the array is from Delphi. It's just an array of double. 

[fjames 0]

Thanks for taking the time to try and help.

2 hours ago, David Heffernan said:

It seems odd to me that you won't use the solution that I outlined above which is known to work.

I guess I just don't understand how your original post is a solution to the problem.

 

2 hours ago, David Heffernan said:

It seems odd to me that you won't use the solution that I outlined above which is known to work.

 

Because what you are asking is really a numpy question. I'd ask the question on SO and tag it python and numpy. Doesn't matter at all that the array is from Delphi. It's just an array of double. 

I don't really care about it being numpy, I care about it being Python. For example, I want to be able to do the exact same thing I did in the code snippet above using a Python list.

# Where this is a Python List using values from Delphi
inputData = DelphiModule.InputArray

# I use this data to create new data 
# (should be able to use inputData like any 
# other list, just cannot modify)
ResultData = [x**2 for x in inputData]

So here there is no numpy whatsoever, but my goal is the same, allow python to manipulate memory that was allocated by Delphi, and allow Delphi to manipulate memory that was allocated by python.

 

Right now, it seems like I should be able to create a new Python List as a PPyObject, and then get the Buffer associated with the list (using PyObject_GetBuffer), and then tell the buffer to point to the memory allocated by Delphi for an array. Does this sound possible?

[David Heffernan 2345]

22 minutes ago, fjames said:

Does this sound possible?

No.

 

Python buffer API gives you access to the internal buffer of Python objects. Pass that to Delphi and have your delphi code populate it. 

[fjames 0]

Reading up on Python Buffers, it seems that its possible in  C - 

Quote

Buffer structures (or simply “buffers”) are useful as a way to expose the binary data from another object to the Python programmer. They can also be used as a zero-copy slicing mechanism. Using their ability to reference a block of memory, it is possible to expose any data to the Python programmer quite easily. The memory could be a large, constant array in a C extension, it could be a raw block of memory for manipulation before passing to an operating system library, or it could be used to pass around structured data in its native, in-memory format.

And since you said a Delphi array is:

 

22 hours ago, David Heffernan said:

just an array of double. 

It seems I can give the memory of a Delphi array to a Python Buffer, the way I would give a C array to a Python Buffer, unless I am misunderstanding something.

 

While I could not find an example using python arrays, I found an example using numpy, and according to the post, the numpy array will have access to the values in the C array until the C array is freed (Source).

PyObject *makearray(int array[], size_t size) {
    npy_int dim = size;
    return PyArray_SimpleNewFromData(1, &dim, (void *)array);
}

 

[David Heffernan 2345]

As I read the documentation, I think you are right. Well done. On the home straight now. As I said, this is fundamentally a numpy issue. No doubt a numpy expert would have been aware of this. 

[fjames 0]

1 minute ago, David Heffernan said:

@David Heffernan - As I read the documentation, I think you are right. Well done. On the home straight now. As I said, this is fundamentally a numpy issue. No doubt a numpy expert would have been aware of this. 

Thanks again for your responses. I think I will end up having some questions much more suited for SO as I get more into the nitty gritty.

[Fr0sT.Brutal 900]

On 11/3/2020 at 1:40 PM, David Heffernan said:

Are you aware of a way to do this with numpy that I have missed?

I've no idea about numpy and how it organizes its arrays but making array object point to a buffer coming from Delphi is exactly what I meant

[darnocian 84]

FYI. I recall that you can reference the base ptr on the underlying PyArray. Definitely coming from unmanaged code, you want to try avoid copying if you can.

 

C++ Boost does it quite nicely actually. I last bridged the c++ / python route a number of years ago, so with delphi it will be possible.

 

https://www.boost.org/doc/libs/1_64_0/libs/python/doc/html/numpy/tutorial/ndarray.html

 

https://github.com/boostorg/python/blob/develop/src/numpy/ndarray.cpp

 

inside there is from_data_impl which I think should also be an interesting reference. 

 

[hsauro 40]

On 11/5/2020 at 3:55 AM, Fr0sT.Brutal said:

I've no idea about numpy and how it organizes its arrays but making array object point to a buffer coming from Delphi is exactly what I meant

Did you ever resolve your question? I am also interested in accessing numpy to and from Delphi.

[fjames 0]

19 hours ago, hsauro said:

Did you ever resolve your question? I am also interested in accessing numpy to and from Delphi.

The main points for accessing numpy to and from delphi are that first you need to hook up the numpy c-api methods (similar to how the Python c-api methods are hooked up in MapDLL). This will allow you to create numpy arrays using memory allocated by Delphi.

[PingPIng 23]

https://github.com/Pigrecos/Keras4Delphi/tree/master/src/NumPy

In my partial conversion, there is something about it