Memory exception fault when using PyArg_ParseTuple(y*) with buffer pointer

[dnbif72 0]

Hello,

 

I am getting a memory exception when parsing numpy data buffers passed using the PyArg_ParseTuple() method.  I am using C++Builder in 11.3.  I updated my P4D yesterday, using the auto installer and code from the master repo.  My code on the C++ end is:

PPyObject array1( PPyObject self, PPyObject args )
{
  TPythonEngine * eng = GetPythonEngine();

  unsigned int i1;
  unsigned long num_points;
  double * data;

 

  int ret = eng->PyArg_ParseTuple(args,"y*",&data);

 

  // copy data
  num_points = python_interface::num_points;
  for (i1=0; i1<num_points; i1++)
    python_interface::test_array1[i1] = *(data+i1);

 

  return eng->ReturnNone();
}

 

on the python side I have:

 

import numpy as np
import python_module

num_pts = 10

 

python_module.array_size(num_pts)

 

test_arr1 = np.linspace(1.0,10.0,num_pts,dtype=float)
python_module.array1(test_arr1.data)

 

test_arr2 = np.linspace(2.0,20.0,num_pts,dtype=float)
python_module.array2(test_arr2.data)
 

 

The data is passed fine, and the values are consistent.  The exception occurs when the ReturnNone() call is made.

 

Project p4d.exe raise exception class $C0000005 with message 'access violation at 0x00c1d000: read of address 0x000000b8'.

 

Exception occurs regardless of whether or not data is copied.  

 

The script does not continue pass this point, and returns the exception so the second array is never passed.

 

Any ideas?  Please let me know if you need any other info, and any help would be appreciated.

 

 

Edited October 24, 2023 by dnbif72
typos and added one line

[pyscripter 583]

8 hours ago, dnbif72 said:

double * data;

 

  int ret = eng->PyArg_ParseTuple(args,"y*",&data);

What do you expect to be passed to data?

Please see the docs: Parsing arguments and building values — Python 3.12.0 documentation

 

"y*" is one of the most esoteric formats.  This is the correct way of using it by Victor Stinner one of the python gurus (from this web page).

static PyObject * 
getargs_y_star(PyObject *self, PyObject *args) 
{ 
	Py_buffer buffer; 
	PyObject *bytes; 
	if (!PyArg_ParseTuple(args, "y*", &buffer)) return NULL; 
	bytes = PyBytes_FromStringAndSize(buffer.buf, buffer.len); 
	PyBuffer_Release(&buffer); 
	return bytes; 
}

Note that the Py_buffer structure and related functions are not yet defined in P4D, so you will have to get it from the python headers.

Edited October 24, 2023 by pyscripter

[pyscripter 583]

@dnbif72By the way, since I am not using C++ Builder, I am looking for someone to provide detailed instructions for installing P4D in C++ Builder.    The instructions provide David Intersimone are outdated.   See for example: No Python_D.dproj project file inside python4delphi-master\Packages\Delphi\Delphi 10.4+. · Issue #416 · pyscripter/python4delphi (github.com) 

 

Could you please help?

[dnbif72 0]

Hello,

 

Well somewhere along the line I must of thought I was getting an address to the pointer to the memory block that held the data, and that it would magically cast to what I wanted!  I can see now, that should be a Py_buffer object, which does have the void* buffer at the beginning of the structure so that's why I was seeing and able to copy the data, but I guess it (and or the associated PyObject) needs to be released.  I can copy the structure and likely get that to work, but I would still need to replicate the Py_Buffer_Release() routine.  Unless your seeing something else I am missing here?

 

I believe I tried mixing calls to the python API while using P4D a while back and that was a no go....  Are there plans to implement this functionality in P4D in future?

 

Essentially what I was trying to do was pass a C-style buffer and preferably not have to re-copy the buffer, but the copy is not a huge deal.  Is there another way to go about this that you would recommend?

 

As far as steps for the install, I can help with that, but may need a little time to get back with that.  For the latest install I used the auto installer so that cuts out a lot of the manual compiling of the packages that David was describing, but yes I can write up the steps.

 

Thanks,

Derek

 

 

[pyscripter 583]

51 minutes ago, dnbif72 said:

Well somewhere along the line I must of thought I was getting an address to the pointer to the memory block that held the data, and that it would magically cast to what I wanted!  I can see now, that should be a Py_buffer object, which does have the void* buffer at the beginning of the structure so that's why I was seeing and able to copy the data, but I guess it (and or the associated PyObject) needs to be released.  I can copy the structure and likely get that to work, but I would still need to replicate the Py_Buffer_Release() routine.  Unless your seeing something else I am missing here?

Your code leads to memory corruption.    PyArg_ParseTuple(args,"y*",&data);   will copy the Py_Buffer structure to the address of data overwriting whatever was after that.

If you wait for a few days, I will add the Py_Buffer stuff to P4D and provide an example of getting raw access to nympy.ndarray data.

 

Looking forward to receiving C++ Builder installation instructions!

Edited October 25, 2023 by pyscripter

[pyscripter 583]

@dnbif72 Support for the buffer protocol was added to P4D.

A new demo (Demo35) shows how to access (read and write) numpy arrays super fast using the buffer protocol.