How do I read a C char array into a python bytearray with cython?

Question

I have an array with bytes and its size:

cdef char *bp
cdef size_t size

How do I read the array into a Python bytearray (or another appropriate structure that can easily be pickled)?

Answer 1

Three reasonably straightforward ways to do it:

1. Use the appropriate C API function as I suggested in the comments:

    from cpython.bytes cimport PyBytes_FromStringAndSize
   
    output = PyBytes_FromStringAndSize(bp,size)
   

   This makes a copy, which may be an issue with a sufficiently large string. For Python 2 the functions are similarly named but with PyString rather than PyBytes.

2. View the char pointer with a typed memoryview, get a numpy array from that:

   cdef char[::1] mview = <char[:size:1]>(bp)
   output = np.asarray(mview)
   

   This shouldn't make a copy, so could be more efficient if large.

3. Do the copy manually:

    output = bytearray(size)
    for i in range(size):
        output[i] = bp[i]
   

   (this could be somewhat accelerated with Cython if needed)

---

This issue I think you're having with ctypes (based on the subsequent question you linked to in the comments) is that you cannot pass C pointer to the ctypes Python interface. If you try to pass a char* to a Python function Cython will try to convert it to a string. This fails because it stops at the first 0 element (hence you need size). Therefore you aren't passing ctypes a char*, you're passing it a nonsense Python string.