Passing C++ vector to Numpy through Cython without copying and taking care of memory management automatically

I think @FlorianWeimer's answer provides a decent solution (allocate a vector and pass that into your C++ function) but it should be possible to return a vector from doit and avoid copies by using the move constructor.

from libcpp.vector cimport vectorcdef extern from "<utility>" namespace "std" nogil:  T move[T](T) # don't worry that this doesn't quite match the c++ signaturecdef extern from "fast.h":    vector[int] doit(int length)# define ArrayWrapper as holding in a vectorcdef class ArrayWrapper:    cdef vector[int] vec    cdef Py_ssize_t shape[1]    cdef Py_ssize_t strides[1]    # constructor and destructor are fairly unimportant now since    # vec will be destroyed automatically.    cdef set_data(self, vector[int]& data):       self.vec = move(data)       # @ead suggests `self.vec.swap(data)` instead       # to avoid having to wrap move    # now implement the buffer protocol for the class    # which makes it generally useful to anything that expects an array    def __getbuffer__(self, Py_buffer *buffer, int flags):        # relevant documentation http://cython.readthedocs.io/en/latest/src/userguide/buffer.html#a-matrix-class        cdef Py_ssize_t itemsize = sizeof(self.vec[0])        self.shape[0] = self.vec.size()        self.strides[0] = sizeof(int)        buffer.buf = <char *>&(self.vec[0])        buffer.format = 'i'        buffer.internal = NULL        buffer.itemsize = itemsize        buffer.len = self.v.size() * itemsize   # product(shape) * itemsize        buffer.ndim = 1        buffer.obj = self        buffer.readonly = 0        buffer.shape = self.shape        buffer.strides = self.strides        buffer.suboffsets = NULL

You should then be able to use it as:

cdef vector[int] array = doit(length)cdef ArrayWrapper ww.set_data(array) # "array" itself is invalid from here onnumpy_array = np.asarray(w)

Edit: Cython isn't hugely good with C++ templates - it insists on writing std::move<vector<int>>(...) rather than std::move(...) then letting C++ deduce the types. This sometimes causes problems with std::move. If you're having issues with it then the best solution is usually to tell Cython about only the overloads you want:

 cdef extern from "<utility>" namespace "std" nogil:    vector[int] move(vector[int])

When you return from doit, the WhyNot object goes out of scope, and the array elements are deallocated. This means that &WhyNot[0] is no longer a valid pointer. You need to store the WhyNot object somewhere else, probably in a place provided by the caller.

One way to do this is to split doit into three functions, doit_allocate which allocates the vector and returns a pointer to it, doit as before (but with an argument which receives a pointer to the preallocated vector, anddoit_free` which deallocates the vector.

Something like this:

vector<int> *doit_allocate(){    return new vector<int>;}int *doit(vector<int> *WhyNot, int length){    // Something really heavy    cout << "C++: doing it fast " << endl;     // Heavy stuff - like reading a big file and preprocessing it    for(int i=0; i<length; ++i)        WhyNot->push_back(i); // heavy stuff    cout << "C++: did it really fast" << endl;    return WhyNot->front();}voiddoit_free(vector<int> *WhyNot){    delete WhyNot;}