Numpy import fails on multiarray extension library when called from embedded Python within a C++ application
Root Cause
This error occurs because multiarray.cpython-35m-x86_64-linux-gnu.so
module in numpy depends on libpythonx.x.so
, be it is not explicit link the libpythonx.x.so
. So if you use ldd -d multiarray.cpython-35m-x86_64-linux-gnu.so
you will not see the python in the list.
Python doesn't have issue because python binary depends on libpython.x.x.so
, so when numpy load multiarray.cpython-35m-x86_64-linux-gnu.so
by using dlopen
. libdl.so
will try to resolve the undefined symbols by checking the dependent shared library of the main program which is python. It will find it in libpython.x.x.so
.
Solution
After knowing the root cause the solution is very easy, just help libdl.so
to be able to find libpython.x.x.so
. There are at least two way to achieve that:
- Use
dlopen("libpythonx.x.so", RTLD_GLOBAL)
. After open this so useRTLD_GLOBAL
flag, it make symbol in libpythonx.x.so available for symbol resolution of subsequently loaded shared objects. - In main program which embed python, add the
libpythonx.x.so
into its dependency library.
I had a similar error with linking an application against a libpython3.5m.a (archive, not dynamic). Once it loaded something like multiarray.cpython-35m-x86_64-linux-gnu.so
, it would expect symbols like PyFloat_Type
to exist.
In diagnosing why Python could be called directly and it would work, but my application would not, I noticed thatreadelf -s myapplication
had a PyFloat_Type symbol in the .symtab
table but not in the .dynsym
table.
However, readelf -s /asb/path/to/python3
had a PyFloat_Type symbol in both tables.
Adding:target_link_options(myapplication PUBLIC "LINKER:-export-dynamic")
in CMake ensured that the symbols needed were also available in the .dynsym
table. After this, it the application worked correctly.