How to build and distribute a Python/Cython package that depends on third party libFoo.so

As you probably know, the recommended way of distributing a Python module with compiled components is to use the wheel format. There doesn't appear to be any standard cross-platform way of bundling third-party native libraries into a wheel. However, there are platform-specific tools for this purpose.

On Linux, use auditwheel.

auditwheel modifies an existing Linux wheel file to add any third-party libraries which are not included in the basic "manylinux" standard. Here's an walkthrough of how to use it with your project on a clean install of Ubuntu 17.10:

First, install basic Python development tools, and the third-party library with its headers:

root@ubuntu-17:~# apt-get install cython python-pip unziproot@ubuntu-17:~# apt-get install libsundials-serial-dev

Then build your project into a wheel file:

root@ubuntu-17:~# cd cython-example/root@ubuntu-17:~/cython-example# python setup.py bdist_wheel[...]root@ubuntu-17:~/cython-example# cd dist/root@ubuntu-17:~/cython-example/dist# lltotal 80drwxr-xr-x 2 root root  4096 Nov  8 11:28 ./drwxr-xr-x 7 root root  4096 Nov  8 11:28 ../-rw-r--r-- 1 root root 70135 Nov  8 11:28 poc-0.0.0-cp27-cp27mu-linux_x86_64.whlroot@ubuntu-17:~/cython-example/dist# unzip -l poc-0.0.0-cp27-cp27mu-linux_x86_64.whlArchive:  poc-0.0.0-cp27-cp27mu-linux_x86_64.whl  Length      Date    Time    Name---------  ---------- -----   ----    62440  2017-11-08 11:28   poc/do_stuff.so        2  2017-11-08 11:28   poc/__init__.py   116648  2017-11-08 11:28   poc/cython_extensions/helloworld.so        2  2017-11-08 11:28   poc/cython_extensions/__init__.py       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt      105  2017-11-08 11:28   poc-0.0.0.dist-info/WHEEL      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA      793  2017-11-08 11:28   poc-0.0.0.dist-info/RECORD---------                     -------   180382                     10 files

The wheel file can now be installed locally and tested:

root@ubuntu-17:~/cython-example/dist# pip install poc-0.0.0-cp27-cp27mu-linux_x86_64.whl[...]root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"hello cython0.841470984808trying to load the sundials program3-species kinetics problemAt t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02    rootsfound[] =   0   1At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02[...]

Now we install the auditwheel tool. It requires Python 3, but it's capable of processing wheels for Python 2 or 3.

root@ubuntu-17:~/cython-example/dist# apt-get install python3-piproot@ubuntu-17:~/cython-example/dist# pip3 install auditwheel

auditwheel uses another tool called patchelf to do its job. Unfortunately, the version of patchelf included with Ubuntu 17.10 is missing a bugfix without which auditwheel will not work. So we'll have to build it from source (script taken from the manylinux Docker image):

root@ubuntu-17:~# apt-get install autoconfroot@ubuntu-17:~# PATCHELF_VERSION=6bfcafbba8d89e44f9ac9582493b4f27d9d8c369root@ubuntu-17:~# curl -sL -o patchelf.tar.gz https://github.com/NixOS/patchelf/archive/$PATCHELF_VERSION.tar.gzroot@ubuntu-17:~# tar -xzf patchelf.tar.gzroot@ubuntu-17:~# (cd patchelf-$PATCHELF_VERSION && ./bootstrap.sh && ./configure && make && make install)

Now we can check which third-party libraries the wheel requires:

root@ubuntu-17:~/cython-example/dist# auditwheel show poc-0.0.0-cp27-cp27mu-linux_x86_64.whlpoc-0.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with thefollowing platform tag: "linux_x86_64".The wheel references external versioned symbols in these system-provided shared libraries: libc.so.6 with versions {'GLIBC_2.4','GLIBC_2.2.5', 'GLIBC_2.3.4'}The following external shared libraries are required by the wheel:{    "libblas.so.3": "/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1",    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.26.so",    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",    "libgfortran.so.4": "/usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0",    "liblapack.so.3": "/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1",    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.26.so",    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.26.so",    "libquadmath.so.0": "/usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0",    "libsundials_cvodes.so.2": "/usr/lib/libsundials_cvodes.so.2.0.0",    "libsundials_nvecserial.so.0": "/usr/lib/libsundials_nvecserial.so.0.0.2"}In order to achieve the tag platform tag "manylinux1_x86_64" thefollowing shared library dependencies will need to be eliminated:libblas.so.3, libgfortran.so.4, liblapack.so.3, libquadmath.so.0,libsundials_cvodes.so.2, libsundials_nvecserial.so.0

And create a new wheel which bundles them:

root@ubuntu-17:~/cython-example/dist# auditwheel repair poc-0.0.0-cp27-cp27mu-linux_x86_64.whlRepairing poc-0.0.0-cp27-cp27mu-linux_x86_64.whlGrafting: /usr/lib/libsundials_nvecserial.so.0.0.2 -> poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2Grafting: /usr/lib/libsundials_cvodes.so.2.0.0 -> poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0Grafting: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 -> poc/.libs/liblapack-549933c4.so.3.7.1Grafting: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 -> poc/.libs/libblas-52fa99c8.so.3.7.1Grafting: /usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0 -> poc/.libs/libgfortran-2df4b07d.so.4.0.0Grafting: /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0 -> poc/.libs/libquadmath-0d7c3070.so.0.0.0Setting RPATH: poc/cython_extensions/helloworld.so to "$ORIGIN/../.libs"Previous filename tags: linux_x86_64New filename tags: manylinux1_x86_64Previous WHEEL info tags: cp27-cp27mu-linux_x86_64New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64Fixed-up wheel written to /root/cython-example/dist/wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whlroot@ubuntu-17:~/cython-example/dist# unzip -l wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whlArchive:  wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl  Length      Date    Time    Name---------  ---------- -----   ----      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json     1400  2017-11-08 12:08   poc-0.0.0.dist-info/RECORD      110  2017-11-08 12:08   poc-0.0.0.dist-info/WHEEL    62440  2017-11-08 11:28   poc/do_stuff.so        2  2017-11-08 11:28   poc/__init__.py   131712  2017-11-08 12:08   poc/cython_extensions/helloworld.so        2  2017-11-08 11:28   poc/cython_extensions/__init__.py   230744  2017-11-08 12:08   poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0  7005072  2017-11-08 12:08   poc/.libs/liblapack-549933c4.so.3.7.1   264024  2017-11-08 12:08   poc/.libs/libquadmath-0d7c3070.so.0.0.0  2039960  2017-11-08 12:08   poc/.libs/libgfortran-2df4b07d.so.4.0.0    17736  2017-11-08 12:08   poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2   452432  2017-11-08 12:08   poc/.libs/libblas-52fa99c8.so.3.7.1---------                     ------- 10206026                     16 files

If we uninstall the third-party libraries, the previously-installed wheel will stop working:

root@ubuntu-17:~/cython-example/dist# apt-get remove libsundials-serial-dev && apt-get autoremove[...]root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"Traceback (most recent call last):  File "<string>", line 1, in <module>  File "poc/do_stuff.pyx", line 1, in init poc.do_stuffImportError: libsundials_cvodes.so.2: cannot open shared object file: No such file or directory

But the wheel with the bundled libraries will work fine:

root@ubuntu-17:~/cython-example/dist# pip uninstall poc[...]root@ubuntu-17:~/cython-example/dist# pip install wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl[...]root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"hello cython0.841470984808trying to load the sundials program3-species kinetics problemAt t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02    rootsfound[] =   0   1At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02[...]

On OSX, use delocate.

delocate for OSX apparently works very similarly to auditwheel. Unfortunately I don't have an OSX machine available to provide a walkthrough.

Combined example:

One project which uses both tools is SciPy. This repository, despite its name, contains the official SciPy build process for all platforms, not just Mac. Specifically, compare the Linux build script (which uses auditwheel), with the OSX build script (which uses delocate).

To see the result of this process, you might want to download and unzip some of the SciPy wheels from PyPI. For example, scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl contains the following:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

While scipy-1.0.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl contains this:

   273072  2017-10-25 07:03   scipy/.dylibs/libgcc_s.1.dylib  1550456  2017-10-25 07:03   scipy/.dylibs/libgfortran.3.dylib   279932  2017-10-25 07:03   scipy/.dylibs/libquadmath.0.dylib

To enhance mhsmith's excellent answer, here are the steps performed on MacOS with delocate:

1. Install sundials, for example with Homebrew:

   $ brew install sundials

2. Build the package:

   $ python setup.py bdist_wheel

3. The pendants to auditwheel show/auditwheel repair are delocate-listdeps/delocate-wheel, so first analyze the resulting wheel file:

   $ delocate-listdeps --all dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl/usr/lib/libSystem.B.dylib/usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib/usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib

4. Fixing the wheel file:

   $ delocate-wheel -v -w dist_fixed dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl Fixing: dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whlCopied to package .dylibs directory:  /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib  /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib

In the dist_fixed directory, you will have the bundled wheel. You will notice the size difference:

$ ls -l dist/ dist_fixed/dist/:total 72-rw-r--r--  1 hoefling  wheel  36030 10 Nov 20:25 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whldist_fixed/:total 240-rw-r--r--  1 hoefling  wheel  120101 10 Nov 20:34 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

If you list deps for the bundled wheel, you will notice the needed libraries are now bundled (indicated by prefix @loader_path):

$ delocate-listdeps --all dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl /usr/lib/libSystem.B.dylib@loader_path/../.dylibs/libsundials_cvodes.2.9.0.dylib@loader_path/../.dylibs/libsundials_nvecserial.2.7.0.dylib

Installing the bundled wheel (notice the bundled libs are installed correctly):

$ pip install dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl Processing ./dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whlInstalling collected packages: pocSuccessfully installed poc-0.0.0$ pip show -f pocName: pocVersion: 0.0.0Summary: UNKNOWNHome-page: UNKNOWNAuthor: UNKNOWNAuthor-email: UNKNOWNLicense: UNKNOWNLocation: /Users/hoefling/.virtualenvs/stackoverflow-py27/lib/python2.7/site-packagesRequires: Files:  poc-0.0.0.dist-info/DESCRIPTION.rst  poc-0.0.0.dist-info/INSTALLER  poc-0.0.0.dist-info/METADATA  poc-0.0.0.dist-info/RECORD  poc-0.0.0.dist-info/WHEEL  poc-0.0.0.dist-info/metadata.json  poc-0.0.0.dist-info/top_level.txt  poc/.dylibs/libsundials_cvodes.2.9.0.dylib  poc/.dylibs/libsundials_nvecserial.2.7.0.dylib  poc/__init__.py  poc/__init__.pyc  poc/cython_extensions/__init__.py  poc/cython_extensions/__init__.pyc  poc/cython_extensions/helloworld.so  poc/do_stuff.so

I'd suggest to take a completely different approach. Set up a Linux package management infrastructure. On Ubuntu/Debian this could be done with reprepro. https://wiki.ubuntuusers.de/reprepro/ could be a start, but there are much more tutorials available. You could then build your own Linux package distributing your libraries and all necessary files together with your Python application.

This would be a very clean and convenient approach for your clients. Especially regarding updates. (You even can address different OS releases as necessary at the same time.)

As always a clean approach comes with a cost. This clean approach takes quite some effort for you to implement. You need to not only set up a server - that's the easier part - but get into how to build packages - which is not difficult but you'll be required to read a bit how to do that and do quite a bit of experimentation to end up with packages being exactly as you want them. However everything WILL be the way you want it then. And future updates are really easy for you as well as your client machines.

I'd recommend that approach iff you want to simplify updates in the future, want to learn about Linux and might have requirements for own packages in the future. Or a large amount of clients.

That about a very "high level" approach. In contrast very "low level" approach would be the following one:

• Check the presence of your libraries on startup of your program
• If not present: Terminate the application. Print a text that refers to a script how to install the necessary libraries. That could even be an URL where to download the script, f.e. with:

bash <(curl -s http://mywebsite.com/myscript.txt)