How to organize Python modules for PyPI to support 2.x and 3.x

I found that setup.py for httplib2 seems to have an elegant way to support Python 2.x and 3.x. So I decided to copy that method.

The task is to craft a single setup.py for the package distribution that works with all the supported Python distributions. Then with the same setup.py, you can do:

python2 setup.py install

as well as

python3 setup.py install

It should be possible to keep setup.py simple enough to be parsed with all the supported Python distributions. I've successfully done so with a package cobs that supports 2.4 through 2.6 as well as 3.1. That package includes pure Python code (separate code for Python 2.x and 3.x) and C extensions, written separately for 2.x and 3.x.

To do it:

1) I put the Python 2.x code into a python2 subdirectory, and Python 3.x code in a python3 subdirectory.

2) I put the C extension code for 2.x and 3.x in a src directory under python2 and python3.

So, the directory structure is:

root  |  +--python2  |     |  |     +--src  |  +--python3  |     |  |     +--src  |  +--setup.py  +--MANIFEST.in

3) In the setup.py, I had these lines near the top:

if sys.version_info[0] == 2:    base_dir = 'python2'elif sys.version_info[0] == 3:    base_dir = 'python3'

4) In the call to setup, I specified the packages as normal:

setup(    ...    packages=[ 'cobs', 'cobs.cobs', 'cobs.cobsr', ],

5) I specified the base directory for the Python code using a package_dir option (refer to step 3 for base_dir):

    package_dir={        'cobs' : base_dir + '/cobs',    },

6) For the C extensions, I gave the path:

    ext_modules=[        Extension('cobs.cobs._cobs_ext', [ base_dir + '/src/_cobs_ext.c', ]),        Extension('cobs.cobsr._cobsr_ext', [ base_dir + '/src/_cobsr_ext.c', ]),    ],

That was about it for setup.py. The setup.py file is parsable by both Python 2.x and 3.x.

7) Finally, if you build a source distribution using:

python2 setup.py sdist

then it will by default pull in only the files that are specifically needed to build for that Python. E.g. in the above case, you would only get the files under python2 in the source distribution, but not those under python3. But for a complete source distribution, you want to include the files for both 2.x and 3.x. To do that, create a MANIFEST.in file that contains something like this:

include *.txtrecursive-include python2 *recursive-include python3 *

To see what I did, see the cobs source code on PyPI or BitBucket.

The simplest solution is to use a single source distribution.