packaging with numpy and test suite

Here is a setup.py that works for me:

# pkg - A fancy software package# Copyright (C) 2013  author (email)## This program is free software: you can redistribute it and/or modify# it under the terms of the GNU General Public License as published by# the Free Software Foundation, either version 3 of the License, or# (at your option) any later version.## This program is distributed in the hope that it will be useful,# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the# GNU General Public License for more details.## You should have received a copy of the GNU General Public License# along with this program.  If not, see http://www.gnu.org/licenses/gpl.html."""pkg: a software suite for Hey look at me I'm a long descriptionBut how long am I?"""from __future__ import division, print_function#ideas for setup/f2py came from:#    -numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07#    -winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07#    -needing to use #        import setuptools; from numpy.distutils.core import setup, Extension: #        http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07#    -wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07#    -numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07#    -manifest files in disutils: #        'distutils doesn't properly update MANIFEST. when the contents of directories change.'#        https://github.com/numpy/numpy/blob/master/setup.py         #    -if things are not woring try deleting build, sdist, egg directories  and try again: #        https://stackoverflow.com/a/9982133/2530083 2013-11-07#    -getting fortran extensions to be installed in their appropriate sub package#        i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])" #        Note that sources is a list even if one file: #        http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07#    -install fortran source files into their appropriate sub-package #        i.e. "package_data={'': ['*.f95','*.f90']}# Note it's a dict and list":#        https://stackoverflow.com/a/19373744/2530083 2013-11-07#    -Chapter 9 Fortran Programming with NumPy Arrays: #        Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.#    -Hitchhikers guide to packaging :#        http://guide.python-distribute.org/#    -Python Packaging: Hate, hate, hate everywhere : #        http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/#    -How To Package Your Python Code: #        http://www.scotttorborg.com/python-packaging/#    -install testing requirements: #        https://stackoverflow.com/a/7747140/2530083 2013-11-07import setuptoolsfrom numpy.distutils.core import setup, Extensionimport osimport os.path as ospdef readme(filename='README.rst'):    with open('README.rst') as f:        text=f.read()    f.close()    return textdef get_package_data(name, extlist):    """Return data files for package *name* with extensions in *extlist*"""    #modified slightly from taken from http://code.google.com/p/winpython/source/browse/setup.py 2013-11-7    flist = []    # Workaround to replace os.path.relpath (not available until Python 2.6):    offset = len(name)+len(os.pathsep)    for dirpath, _dirnames, filenames in os.walk(name):        for fname in filenames:                        if not fname.startswith('.') and osp.splitext(fname)[1] in extlist:#                flist.append(osp.join(dirpath, fname[offset:]))                flist.append(osp.join(dirpath, fname))    return flistDOCLINES = __doc__.split("\n")CLASSIFIERS = """\Development Status :: 1 - PlanningLicense :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)Programming Language :: Python :: 2.7Topic :: Scientific/Engineering"""NAME = 'pkg'MAINTAINER = "me"MAINTAINER_EMAIL = "me@me.com"DESCRIPTION = DOCLINES[0]LONG_DESCRIPTION = "\n".join(DOCLINES[2:])#readme('readme.rst')URL = "http://meeeee.mmemem"DOWNLOAD_URL = "https://github.com/rtrwalker/geotecha.git"LICENSE = 'GNU General Public License v3 or later (GPLv3+)'CLASSIFIERS = [_f for _f in CLASSIFIERS.split('\n') if _f]KEYWORDS=''AUTHOR = "me"AUTHOR_EMAIL = "me.com"PLATFORMS = ["Windows"]#, "Linux", "Solaris", "Mac OS-X", "Unix"]MAJOR = 0MINOR = 1MICRO = 0ISRELEASED = FalseVERSION = '%d.%d.%d' % (MAJOR, MINOR, MICRO)INSTALL_REQUIRES=[]ZIP_SAFE=FalseTEST_SUITE='nose.collector'TESTS_REQUIRE=['nose']DATA_FILES = [(NAME, ['LICENSE.txt','README.rst'])]PACKAGES=setuptools.find_packages()PACKAGES.remove('tools')PACKAGE_DATA={'': ['*.f95','*f90']}               ext_files = get_package_data(NAME,['.f90', '.f95','.F90', '.F95'])ext_module_names = ['.'.join(osp.splitext(v)[0].split(osp.sep)) for v in ext_files]EXT_MODULES = [Extension(name=x,sources=[y]) for x, y in zip(ext_module_names, ext_files)]      setup(    name=NAME,    version=VERSION,    maintainer=MAINTAINER,    maintainer_email=MAINTAINER_EMAIL,    description=DESCRIPTION,    long_description=LONG_DESCRIPTION,    url=URL,    download_url=DOWNLOAD_URL,    license=LICENSE,    classifiers=CLASSIFIERS,    author=AUTHOR,    author_email=AUTHOR_EMAIL,    platforms=PLATFORMS,    packages=PACKAGES,    data_files=DATA_FILES,    install_requires=INSTALL_REQUIRES,    zip_safe=ZIP_SAFE,    test_suite=TEST_SUITE,    tests_require=TESTS_REQUIRE,    package_data=PACKAGE_DATA,        ext_modules=EXT_MODULES,    )

To install, at the command line I use:

python setup.py installpython setup.py clean --all

The only issue I seem to have is a minor one. when I look in site-packages for my package it is installed inside the egg folder C:\Python27\Lib\site-packages\pkg-0.1.0-py2.7-win32.egg\pkg. Most other packages I see there have a C:\Python27\Lib\site-packages\pkg folder separate to the egg folder. Does anyone know how to get that separation?

As for testing, after installing, I type the following at the command line:

nosetests package_name -v

Try investigating python setup.py develop (Python setup.py develop vs install) for not having to install the package after every change.

As I commented in the code I found the following useful:

• numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
• winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
• needing to use import setuptools; from numpy.distutils.core import setup, Extension: http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
• wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
• numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
• manifest files in disutils: 'distutils doesn't properly update MANIFEST. when the contents of directories change.' https://github.com/numpy/numpy/blob/master/setup.py
• if things are not woring try deleting build, sdist, egg directories and try again: https://stackoverflow.com/a/9982133/2530083 2013-11-07
• getting fortran extensions to be installed in their appropriate sub package i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])" Note that sources is a list even if one file: http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
• install fortran source files into their appropriate sub-package i.e. "package_data={'': ['.f95','.f90']}# Note it's a dict and list": https://stackoverflow.com/a/19373744/2530083 2013-11-07
• Chapter 9 Fortran Programming with NumPy Arrays: Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
• Hitchhikers guide to packaging : http://guide.python-distribute.org/
• Python Packaging: Hate, hate, hate everywhere : http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
• How To Package Your Python Code: http://www.scotttorborg.com/python-packaging/
• install testing requirements: https://stackoverflow.com/a/7747140/2530083 2013-11-07
• 'python setup.py develop' : https://stackoverflow.com/a/19048754/2530083

Here is setup.py from a project I made. I have found figuring out setup.py / packaging to be frustrating with no solid answers and definitely not pythonic in the sense of having one and only one obvious way to do something. Hopefully this will help a little.

The points you may find useful are:

• find_packages which removes the drudgery of including lots of files or messing around with generating manifest.
• package_data which allows you to easily specify non .py files to be included
• install_requires / tests_require

You'll need to find the source for distribute_setup.py if you don't have it already.

• Is it really necessary to add the path to every single source file of the extension? (I.e. volterra/integral.f90) Can't I give a parameter which says, look for stuff in volterra/? The top_path, and package_dir parameters didn't do the trick.
• Currently, the init.py file is not included in the build. Why is that?

Hopefully find_packages() will solve both of those. I don't have much experience packaging but I haven't had to go back to manual inclusion yet.

• How can I run my tests in this setup?

I think this is probably a different question with many answers depending on how you are doing tests. Maybe you can ask it separately?

As a side note, I am under the impression that the standard is to put your tests directory at the top level. I.e. volterra/volterra and volterra/tests.

• What's the best workflow for doing development in such an environment? I don't want to install my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?

This might be worth another question as well. I don't see why you would need to install your package for every single change. If you are uploading the package, just don't install it on your dev system (except to test installation) and work directly from your development copy. Maybe I'm missing something though since I don't work with compiled extensions.

Here is the example

try:    from setuptools import setup, find_packagesexcept ImportError:    from distribute_setup import use_setuptools    use_setuptools()    from setuptools import setup, find_packagessetup(    # ... other stuff    py_modules=['distribute_setup'],    packages=find_packages(),    package_data={'': ['*.png']},  # for me to include anything with png    install_requires=['numpy', 'treenode', 'investigators'],    tests_require=['mock', 'numpy', 'treenode', 'investigators'],)