Running Scipy on Heroku

I managed to get this working on the cedar stack by building numpy and scipy offline as bdists and then modifying the heroku python buildpack to unzip these onto the dyno's vendor/venv areas directly. You can also use the buildpack to set environment variables that persist.

Heroku haven't officially published buildpacks yet - search for 'heroku buildpacks' for more thirdparty/heroku ones and information.

My fork of the python build pack is here:https://wyn@github.com/wyn/heroku-buildpack-python.git

The changes are in the bin/compile where I source two new steps, a scipy/numpy step and an openopt step. The scripts for the two steps are in bin/steps/npscipy and bin/steps/openopt. I also added some variables to bin/release. Note that I am assuming installation through a setup.py file rather than the requirements.txt approach (c.f. https://devcenter.heroku.com/articles/python-pip#traditional_distributions).

I also download the blas/lapack/atlas/gfortran binaries that were used to build numpy/scipy onto the dyno as there are c extensions that need to link through to them. The reason for building everything offline and downloading is that pip-installing numpy/scipy requires you to have a fortran compiler + associated dev environment and this made my slugs too big.

It seems to work, the slug size is now 35mb and scaling seems fast too. All but one of the numpy tests pass and all of the scipy tests pass.

This is still work in progress for me but I thought I'd share.

In case someone else comes to this like I did...

The excellent answer on this question from @coshx sadly no longer works (at least I could not get it to work). What I did, however was the following:

1. I forked npsicpy-binaries repository from @coshx and changed all the tar files such that they do not have venv as the root folder inside (my fork is here)

2. I forked the npsp-helloworld repository from @coshx and made it use a requirements.txt file instead of the setup.py (my fork is here - this means that you can use the whole pip approach).

3. I forked the heroku-buildpack-python repository from Heroku, took the npscipy file from @coshx and changed it to work with this latest version of the build pack (my fork is here - you can see that there is no venv set up, for example).

After doing those three things I have the npsp-helloworld application working perfectly. You just need to make sure you set the buildpack appropriately when creating the application and you are good to go:

$ heroku create --stack=cedar --buildpack=https://github.com/kmp1/heroku-buildpack-python.git

NOTE: I haven't worked out how to make my own binaries yet, so the three libraries (scipy, numpy and scikit-learn) are not the latest versions so make sure when you install it you do (if anyone can build these I would gladly accept a pull request for them):

pip install scipy==0.11.0pip install numpy==1.7.0pip install scikit-learn==0.13.1

By the way - I am really sorry if I have not done things in the correct way, etiquette-wise. I'm still learning git and this whole open source thing.

Another good option is the conda buildpack, which allows you to add any of the free Linux64 packages available through Anaconda/Miniconda to a Heroku app. Some of the most popular packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. While the buildpack makes it fairly simple to add packages to an app, the downsides are that the buildback takes up a lot of space and that you have to wait on Anaconda to update the libraries in the repository.

If you are starting a new Python app on Heroku, you can add the conda buildpack using the command:

$ heroku create YOUR_APP_NAME --buildpack https://github.com/kennethreitz/conda-buildpack.git

If you have already setup a Python app on Heroku, you can add the conda buildpack to the existing app using the command:

$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git

Or, if you need to specify the app by name:

$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git --app YOUR_APP_NAME

To use the buildpack, you will need to include two text files in the app directory, requirements.txt and conda-requirements.txt. Just as with the standard Python buildpack, the requirements.txt file lists packages that should be installed using pip. Packages that should be installed using conda are listed in the conda-requirements.txt file. Some of the most useful scientific packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. The full list of available conda packages can be found at repo.continuum.io.

For example:

$ cat requirements.txtgunicorn==0.14.2requests==0.11.1$ cat conda-requirements.txtscipynumpycvxopt

That’s it! You can now add Anaconda packages to a Python app on Heroku.