Speedup virtualenv creation with numpy and pandas
You could make use of the wheel package. We do this over at pandas for our continuous integration builds so that we can basically download them and install them extremely fast.
Take a look at ci/speedpack/build.sh. This script essentially builds a bunch of wheels that we use (numpy and scipy included) for CI. They are actually stored on server and then pulled from there when travis-ci runs.
Take a look at ci/install.sh to see how the installation process works.
In your situation a server might be overkill, but you could setup a local repo and install wheels from there.
Old question, but thought I could add some recent insight. I work on both OSX and Windows on the same project. I have had numerous delays (mostly just getting the correct files etc) on Windows side in trying to get binary installs for numpy etc. Have switched to using Anaconda Distribution recently and it does a wonderful job of simplifying life. It has its own flavour of virtual environments baked in and simplifies life considerably.