How to configure pip for fast installs of C code libraries

Some old tips for how to get pip to cache wheel files

2019 update: pip has long since done this automatically

Many useful python libraries are wrappers around C code and need to be compiled as part of installation. If you use virtualenvs to isolate the library dependencies of different projects, you might find yourself compiling and installing the same package repeatedly. If you use tox you will be reinstalling whenever tox decides to recreate your virtualenv.

“Wheels” are a recent-ish addition to python packaging which allows for building binary packages that are much faster to install (but they have to be built the first time). Even better, once you’ve built the wheel file you can copy it between machines to save yourself having to build on each machine.

Scipy takes about 30 minutes and 1GB of memory to build on my machine, but only seconds to install from a wheel.

Here is how to set it up correctly and save yourself time.

  1. Set up pip to have a default wheelhouse

Place the following config as /home/$USER/.config/pip/pip.conf:

[global]
find-links = /home/$USER/.pip/wheelhouse/

[install]
use-wheel = yes

[wheel]
wheel-dir = /home/$USER/.pip/wheelhouse/

  1. Make sure you have a recent version of pip (newer than 6.0.0 definitely works).

    • you can use pip to install pip: pip install pip --upgrade

  2. Make sure you have the wheel package installed:

    • pip install wheel

  3. Make sure you replace $USER with your username.

  4. Make sure that /home/$USER/.pip/wheelhouse/ exists

Build some wheels of commonly built packages

For example, to build SQLAlchemy:

pip wheel SQLAlchemy==0.9.8

This will build and compile SQLAlchemy and store the wheel in /home/$USER/.pip/wheelhouse. Next time you install, pip will use the wheel to speed up installation. This will work even if you use pip inside a virtualenv.

When you run pip wheel, pip will recursively build wheels of that package’s dependencies. So if you build a wheel of flask, pip will also build wheels of Werkzeug, Jinja2, itsdangerous and so on.

I wrote this page because I was so annoyed that it took me so long to discover this simple speedup. If there is anything you can think of that I can add, please email me: cal@calpaterson.com