Painlessly developing Python on NixOS with pipenv

For a long time, I’ve wanted to develop Python code on NixOS, but using Nix to manage dependencies was a major pain. If the dependencies you want are already in Nixpkgs, then you’re good, but otherwise you need to use things like pypi2nix to turn Pypi packages into nix derivations. This never quite worked out for me, so I ended up writing the nix derivations myself… which sucked.

I’ve been aware of pipenv for a while now, and it seemed like the ideal solution. It gives Nix/Stack/Yarn-like reproducibility, while still being actively maintained, unlike some of these Nix-specific Python package tools. I had never managed to get this to work for me, but I’ve finally got a configuration working on Nix.

First attempt

My original instinct was to throw together a default.nix file:

with import <nixpkgs> {};
mkDerivation {
  name = "my-python-project";
  buildInputs = [ python36 pipenv ];
}

and just open nix-shell and run all my pipenv commands from there.

The problem

Unfortunately, whenever I tried to install a package that required compiling lots of C source (like numpy or pandas), the pipenv install command would take forever, and often either seg-fault in the process or just never complete.

From using pipenv on other, non-NixOS machines, I knew that installing numpy and pandas was much faster on those machines, but I didn’t know why. Finally, I found out that on OS X, Windows, and some distributions of Linux, Python installs “wheels”, which contain the prebuilt binary code, rather than building from source. It only does this on Linux if it detects that your distribution supports wheels from the manylinux project. A quick test reveals that my Python distribution is not manylinux-compatible:

[sidharth@nixos:~]$ python3
Python 3.6.4 (default, Dec 19 2017, 05:36:13) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _manylinux
>>> _manylinux.manylinux1_compatible
False

The solution

In order to use manylinux whls to get fast package installs, I needed to

1. set up my system so that manylinux whls work (i.e. make it many-linux compatible),
2. convince Python that my system is manylinux-compatible.

Step 1

The first step is to make an environment that follows the manylinux1 policy:

with import <nixpkgs> {};

buildFHSUserEnv {
  name = "my-python-env";
  targetPkgs = pkgs: with pkgs; [
    python3
    pipenv
    which
    gcc
    binutils

    # All the C libraries that a manylinux_1 wheel might depend on:
    ncurses
    xorg.libX11
    xorg.libXext
    xorg.libXrender
    xorg.libICE
    xorg.libSM
    glib
  ];

  runScript = "$SHELL";
}

This creates an environment that contains all the C libraries that a manylinux_1 wheel might depend on, all in /usr/lib/, where the binaries expect it. Save this in default.nix, run nix-build, and you can enter this environment with ./result/bin/my-python-env.

Step 2

To convince Python that this environment is manylinux_1-compatible, you need _manylinux.manylinux1_compatible to be true. We can do this by making a file called _manylinux.py that contains:

print("in _manylinux.py")
manylinux1_compatible = True

To make sure that pipenv finds this code, you need to save this file somewhere in your PYTHONPATH. I did this by adding

let
  manyLinuxFile =
    writeTextDir "_manylinux.py"
      ''
        print("in _manylinux.py")
        manylinux1_compatible = True
      '';

to the top of my default.nix, and by adding the following to my derivation in default.nix:

  profile = ''
    export PYTHONPATH=${manyLinuxFile.out}:/usr/lib/python3.6/site-packages
  '';

Conclusion

With the above steps, I am able to pipenv install any package, and it uses the whl, as long as I am within my nix environment (by running ./result/bin/my-python-env). Hooray!