For several years now I’ve been using Dokku on my personal web server as an easy way to spin up new web apps I want to play around with. Like the Heroku platform on which it’s modeled, Dokku allows me to deploy a new version of an app by simply running git push, but Dokku is fully open-source and I can self-host it.

I’ve written before that Dokku imposes constraints I don’t like, so as part of a plan to replace it with something that more exactly meets my needs, I wrote “The most general reverse proxy”. That only addressed a small portion of what Dokku is currently doing for me, though, and I’m starting to think I need to write my own service manager to capture the rest; neither systemd nor its competitors seem to have any support for rolling deployments.

Meanwhile, I’ve also been working on addressing the things I find most painful about Dokku as I continue using it. And I’ve concluded that what frustrates me most in routine usage of Dokku is all inherited from Heroku’s design, by way of the Herokuish compatibility layer that Dokku uses by default.

Let’s be clear here: both Dokku and Heroku are great tools, and if you do any web development, or just want to self-host web apps that others have developed, you should definitely try them out. But it’s always worth trying to improve things, and I’ve come up with a bit of a hack that’s already working much better for my needs.

How it works now

The core of my complaint is the method that Heroku, and by extension Herokuish, uses to construct an application container image.

Every time I run git push, the new version of my app needs to be combined with all its dependencies into a bundle that the hosting platform can execute. The difficulty is in how I as an app developer should specify those dependencies in such a way that the platform can find and install them for me.

Heroku uses two different approaches for this. First, Heroku provides a base image containing a stock Ubuntu install and a collection of commonly-used packages, such as compilers and database libraries. (See the official Heroku Stack Images for details.) This stack image is around 1.5GB of stuff that a lot of people need some of the time, but of course for any given app the vast majority of it is unused.

After that, buildpacks provide language-specific autodetection of dependencies.¹ For example:

• the Python buildpack looks for common Python config files like requirements.txt or Pipfile.lock and installs whatever Python packages those files specify;
• the PHP buildpack looks for composer.json,
• the Node.js buildpack looks for package.json,
• the Ruby buildpack looks for Gemfile.lock,

and so on. Heroku has a collection of officially-supported buildpacks but other developers can write their own and share them. If one or more of the 7,246 existing buildpacks applies to your project, this makes getting started really easy: you don’t need to understand anything about how Heroku works, only how to use your chosen development environment’s tools. It’s magic!

If neither the stack image nor any of the available buildpacks does what you need, though, then you fall off a complexity cliff. Suddenly you need to understand the filesystem layout of a Heroku container, think about which version and architecture of Ubuntu it’s based on, and dig deep into what the buildpacks were doing for you up to that point. It’s too much magic.

Buildpacks are specialized enough that people fall off that complexity cliff all the time, as evidenced by the fact that there are over seven thousand buildpacks. That’s an average of two or three new buildpacks every day since Heroku announced buildpacks in 2012.

That shallowly-buried complexity was already enough to make me want something better, but what really bothers me every time I run git push is how slow the build process is. I think there are two major causes for this:

1. It’s entirely sequential, which can’t be fixed without collecting information about the dependencies between build steps. This is the usual situation for Docker images in general, not just Heroku.

2. Although buildpacks have always been allowed to cache build products from one deploy to the next, it’s up to the buildpack author to use the cache correctly, and that’s hard to get right. I suspect buildpack maintainers who try to implement really aggressive caching are punished by more bugs and headaches than it’s worth to them.

Last and probably least, it bothers me that there’s so much unnecessary stuff in the final deployed image. Programs which are only needed at build time, such as C compilers and static site generators, remain in the image when deployed; and then, of course, there’s the 1.5GB of mostly-unused Ubuntu packages. Because Docker only stores one copy of the stack image and shares it between all the apps which use it, this is more of an aesthetic consideration rather than a real problem. That said, if your app has a security hole, this is a huge attack surface which can make exploiting bugs easier.

I have a fix for all of these problems… in exchange for creating new ones, naturally.

Building minimal Docker images quickly with Nix

I’ve started using the Nix package manager to build the Docker images that I deploy with Dokku. Most people aren’t familiar with Nix, so I’m going to give a little background, but since this blog post isn’t really about Nix, I’m going to oversimplify a bunch of details. If you are familiar with Nix, please just ignore the inaccuracies. kthx 😅

Nix provides a unified way to describe the complete build-time and run-time dependencies of arbitrary software. Instead of writing giant shell scripts to sequence all your build steps, Nix gives you a programming language that treats build products as first-class objects. The result of running a Nix program (a “derivation”) is the dependency tree of all the steps needed to build the specified software from scratch. That dependency tree can then be “realized” to run all the steps and complete the build.

Nix enforces that a derivation’s dependencies are specified correctly, by building it in an environment where only the requested versions of those dependencies are available. As a result, Nix can automatically and reliably determine whether two builds can proceed in parallel and whether a previous build of a derivation can be reused because its inputs haven’t changed.

So Nix builds get parallelism and aggressive caching for free. In fact, it has built-in support for distributing builds across, and sharing caches between, multiple computers.

Of course, somebody has to write all those Nix recipes, and it’s easiest if you can just reuse recipes that somebody else already wrote. The usual source for those is the Nix Packages collection, Nixpkgs, although various communities also provide package “overlays” that extend Nixpkgs.

But just like having thousands of buildpacks available still doesn’t cover every use case, sometimes I’ve needed something that wasn’t packaged.² Compared to writing buildpacks, though, Nix build recipes have a much more gradual learning curve,³ and the Nixpkgs manual offers extensive documentation for the many different helpers that Nixpkgs maintainers have come up with over the years.

Once you have a recipe that builds your application, Nixpkgs provides an easy way to turn that into a Docker image. For example, I build this blog with a Nix expression that looks something like this, where pkgs is an instance of Nixpkgs and config is the result of another derivation that writes the lighttpd.conf I want:

pkgs.dockerTools.streamLayeredImage {
  name = "jamey.thesharps.us";
  config.Cmd = [ "${pkgs.lighttpd}/bin/lighttpd" "-D" "-f" config ];
}

Nix builds are isolated so it can’t load the image into Docker directly, so the result of that recipe is actually a shell script which constructs the Docker image. To load it into Docker, I run:

$(nix-build) | docker load

I generate my lighttpd config file with a recipe like the following, where docroot is the result of yet another derivation which runs Jekyll to build the static HTML of this blog. One neat thing is that the checkPhase script will fail the build if lighttpd can’t parse the configuration, rather than having lighttpd fail to start after I’ve already deployed the broken configuration.

pkgs.writeTextFile {
  name = "lighttpd.conf";

  checkPhase = ''
    PORT=5000 ${pkgs.lighttpd}/bin/lighttpd -tt -f $n
  '';

  text = ''
    server.document-root = "${docroot}"
    server.port = env.PORT
    include "${pkgs.lighttpd}/share/lighttpd/doc/config/conf.d/mime.conf"

    etag.use-mtime = "disable"
    setenv.set-response-header = ("Last-Modified" => "")

    # more settings follow...
  '';
}

This illustrates one “gotcha” of using Nix, which sets the mtime of all files to 0. That means mtime is not useful in ETag or Last-Modified headers. Cache validation can still get correct results because all the inodes should change on every rebuild, so the generated ETag should change if any content changes. That invalidates more cache entries than necessary but at least it’s sufficient for correctness.

Since config.Cmd references both the lighttpd binary and my config file, both are automatically included into the image. In addition, lighttpd needs some libraries, and the config file references the Jekyll-built HTML, so all of those are included too.

But after the HTML is generated I don’t need Jekyll or Ruby any more, so those are not added to the image. Only the derivations which are transitively referenced are included, so there’s nothing extraneous being added at all. The Docker image for this blog is about 42MB, instead of 1.5-2GB, and five-sixths of that comes from glibc and OpenSSL, which aren’t worth getting rid of.⁴

So switching to Nix addresses all the complaints I mentioned earlier: it has a gradual learning curve rather than a complexity cliff, it builds images more quickly, and the images it builds are as small as possible.

Integrating Nix with Dokku

Now for the part which is currently more of an ugly hack than ready for production. The first time you git push to a new Dokku-hosted repo, Dokku creates a git pre-receive hook which looks something like this:

#!/usr/bin/env bash
set -e
set -o pipefail

cat | DOKKU_ROOT="/home/dokku" dokku git-hook jamey.thesharps.us

All the work of building a Docker image using Herokuish is done by Dokku’s internal git-hook trigger.

To make git push trigger a Nix build instead, I replaced that with this script:

#!/usr/bin/env bash
set -e; set -o pipefail;

export NIX_PATH="/nix/var/nix/profiles/per-user/root/channels"
export PATH="/nix/var/nix/profiles/default/bin:$PATH"
export DOKKU_ROOT="/home/dokku"

app=jamey.thesharps.us

while read old new ref; do
	if ! test "$ref" = "refs/heads/master"; then
		echo "skipping ref $ref--the deployment branch is master" >&2
		continue
	fi

	script=$(nix-build --expr "import \"\${builtins.fetchGit {
		url = ./.;
		rev = \"$new\";
	}}/docker.nix\" {}")
	image=$($script | docker load | sed -n '$s/^Loaded image: //p')
	docker image tag "$image" dokku/"$app":latest
	dokku tags:deploy "$app"
done

This new script does the following steps:

1. Tell Nix to find and evaluate the docker.nix file in the newly-pushed revision. This builds all the dependencies and the image-generating shell script.

2. Generate the image, load it into Docker, and find out what tag it got assigned. The same commit might get deployed to multiple different apps, so it shouldn’t have the app name hard-coded into it, and I’m letting the Nixpkgs docker tools generate the image tag from a hash of its dependencies.

3. Apply a new tag to the image using the naming convention that Dokku expects. For example: dokku/jamey.thesharps.us:latest

4. Finally, tell Dokku to deploy the tagged image.

I’d rather do all this in a Dokku plugin, but I dug around the source code a bit and didn’t see any way that plugins can provide alternatives to Herokuish or Dockerfile deployments. If I have enough energy for it, at some point I’ll open an issue about this on the Dokku repo.

Dokku supports deploying Docker images built on a CI server (as does Heroku, I gather), so as an alternative I could have set up a CI server that I git push to and which deploys the build result into Dokku.

But this hack works for me, and I’ve switched most of the apps on my personal server over to it.

Conclusion

Switching from Herokuish to Nix while still deploying using Dokku is a pretty big change. For me, it’s already paid off several times over: I can concisely express exactly the configuration and dependencies that an app requires, and I feel much more comfortable deploying now that the build step is usually seconds instead of minutes.

I’m less happy about the method I’ve used to glue Nix to Dokku. But Dokku is being actively developed, so I have hope that at some point the maintainers will introduce more flexible triggers that allow plugins to implement alternative build methods like this.

I don’t see any reason Heroku or other similar platforms couldn’t offer Nix builds as an option, for that matter.

Nix may not be the right tool for you. Dokku and Heroku may not meet your needs either. But I think they’re all tools worth knowing about, at the very least.

Appendix: Language-specific examples

The original purpose of the buildpack abstraction was to make it easy to get a container from source code written in a wide range of programming languages. So if we replace buildpacks with Nix, how much work does it take to get started?

The answer varies a lot from one language to the next, but here are a couple of examples from the deployments I’ve done using Nix recently.

Python

A new tool that’s made deploying my Python projects almost trivial is poetry2nix, which gets all of the packaging information it needs from the poetry.lock file written by the Python Poetry package manager. I can run a command like poetry add --lock gunicorn to declare a new dependency on a Python package and the Nix build immediately picks it up, using the exact version that Poetry’s dependency resolver selected.

If you’re using either the unstable version of Nixpkgs or the overlay that the poetry2nix developers provide, you should be able to build a working Docker image from any Poetry-managed project with a recipe like this one, which I’ve simplified from the docker.nix in my predictable project:

let
  pkgs = import <nixpkgs> {};
  app = pkgs.poetry2nix.mkPoetryApplication {
    projectDir = ./.;
  };
in pkgs.dockerTools.streamLayeredImage {
  name = "predictable";
  contents = [ app.dependencyEnv ];
  config.Cmd = [ "/bin/gunicorn" "predictable:app" ];
}

The best part is that poetry2nix builds each Python package as a separate Nix derivation. So unlike standard Python tools like Pip which install all packages into one site-packages directory, this means

• multiple Python packages can build in parallel,
• only Python packages which have changed need to be rebuilt,
• and each Python package is built in an isolated environment with only its declared dependencies available, avoiding surprises later from unspecified dependencies.

This tooling also works nicely together with direnv (which I’ve written about before in “Per-project Postgres”) for managing a local development environment. Here’s a simplified version of the shell.nix which you can find alongside the above docker.nix:

let
  pkgs = import <nixpkgs> {};
  app = pkgs.poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    editablePackageSources.predictable = ./.;
  };
in pkgs.mkShell { buildInputs = [ app pkgs.poetry ]; }

Then if I just write use nix in .envrc and run direnv allow, when I cd into that project’s working copy, I get a complete development environment automatically.

PHP

I’m not real familiar with the PHP ecosystem, but there are web apps I want to use that are written in PHP, so I dug into the Nixpkgs support for PHP a bit and ended up with a Nix recipe that looks something like this:

let
  pkgs = import <nixpkgs> {};
  lighttpd = pkgs.lighttpd;
  php = pkgs.php;

  config = pkgs.writeTextFile {
    name = "lighttpd.conf";

    checkPhase = ''
      PORT=5000 ${lighttpd}/bin/lighttpd -tt -f $n
    '';

    text = ''
      server.port = env.PORT
      server.upload-dirs = ("/tmp")

      fastcgi.server = (".php" => ((
        "bin-path" => "${php}/bin/php-cgi",
        "bin-environment" => (
          # Nixpkgs 20.09 wraps bin/php to set
          # this but does not wrap bin/php-cgi
          "PHP_INI_SCAN_DIR" => "${php}/lib",
        ),
        # see lighttpd and php docs for more that should go here
      )))
    '';
  };
in pkgs.dockerTools.streamLayeredImage {
  name = "phpapp";
  extraCommands = ''
    # PHP expects to be able to create a lockfile in /tmp
    mkdir -m 1777 tmp
  '';
  config.Cmd = [ "${lighttpd}/bin/lighttpd" "-D" "-f" config ];
}

There were two non-obvious things I had to do. One is that there’s no /tmp in Nixpkgs-generated Docker images by default—in fact, there’s nothing except /nix/store/—and both PHP and lighttpd sometimes need a place to write temporary files. So I used the extraCommands option to create that directory. (lighttpd defaults to /var/tmp which of course doesn’t exist either, so I configured it to use the same /tmp directory instead.)

The other is, I think, a bug in the current release of Nixpkgs, which I ought to open an issue about. PHP quite naturally doesn’t know how to find anything in the directory structure that Nix uses, so the Nix package of PHP auto-generates a php.ini file with the full path to each enabled PHP extension. But then, PHP doesn’t know how to find that php.ini file either, so the package arranges that the php command is actually a shell script that sets PHP_INI_SCAN_DIR to the right path before running the real php binary. However, the package doesn’t wrap php-cgi in the same way, so that binary can’t find php.ini either. As a workaround, I set the variable in the lighttpd config instead.

There’s one more useful trick I learned. By default, the PHP package enables a bunch of extensions, which is good for getting started quickly, but it also pulls in a lot of unnecessary dependencies. Once I had things working, I replaced the php = pkgs.php; line above with this:

  php = pkgs.php.buildEnv {
    extensions = { all, ... }: with all; [
      json
      session
      # any other extensions you want here
    ];
    extraConfig = ''
      upload_max_filesize = 64M
      ; more custom php.ini settings follow ...
    '';
  };

Footnotes