baiji ===== Baiji is a package building tool for Debian environments. The overall design borrows heavily from an existing tool called `whalebuilder`, which in turn is in effect a reimplementation of `pbuilder` using Docker to do the heavy lifting. Clean room build environment ---------------------------- `pbuilder` is one of several tools all of which have the same basic goal: building packages in an environment specified by the package's `control` file with minimal dependencies on the native environment of the machine where the build is being run. `pbuilder` is probably the oldest surviving example of this kind of build environment for the Debian universe, and uses a fairly simple approach: it keeps a pristine build environment in a compressed tarball, which it unpacks into a freshly-chrooted tree for every build, thus insuring a predictable environment. The pristine base environment only includes the "build-essentials" environment that every Debian package is allowed to assume is always installed: anything else specified in the `control` file is installed on demand every time. While a bit finicky to set up, this is pretty simple in overall concept, and pretty reliable. Unfortunately, it's also pretty slow. `pbuilder` lacks direct support for dependencies on packages not known to the Debian project, but it does support hook scripts, which is how we taught it to use the local package collections that our current build system stores in `/var/cache/pbuilder/local-packages/`. If one looks closely at this setup, it turns out to have a lot in common with Docker containers. The specific mechanisms are different, but the overall effect is the same: an isolated environment running a controlled set of packages and scripts independent of the machine on which it's running. The difference is that Docker has a lot of support for building and distributing images based on other images. There's a lovely little tool called `whalebuilder` which to a first approximation is a reimplementation of `pbuilder` on top of Docker. It turns out that `whalebuilder` doesn't do quite what we want, but it's close enough to serve as a (very) useful model for a new tool baiji ----- `baiji` is intended to incorporate the most useful features of both `whalebuilder` and `git-buildpackage`, along with some custom features such as the ability to load locally-built `.deb` packages directly into the constructed Docker image. Like both `pbuilder` and `whalebuilder`, `baiji` uses a "base image", which is constructed using the `debbootstrap` package, along with a few things needed by `baiji` itself. Like `whalebuilder` but unlike `pbuilder`, `baiji` also builds package-specific images for building a particular source package. While this may sound wasteful, it turns out that it's not: everything involved in the per-package image construction process is work that `baiji` would have to do anyway, and by saving the result `baiji` makes it possible to reuse that image if it turns out that one has to build the same version of the same source package repeatedly. By default, `baiji` only generates a new package-specific build image when the package's version or build dependencies have changed, but the user can override this to force a new image. Also like `whalebuilder` but unlike `pbuilder`, `baiji` performs the build itself as a non-root user, which is a better fit for the base Debian model of building packages, and runs with networking turned off, so that it can catch undeclared dependencies like a package quietly attempting to `pip install` 25 new packages from PyPi. Like `debuild` and the `pdebuild` but unlike `whalebuilder`, `baiji` can generate the source package from the source tree when needed. As mentioned above, `baiji` supports pre-loading local binary packages directly into a local APT repository within the package-specific Docker image. `baiji`'s base images can be pre-configured to know about local APT servers,, so that any packages needed for a build can be downloaded directly. `apt` running within the usual build tools will use the package version numbers to decide which version of a package to use when more than one is available. `baiji`'s current Interface to Docker just launches the Docker CLI tool in a subprocess, which turns out to be a lot simpler than using the official Docker Python API; the latter is only available from GitHub and PyPi, and depends on enough other PyPi packages that it's a bit of a maintenance nightmare. Perhaps this will calm down someday, but for now it's worth the minor inefficiency of using the Docker CLI to avoid the swamp. Unlike `whalebuilder`, which uses uses Ruby's built-in Jinja-like template mechanism, `baiji` currently just includes the handful of templates it needs inline using Python's `str.format()` method. Some day `baiji` may outgrow this, but it suffices for now.