baiji

baiji

Baiji is a package building tool for Debian environments. The overall design borrows heavily from an existing tool called whalebuilder, which in turn is in effect a reimplementation of pbuilder using Docker to do the heavy lifting.

Clean room build environment

pbuilder is one of several tools all of which have the same basic goal: building packages in an environment specified by the package's control file with minimal dependencies on the native environment of the machine where the build is being run.

pbuilder is probably the oldest surviving example of this kind of build environment for the Debian universe, and uses a fairly simple approach: it keeps a pristine build environment in a compressed tarball, which it unpacks into a freshly-chrooted tree for every build, thus insuring a predictable environment. The pristine base environment only includes the "build-essentials" environment that every Debian package is allowed to assume is always installed: anything else specified in the control file is installed on demand every time. While a bit finicky to set up, this is pretty simple in overall concept, and pretty reliable. Unfortunately, it's also pretty slow.

pbuilder lacks direct support for dependencies on packages not known to the Debian project, but it does support hook scripts, which is how we taught it to use the local package collections that our current build system stores in /var/cache/pbuilder/local-packages/.

If one looks closely at this setup, it turns out to have a lot in common with Docker containers. The specific mechanisms are different, but the overall effect is the same: an isolated environment running a controlled set of packages and scripts independent of the machine on which it's running. The difference is that Docker has a lot of support for building and distributing images based on other images.

There's a lovely little tool called whalebuilder which to a first approximation is a reimplementation of pbuilder on top of Docker. It turns out that whalebuilder doesn't do quite what we want, but it's close enough to serve as a (very) useful model for a new tool

baiji

baiji is intended to incorporate the most useful features of both whalebuilder and git-buildpackage, along with some custom features such as the ability to load locally-built .deb packages directly into the constructed Docker image.

Like both pbuilder and whalebuilder, baiji uses a "base image", which is constructed using the debbootstrap package, along with a few things needed by baiji itself.

Like whalebuilder but unlike pbuilder, baiji also builds package-specific images for building a particular source package. While this may sound wasteful, it turns out that it's not: everything involved in the per-package image construction process is work that baiji would have to do anyway, and by saving the result baiji makes it possible to reuse that image if it turns out that one has to build the same version of the same source package repeatedly. By default, baiji only generates a new package-specific build image when the package's version or build dependencies have changed, but the user can override this to force a new image.

Also like whalebuilder but unlike pbuilder, baiji performs the build itself as a non-root user, which is a better fit for the base Debian model of building packages, and runs with networking turned off, so that it can catch undeclared dependencies like a package quietly attempting to pip install 25 new packages from PyPi.

Like debuild and the pdebuild but unlike whalebuilder, baiji can generate the source package from the source tree when needed.

As mentioned above, baiji supports pre-loading local binary packages directly into a local APT repository within the package-specific Docker image.

baiji's base images can be pre-configured to know about local APT servers,, so that any packages needed for a build can be downloaded directly. apt running within the usual build tools will use the package version numbers to decide which version of a package to use when more than one is available.

baiji's current Interface to Docker just launches the Docker CLI tool in a subprocess, which turns out to be a lot simpler than using the official Docker Python API; the latter is only available from GitHub and PyPi, and depends on enough other PyPi packages that it's a bit of a maintenance nightmare. Perhaps this will calm down someday, but for now it's worth the minor inefficiency of using the Docker CLI to avoid the swamp.

Unlike whalebuilder, which uses uses Ruby's built-in Jinja-like template mechanism, baiji currently just includes the handful of templates it needs inline using Python's str.format() method. Some day baiji may outgrow this, but it suffices for now.