Trying Out Buildkite

I recently started playing with Buildkite to explore building an integration with Review Board. Buildkite is one of the few Continuous Integration tools that I see people talk about without a lot of cursing, so I was particularly interested to try it out.

One thing I like to do when exploring a CI tool is set up package tests and builds for some of our repos. At Beanbag everything we make is designed to be installed on-prem, and we’re not in control of when users choose to upgrade. This is an old-fashioned way of doing things, and a lot of modern tooling is oriented towards different use cases.

At any given time, our support matrix looks a little crazy. For Review Board we’ll have at least four active versions:

As we do fixes and updates, we’ll merge these branches up the chain. A fix that gets shipped on release-7.x has to merge through release-8.x and release-9.x before it finally hits master.

This is further complicated because we’re not using a monorepo. Our code is broken up into many different packages, some of which we’ll release at different times. Each active branch of reviewboard has a corresponding active branch of djblets, and then we have rbintegrations, Review Bot, and more. Some of these packages also have their own support matrixes: for example, a particular major version of Review Bot may support multiple recent major version of Review Board. Our developer documentation is littered with tables showing which branches need to be checked out for which releases.

When considering what we need from CI, our primary goal is to be able to ensure that builds and tests across all our packages stay green. If we’re landing a fix in djblets release-6.x, we not only want to run tests for that repo/branch, but once that’s done we also want to run tests for reviewboard release-8.x. Once we know that the reviewboard tests succeed, we may also want to run tests rbintegrations release-5.x, reviewbot release-4.x, and more.

We’re in the process of simplifying this, but I suspect as long as we remain committed to letting people install things into their own hardware, we’re stuck with some level of complexity.

How our current builds work

When we decided to revamp our production CI system many years ago, we ended up building on Concourse. This is a system that is extremely powerful in some ways and extremely limiting in others. It allows us to handle the problem of doing a build which can span many different repositories, branches, and versions. It mostly doesn’t crash. It includes pretty nice isolation primitives for every single step, and legitimately great CLI tooling for interacting with the live server from a developer workstation.

Each job in Concourse has a set of inputs, a set of steps, and an output. These are then assembled together into pipelines:

1. An initial -prep job which takes in one or more repositories as its inputs and creates a working environment as its output. For Python packages this environment is essentially the virtualenv with the package set up and all dependencies installed. For JavaScript, its a node_modules directory.
2. A -test job. This runs our test suite, and for Python we’ll run individual tasks for each supported Python version.
3. -audit jobs which run pip audit, npm audit, and govulncheck.
4. A final -build job that builds and stores the package.

Because we have the output from both the -prep and -build jobs, we can chain together pipelines from different repositories. When a djblets build completes, it can trigger reviewboard for the matching branch, ensuring that when we push code we’re not breaking our downstream dependencies. If tests in one repo effectively need the same setup as another, we can just share the environment as the input.

This generally works well, but it also can be a headache when things go wrong. If a build tries to pull in an output from an extremely old job, sometimes that output has fallen out of storage and the build will fail. Additionally, because we’re passing around Python virtualenvs and node_modules directories, we don’t have any opportunity to parallelize across locations, and the storage requirements can be significant. These can also be quite fragile across versions. Sometimes a new version of setuptools or pip will break everything, requiring us to manually trigger new builds throughout the entire system.

Buildkite package registries

One of the things that immediately piqued my interest as I started researching Buildkite was their package registries. These are private registries that can be pushed to from a CI pipeline. Instead of sharing “build-time” assets like a virtualenv, this would allow us to just push the prerelease builds for each package to the registry, and then other package builds can include that registry when setting up the test/build environments.

By including the registry as an extra index, any package installs that happen when doing builds will just pull matching versions from there. I’ve done this for all builds via my buildkite-agent environment file:

export PIP_TRUSTED_HOST=pip-cache.local
export PIP_INDEX_URL=http://pip-cache.local:49950/index

# Add local package registries
if [ -n "$PACKAGE_REGISTRY_TOKEN" ]; then
    export PIP_EXTRA_INDEX_URL="https://buildkite:${PACKAGE_REGISTRY_TOKEN}@packages.buildkite.com/example-org/python/pypi/simple"
fi

The one annoyance I ran into is the way that the registries handle packages with the same version. If I’m doing a prerelease build of djblets release-6.x, that’s going to end up creating a package named something like djblets-6.0.1a0.dev0-py3-none-any.whl. If we’ve already got a package with that name from a prior commit, everything would work until the publish step, at which point it would fail because there was already a package with version 6.0.1a0.dev0.

To solve this, I created a small buildkite plugin which authenticates to your package registry with OIDC and deletes any existing packages that match the artifacts that exist in the current build. This currently handles Python and JavaScript packages. Add it as a new step between your build and publish steps:

steps:
- key: build
  label: ":python: Build wheel"
  command: bash _build.sh
  plugins:
  - github.com/buildkite-plugins/artifacts-buildkite-plugin#v1.9.4:
      download: _build.sh
  - github.com/buildkite-plugins/docker-buildkite-plugin#v5.13.0:
      image: ...
      environment:
      - PACKAGE_REGISTRY_TOKEN
      - PIP_TRUSTED_HOST
      - PIP_INDEX_URL
      - PIP_EXTRA_INDEX_URL
  secrets:
  - key: PACKAGE_REGISTRY_TOKEN
    environment_variable: PACKAGE_REGISTRY_TOKEN
- key: delete-old-package
  label: ":wastebasket: Delete existing dev package"
  depends_on: package
  command: ''
  plugins:
  - https://github.com/davidt/package-delete-buildkite-plugin.git#5cc0f0fa:
      registry: example-org/python
      artifacts: "*.whl"
- label: ":pypi: Publish to registry"
  depends_on: delete_old_package
  command: ''
  plugins:
  - github.com/buildkite-plugins/publish-to-packages-buildkite-plugin#v2.2.0:
      registry: example-org/python
      artifacts: "*.whl"

Should we switch?

Will we switch to Buildkite? I have no idea. I’m quite impressed playing with it, and the built-in package registries solves a real problem with how things work in our codebases.

Whether or not we eventually move away from Concourse, we’ll definitely be adding support for Buildkite as a CI integration for Review Board.