Table of Contents

Providing CI for 25 Rails Apps

On the code you see
The code you see here may or may not work. It is largely taken from memory because I am not allowed to share my companies code.

Brief Background

I work on something similar to a “platform” team for 130 Ruby on Rails developers. We run and provide continuous integration (CI) via self-hosted Github Actions for the applications that other developers build.

A bit more background
All the compute is handled by several Kubernetes clusters while the actual code lives in multiple repositories all under the same organization. Thus, a lot of the challenges we face are around providing flexibility for our developers without implementation details leaking out to them.

Our apps all follow a fairly similar pattern:

  • they have their own github repository under our organization
  • they have a Dockerfile
  • get built as an docker container image (my terminology might be wrong here)
  • get served via Kubernetes

However, despite serving everything as containers our developers don’t actually develop in containers. So to avoid the issue where production is the first place that our apps see containerized use we run all our tests in containers.

How Do We Run CI?

All our apps have this general structure:

.
├── app_code
├── ...
├── .github
│   └── workflows
│      └── main.yaml
├── Dockerfile
├── bin
│   ├── ci-some-command
│   └── ci-some-other-command
└── docker-compose.test.yaml

Our Dockerfile largely follows the default generated by Rails. The difference are mainly in the certs that we add to the image.

The scripts in /bin as just basic bash scripts that represent the testing process, linting process or whatever. For example:

# bin/ci-lint
yarn lint
bundle exec brakeman
bundle exec rubocop

The idea is that these are the same scripts that a developer could run locally on their machine. Reusing the scripts ensure they are the source of truth for both local and CI processes.

The docker-compose looks something like this:

services:
  app:
    build:
      context: .
    image: app

  test:
    image: app
    command: "bin/ci-some-command"
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 1s
      timeout: 5s
      retries: 5

Running docker compose up locally builds the image, starts Postgres, then runs bin/ci-some-command in against the container with the app. We specify a default command but docker compose allows us to override the command when called via command line.

CI runs tests via the same docker compose and an commands that a developer would run locally. Lets take a look at that.

Getting It All Set Up in Github Actions

So how does this all look as an actions workflow?

jobs:
  test:
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ vars.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx (to allow building a multiarch image)
        uses: docker/setup-buildx-action@v3

      - name: Set up build config
        uses: docker/bake-action@v5
        with:
          pull: true
          load: true
          files: docker-compose.test.yaml

      - name: Test
        id: run_tests
        env:
          DOCKER_RUN_CMD: ${{ inputs.docker-command }}
        shell: sh
        run: |
          docker compose -f docker-compose.test.yaml \
            run --rm "rails-test" "$DOCKER_RUN_CMD"

      - name: RSpec Report
        if: ${{ !cancelled() }}
        uses: our_org/actions/ruby-rspec-report@v4

To sum it up:

  1. we pull down the repo (checkout)
  2. set up docker (login, setup buildx, bake)
  3. run the test in docker compose
  4. print a report.
- name: Test
  id: run_tests
  env:
    DOCKER_RUN_CMD: ${{ inputs.docker-command }}
  shell: sh
  run: |
    docker compose -f docker-compose.test.yaml \
      run --rm "rails-test" "$DOCKER_RUN_CMD"

As mentioned before, the command in the docker-compose file can be override. The DOCKER_RUN_CMD here is passing in commands to run other bin scripts (ex: bin/ci-lint and bin/ci-rspec). I omitted showing the potential inputs that this action might receive.

Benefits and Drawbacks

Benefits

Shifting Docker Left: The majority of our developers don’t develop in docker. Thus, for our apps to see any containerized usage before the apps hit staging or production we need to run tests in docker. This makes docker in CI a necessity.

Platform Agnosticism: Since our testing happens by starting docker compose and running a script with test commands the entire process is platform agnostic. We don’t have to worry about differences in ruby / node / etc on GitHub Actions vs Jenkins vs CircleCI. We set up docker and that’s it. And if we are protected if we ever need to switch to another provider.

Drawbacks

Performance Overhead: In terms of the container overhead, running software directly in the runner will always be faster than running it with docker compose on the runner. Does it really matter? Not really.

Container Build Overhead: Building images is slow. The P90 of the setup step is 15 minutes on our biggest app (when all the layers miss the cache). Fixing this is still a work in progress.

Conclusion

We have moved the complexity around within the actions jobs. Without docker there might be complexity in ensuring all required packages are present to test the app. With docker you get that for free but instead you have to worry about setting up and building the image. This made sense for us, it might not for you.

I left a lot of the detail out of my summary but we have about 25 web apps under us of varying sizes that we support this way. Beyond the initial setup the files in each app repository stay fairly static and developers largely get to ignore them. CI usually just works.

That was a super brief overview of our CI and in future posts I will go a bit more in depth on how we provide a few cool features namely:

  • variably size parallelization of runs via input: runner-count: 8
  • providing arbitrary commands to get run via CI: [{"cmd": "some-command"}]
  • doing all of the above with SimpleCov and the parallel_test gem