Providing CI for 25 Rails Apps
On the code you see
The code you see here may or may not work. It is largely taken from memory because I am not allowed to share my companies code.
Brief Background
I work on something similar to a “platform” team for 130 Ruby on Rails developers. We run and provide continuous integration (CI) via self-hosted Github Actions for the applications that other developers build.
A bit more background
All the compute is handled by several Kubernetes clusters while the actual code lives in multiple repositories all under the same organization. Thus, a lot of the challenges we face are around providing flexibility for our developers without implementation details leaking out to them.
Our apps all follow a fairly similar pattern:
- they have their own github repository under our organization
- they have a
Dockerfile
- get built as an docker container image (my terminology might be wrong here)
- get served via Kubernetes
However, despite serving everything as containers our developers don’t actually develop in containers. So to avoid the issue where production is the first place that our apps see containerized use we run all our tests in containers.
How Do We Run CI?
All our apps have this general structure:
.
├── app_code
├── ...
├── .github
│ └── workflows
│ └── main.yaml
├── Dockerfile
├── bin
│ ├── ci-some-command
│ └── ci-some-other-command
└── docker-compose.test.yaml
Our Dockerfile
largely follows the default generated by Rails. The difference are mainly in the certs that we add to the image.
The scripts in /bin
as just basic bash scripts that represent the testing process, linting process or whatever. For example:
# bin/ci-lint
yarn lint
bundle exec brakeman
bundle exec rubocop
The idea is that these are the same scripts that a developer could run locally on their machine. Reusing the scripts ensure they are the source of truth for both local and CI processes.
The docker-compose
looks something like this:
services:
app:
build:
context: .
image: app
test:
image: app
command: "bin/ci-some-command"
depends_on:
postgres:
condition: service_healthy
postgres:
healthcheck:
test: ["CMD", "pg_isready", "-U", "postgres"]
interval: 1s
timeout: 5s
retries: 5
Running docker compose up
locally builds the image, starts Postgres, then runs bin/ci-some-command
in against the container with the app. We specify a default command but docker compose allows us to override the command when called via command line.
CI runs tests via the same docker compose and an commands that a developer would run locally. Lets take a look at that.
Getting It All Set Up in Github Actions
So how does this all look as an actions workflow?
jobs:
test:
steps:
- name: Checkout the repository
uses: actions/checkout@v4
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx (to allow building a multiarch image)
uses: docker/setup-buildx-action@v3
- name: Set up build config
uses: docker/bake-action@v5
with:
pull: true
load: true
files: docker-compose.test.yaml
- name: Test
id: run_tests
env:
DOCKER_RUN_CMD: ${{ inputs.docker-command }}
shell: sh
run: |
docker compose -f docker-compose.test.yaml \
run --rm "rails-test" "$DOCKER_RUN_CMD"
- name: RSpec Report
if: ${{ !cancelled() }}
uses: our_org/actions/ruby-rspec-report@v4
To sum it up:
- we pull down the repo (checkout)
- set up docker (login, setup buildx, bake)
- run the test in docker compose
- print a report.
- name: Test
id: run_tests
env:
DOCKER_RUN_CMD: ${{ inputs.docker-command }}
shell: sh
run: |
docker compose -f docker-compose.test.yaml \
run --rm "rails-test" "$DOCKER_RUN_CMD"
As mentioned before, the command in the docker-compose
file can be override. The DOCKER_RUN_CMD
here is passing in commands to run other bin
scripts (ex: bin/ci-lint
and bin/ci-rspec
). I omitted showing the potential inputs that this action might receive.
Benefits and Drawbacks
Benefits
Shifting Docker Left: The majority of our developers don’t develop in docker. Thus, for our apps to see any containerized usage before the apps hit staging or production we need to run tests in docker. This makes docker in CI a necessity.
Platform Agnosticism: Since our testing happens by starting docker compose
and running a script with test commands the entire process is platform agnostic. We don’t have to worry about differences in ruby / node / etc on GitHub Actions vs Jenkins vs CircleCI. We set up docker and that’s it. And if we are protected if we ever need to switch to another provider.
Drawbacks
Performance Overhead: In terms of the container overhead, running software directly in the runner will always be faster than running it with docker compose
on the runner. Does it really matter? Not really.
Container Build Overhead: Building images is slow. The P90 of the setup step is 15 minutes on our biggest app (when all the layers miss the cache). Fixing this is still a work in progress.
Conclusion
We have moved the complexity around within the actions jobs. Without docker there might be complexity in ensuring all required packages are present to test the app. With docker you get that for free but instead you have to worry about setting up and building the image. This made sense for us, it might not for you.
I left a lot of the detail out of my summary but we have about 25 web apps under us of varying sizes that we support this way. Beyond the initial setup the files in each app repository stay fairly static and developers largely get to ignore them. CI usually just works.
That was a super brief overview of our CI and in future posts I will go a bit more in depth on how we provide a few cool features namely:
- variably size parallelization of runs via input:
runner-count: 8
- providing arbitrary commands to get run via CI:
[{"cmd": "some-command"}]
- doing all of the above with SimpleCov and the
parallel_test
gem