The changes matrix
Overview
One of my most used Gitlab CI feature is the changes rule. This allows you to run a job or group of jobs based on the changed paths.
Here's a simple example of how rules:changes can be used in a pipeline:
# Run mkdocs when anything under the docs/ directory is changed
docs:
image: squidfunk/mkdocs-material
script: mkdocs build
rules:
- changes:
- docs/**/*
# Run pytest when anything under the src/ directory is changed
tests:
image: python
before_script: pip install pytest
script: pytest
rules:
- changes:
- src/**/*
# Build the Docker image when the Dockerfile is changed
docker:
image: docker:cli
script: docker build
rules:
- changes:
- Dockerfile
I recently discovered a neat trick for conditionally running the same job based on changes in multiple paths.
The problem
I have a repository with a number of Docker image definitions. In my case, these images each have their own context directory. My repository structure looks something like this:
/
|__ src/
|
|__ image1/
| |__ Dockerfile
|
|__ image2/
| |__ Dockerfile
| |__ package.json
|
|__ image3/
|__ Dockerfile
|__ requirements.txt
I want to run a Docker build whenever each respective image is changed, and then push that image to my registry. But of course, it wouldn't make sense to build and publish all 3 images every time one of them is modified. When I make a change to image1, I only want image1 to be built and published. And the same for the other images.
I suppose I could have a separate job for each image:
image1-build:
image: docker:cli
script:
- docker build -t gitlab.org/my-user/my-repo/image1 src/image1/
- docker push gitlab.org/my-user/my-repo/image1
rules:
- changes:
- src/image1/**/*
image2-build:
image: docker:cli
script:
- docker build -t gitlab.org/my-user/my-repo/image2 src/image2/
- docker push gitlab.org/my-user/my-repo/image2
rules:
- changes:
- src/image2/**/*
image3-build:
image: docker:cli
script:
- docker build -t gitlab.org/my-user/my-repo/image3 src/image3/
- docker push gitlab.org/my-user/my-repo/image3
rules:
- changes:
- src/image3/**/*
As you can see, this is cumbersome, not very scalable, and results in quite a bit of duplicated code.
The solution
We can solve this with clever use of the matrix configuration. But first, you'll need to understand what matrix is used for.
What is matrix?
Typically matrix is used to run multiple jobs in parallel with varying values. See this example from Gitlab's documentation:
This creates 3 separate jobs, outputting the respective value:
Test value: value1
Test value: value2
Test value: value3
How does this help us?
With parallel:matrix we can create individual jobs for each of our Docker contexts:
docker:
image: docker:cli
parallel:
matrix:
- IMAGE:
- image1
- image2
- image3
script:
- docker build -t gitlab.org/my-user/my-repo/$IMAGE src/$IMAGE/
- docker push gitlab.org/my-user/my-repo/$IMAGE
rules:
- changes:
- src/$IMAGE/**/*
What does this do?
This job creates 3 parallel jobs: one for image1, image2, and image3. However the $IMAGE variable is also used in rules:changes to determine whether or not to run the job.
Say we modified src/image2/Dockerfile in our most recent commit. When Gitlab evaluates the pipeline, it sees that we have a matrix of the 3 values under the $IMAGE variable. It then evaluates the rules for each job: