Docker & GitLab-CI Pipeline Updates

2020-07-04

Docker pipeline streamlining and how to setup your own private GitLab runners

Hey guys, it’s been a while. Over the last number of months with the Covid-19 pandemic going on I’ve been focusing heavily on docker container building and cluster orchestration both professionally and personally.

Anyways this is an update to my first post on this blog from about 6 months ago.

First up will be setting up your own GitLab-ci runner that you can register to your own private group, a specific project, or setting it up as a shared runner on your own private GitLab instance.

Then after will be outlining some pipeline updates I’ve been using for personal projects.

Setting up a Gitlab-ci Runner for Docker-In-Docker execution

The first steps will be to get a separate docker system setup for this, preferably on it’s own network.

While it would be tempting to set this up as apart of your dev cluster in Kubernetes or Docker Swarm, setting up this machine involves setting up privileged mode for the gitlab-ci runner container, and any containers it spawns having privileged mode, and that is a security risk if it’s running on an important machine, if you continue from this point and still want to setup this on an important docker machine, don’t come after me for damages.

With that out of the way, we will first acquire a registration token

Acquiring a Registration Token

There are a few different spots you can get these from

If you want to register under a specific project (Not recommended), you can go under:
Project Overview -> Settings -> CI/CD -> Runners
and you will see the registration token in the Set up a specific Runner manually section of the menu under Step 3.

If you want to register under a Group/Subgroup in GitLab (preferred), you can go to:
Group Overview -> Settings -> CI/CD -> Runners
Go to the Set up a group Runner manually section, and you will see your token under Step 3.

Finally if you want to register your own shared runner for a private GitLab instance, go under:
Admin Area -> Settings -> CI/CD -> Runners
From here you can get your token, this will setup the runners as instance wide, so make sure you have registrations locked down before you set this up.

Registering Runners

The next step will be registering the runner. You can run this line here to get started:

docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

Change the /srv/gitlab-runner/config mount to whatever folder you see fit.

This will run you through the process of creating a basic config.toml file for your project

First it will ask you for the gitlab instance url you want to use (For gitlab.com this will be https://gitlab.com)

Then it will ask you for your token, a name for the runner, and it’s description

Now it will ask you for your build tags, while using docker is fine, I would recommend you slightly change the tag if your using gitlab.com (e.g docker-GroupName) since it will mix it in with the shared runners of gitlab.com by default.

It’s much easier to use a different tag vs disabling shared runners for every project (If this is for shared runners on a private instance ignore that and use generic tags)

finally it will ask the executor you wish to run, we will pick Docker

With the docker executor it will ask for a default image to run, just use docker:stable

Modifying the config.toml file for Docker in Docker builds

After you register the gitlab runner a config.toml file will appear in your config directory.

It will look similar to this by default:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "example-runner"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker.io/library/docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Under the [runners.docker] config we will want to edit the privileged line to this:
privileged = true
and edit the volumes path to:
volumes = ["/certs/client", "/cache"]

Both of these edits are so we can spin up Docker-in-Docker properly, and make sure the generated TLS certs are shared with each other

(I will say again, make sure this is on a docker system where it can do minimal harm to other projects or servers, that privileged line will automatically have all docker containers that gitlab-ci spins up on this runner in a privileged state within docker)

In the end our config will look like this:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "example-runner"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker.io/library/docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

Creating your auto-restarting runner instance

With all the config out of the way, all that’s left is spinning up an auto restarting docker container for running jobs as they queue up.

Quick and simple, use this run command (Edit the first volume map to be the proper config directory):

docker run -d --name gitlab-runner --restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

To simplify updates & maintenance, you can instead make a docker-compose file:

/srv/gitlab-runner/docker-compose.yml

version: '3.5'
services:
  gitlab-runner:
  image: docker.io/gitlab/gitlab-runner:latest
    restart: always
    volumes:
      - "/srv/gitlab-runner/config:/etc/gitlab-runner"
      - "/var/run/docker.sock:/var/run/docker.sock"

This will allow you to do docker-compose pull && docker-compose up -d to do updates easily and quickly.

But with this you now have a fully functional gitlab runner for building docker images.

Pipeline updates

Here is an up to date copy of my current autobuild yaml file

stages:
  - build
  - test
  - package
  - deploy

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker.io/library/docker:stable-dind
before_script:
  - docker info

.base:
  image:
    name: docker.io/library/docker:stable
  tags:
    - docker-tg
  before_script:
    - echo "Image repo target- $IMAGE_BUILD"
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

.commit-build:
  extends: .base
  stage: build
  variables:
    DOCKER_BUILDKIT: 1
  script:
    - docker build --pull --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_BUILD:$CI_COMMIT_SHORT_SHA" --cache-from "$IMAGE_BUILD:latest" -t "$IMAGE_BUILD:$CI_COMMIT_SHORT_SHA" .
    - docker tag "$IMAGE_BUILD:$CI_COMMIT_SHORT_SHA" "$IMAGE_BUILD:$CI_COMMIT_SHA"
    - docker push "$IMAGE_BUILD:$CI_COMMIT_SHORT_SHA"
    - docker push "$IMAGE_BUILD:$CI_COMMIT_SHA"

.move-to-latest:
  extends: .base
  stage: package
  allow_failure: false
  script:
    - docker pull "$IMAGE_BUILD:$CI_COMMIT_SHA"
    - docker tag "$IMAGE_BUILD:$CI_COMMIT_SHA" "$IMAGE_BUILD:latest"
    - docker push "$IMAGE_BUILD:latest"

.schedule-build:
  extends: .base
  stage: build
  variables:
    DOCKER_BUILDKIT: 1
  script:
    - docker build --pull --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_BUILD:$(date '+%F')" --cache-from "$IMAGE_BUILD:latest" -t "$IMAGE_BUILD:$(date '+%F')" .
    - docker tag "$IMAGE_BUILD:$(date '+%F')" "$IMAGE_BUILD:latest"
    - docker push "$IMAGE_BUILD:$(date '+%F')"
    - docker push "$IMAGE_BUILD:latest"

commit-build-master:
  extends: .commit-build
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE"
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_REF_NAME == "master"'
      when: always

commit-build-branch:
  extends: .commit-build
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME"
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_REF_NAME != "master"'
      when: always

move-to-latest-master:
  extends: .move-to-latest
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE"
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_REF_NAME == "master"'
      when: manual

move-to-latest-branch:
  extends: .move-to-latest
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME"
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_REF_NAME != "master"'
      when: manual

schedule-build-master:
  extends: .schedule-build
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == "master"'
      when: always

schedule-build-branch:
  extends: .schedule-build
  variables:
    IMAGE_BUILD: "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME != "master"'
      when: always

So from a high-level, what has changed?

  • Branch building
  • GitLab’s deprecation for only/except in favor of variable based rules
  • Usage of extends to minimize code copying/spaghetti
  • Image Caching using Docker Build-kit

Branch Building

One thing I realized shortly after releasing the first post is that there was no handling for branches, technically it worked since it used the short sha sums but it meant that any branch could be the latest image if desired.

So I mirrored what I had for scheduled builds and latest moves and applied a bit of logic using only/except and it worked when I wanted to work on a branch but not have it ruin what I had in production at the time.

GitLab’s deprecation for only/except in favor of variable based rules

Shortly afterwards GitLab announced that only/except was being deprecated in favor of it’s rules syntax.
It does suck that I had to replace the logic if I want this to work, but the new rules syntax is more flexible than the only/except syntax was.

Usage of extends to minimize code-copying

At this point since I was experimenting with rules I thought it would make sense to break down the items into template jobs (Jobs with the dot infront of the name) and extends syntax to minimize future maintenance.

One of the first things I had to do was swap $CI_REGISTRY_IMAGE with a different variable. GitLab-CI doesn’t like reassigning variables to itself
(You can’t do CI_REGISTRY_IMAGE: "$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME" as you would with bash)
So swapping it to $IMAGE_BUILD and assigning it based on if it’s a master build or a regular branch build worked out better

Image Caching using Docker Build-kit

Originally this was using the docker pull IMAGE || true and the docker build --cache-from ... structure, but with the build-kit changes included with Docker 19.03 that make caching way simpler, and since I’m only building linux containers, it made sense to swap to BuildKit.

As the BuildKit Docs say, just setting DOCKER_BUILDKIT=1 in the environment makes it the default builder for docker.

However that isn’t enough for my usecase, we need to pass a build-arg into the docker build for caching to work fully:
--build-arg BUILDKIT_INLINE_CACHE=1

This tells BuildKit to generate cache metadata for each layer of the docker build. This is so future builds can reference this data to save space/time for layers that are the same.

Before with older docker builds, if you want to cache layers, you have to pull the images(s) ahead of time
(e.g docker pull ubuntu:latest || true)
With Buildkit’s inline cache it eliminates the need to do this since it will pull layers as needed.

Planned Next Post: Multi Arch Builds

Recently I got my hands on a Raspberry Pi 4 Model-B 8G, so I’m looking at ways to do multi-arch building so I can move some of my smaller containers to the Raspberry Pi instead of leaving my Desktop on all the time.

Best solution at this time that’s stable (Stable in terms of that it’s cli interface for docker won’t be changing) would be using Docker Manifest tagging. However there are the newer options using buildx and buildkit, however since buildx is in tech preview still it will make more sense to wait till it’s ready before I dive into a post on how to set it up and use it.

Tags: #buildkit #docker #docker-compose #gitlab-ci