Setting Up WoodPeckerCI Agents with Docker in Docker

2022-12-29

A while back I setup Drone-CI for some internal projects I run at my house to simplify image builds & testing for the various personal projects I maintain

Shortly after I realized that Drone-CI 2 wasn’t really free as I assumed it was, after about 5000 builds it will complain about the “free trial license” being exceeded and to continue running it means compiling and building it manually with atleast the nolimit tag

I was no-where near that 5000 limit, and rebulding the mainline docker image with the nolimit tag could work if I either just push out auto-builds every week, or kept an eye out for new releases myself, but ultimately this product isn’t for one-off developers such as myself and would be a hassle to keep working and keep it secure.

So I had a choice, I could use the fork of Drone-CI 0.8 floating about called WoodpeckerCI (the last OSS release of Drone-CI before they changed the license) or I could use something more traditional like Jenkins that has been around for years and shouldn’t be going anywhere anytime soon.

I liked the prospect of stability with jenkins since it’s not likely to disappear due to a lack of funding for example.

So I tried it as an experiment for a bit, I was unable to get my head wrapped around how to make it work properly with my mainly docker focused workflow however.

So recently I’ve setup Woodpecker finally for my Gitea server and it’s been wonderful over the last week or two.

The Docker-compose sample

Note this is an additional layer of “sort-of” security, this allows you to isolate your main system’s docker daemon from the once the CI server will use. Your best off making a purpose built system or VM for this (or better yet spin up temporary boxes as needed) But as a quick additional layer without building a whole VM this isn’t a bad way to do it.

version: '2'
services:
  woodpecker-dind:
    image: docker.io/library/docker:dind
    privileged: true
    restart: unless-stopped
    volumes:
      - docker-certs:/certs
      - docker-certs-client:/certs/client
    networks:
      - woodpecker-dind-net
  woodpecker-agent:
    image: docker.io/woodpeckerci/woodpecker-agent:latest
    container_name: woodpecker-agent
    restart: unless-stopped
    command: agent
    depends_on:
      - woodpecker-dind
    networks:
      - woodpecker-dind-net
    volumes:
      - docker-certs-client:/certs/client
    environment:
      - "DOCKER_HOST=tcp://woodpecker-dind:2376"
      - "DOCKER_TLS_CERTDIR=/certs"
      - "DOCKER_TLS_VERIFY=1"
      - "DOCKER_CERT_PATH=/certs/client"
      - "WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=true"
      - "WOODPECKER_SERVER=ci-server.example.com:9000"
      - "WOODPECKER_AGENT_SECRET=NOTAREALSECRETPLEASEDONTUSEME"
      - "WOODPECKER_HOSTNAME=woodpecker-agent"
      - "WOODPECKER_BACKEND=docker"
volumes:
  docker-certs:
  docker-certs-client:
networks:
  woodpecker-dind-net:

The secret sauce here is the docker certificates in the volumes, and the DOCKER host variables

This is a tip/trick I picked up from the gitlab-runner days, the dind image will allow communication to port 2376 as long as the client certificates in /certs/client are used by the end client

so We inform the docker processes in woodpecker to validate via TLS and to use the client certificates,

There could be a problem if we just mapped /certs to one volume and called it a day so instead we map two volumes so the dind daemon can generate the server side private certificates and the private client certificates separately so we can expose only the parts we need to, to woodpecker

Other than that this should work for most pipelines that involve docker builders (Dind within dind seems to work from my limited testing, especially with the plugins/docker image payload)