Setting Up WoodPeckerCI Agents with Docker in Docker
2022-12-29
A while back I setup Drone-CI for some internal projects I run at my house to simplify image builds & testing for the various personal projects I maintain
Shortly after I realized that Drone-CI 2 wasn’t really free as I assumed it was, after about 5000 builds it will complain about the “free trial license” being exceeded and to continue running it means compiling and building it manually with atleast the nolimit tag
I was no-where near that 5000 limit, and rebulding the mainline docker image with the nolimit tag could work if I either just push out auto-builds every week, or kept an eye out for new releases myself, but ultimately this product isn’t for one-off developers such as myself and would be a hassle to keep working and keep it secure.
So I had a choice, I could use the fork of Drone-CI 0.8 floating about called WoodpeckerCI (the last OSS release of Drone-CI before they changed the license) or I could use something more traditional like Jenkins that has been around for years and shouldn’t be going anywhere anytime soon.
I liked the prospect of stability with jenkins since it’s not likely to disappear due to a lack of funding for example.
So I tried it as an experiment for a bit, I was unable to get my head wrapped around how to make it work properly with my mainly docker focused workflow however.
So recently I’ve setup Woodpecker finally for my Gitea server and it’s been wonderful over the last week or two.
The Docker-compose sample
Note this is an additional layer of “sort-of” security, this allows you to isolate your main system’s docker daemon from the once the CI server will use. Your best off making a purpose built system or VM for this (or better yet spin up temporary boxes as needed) But as a quick additional layer without building a whole VM this isn’t a bad way to do it.
version: '2'
services:
woodpecker-dind:
image: docker.io/library/docker:dind
privileged: true
restart: unless-stopped
volumes:
- docker-certs:/certs
- docker-certs-client:/certs/client
networks:
- woodpecker-dind-net
woodpecker-agent:
image: docker.io/woodpeckerci/woodpecker-agent:latest
container_name: woodpecker-agent
restart: unless-stopped
command: agent
depends_on:
- woodpecker-dind
networks:
- woodpecker-dind-net
volumes:
- docker-certs-client:/certs/client
environment:
- "DOCKER_HOST=tcp://woodpecker-dind:2376"
- "DOCKER_TLS_CERTDIR=/certs"
- "DOCKER_TLS_VERIFY=1"
- "DOCKER_CERT_PATH=/certs/client"
- "WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=true"
- "WOODPECKER_SERVER=ci-server.example.com:9000"
- "WOODPECKER_AGENT_SECRET=NOTAREALSECRETPLEASEDONTUSEME"
- "WOODPECKER_HOSTNAME=woodpecker-agent"
- "WOODPECKER_BACKEND=docker"
volumes:
docker-certs:
docker-certs-client:
networks:
woodpecker-dind-net:
The secret sauce here is the docker certificates in the volumes, and the DOCKER
host variables
This is a tip/trick I picked up from the gitlab-runner days, the dind image will allow communication to port 2376
as long as the client certificates in /certs/client
are used by the end client
so We inform the docker processes in woodpecker to validate via TLS and to use the client certificates,
There could be a problem if we just mapped /certs
to one volume and called it a day so instead we map two volumes so the dind daemon can generate the server side private certificates and the private client certificates separately so we can expose only the parts we need to, to woodpecker
Other than that this should work for most pipelines that involve docker builders (Dind within dind seems to work from my limited testing, especially with the plugins/docker image payload)