I’m a big proponent of Kubernetes and I figured how to run Gitlab runners in Kubernetes. This enables CI pipelines for my Gitlab instance and scales easily!
To accomplish this:
- I needed to use the gitlab-runner cli to register a runner
- I needed a custom Docker image to support SSL communication with Gitlab, Kubernetes, and the artifact store (Nexus or Artifactory)
- I needed to deploy the executor to Kubernetes
- I needed to configure the executor which would create runners as pipelines were needed
- I needed to handle the secrets referring to access to my artifact store or my Kubernetes cluster
In the end, I have an architecture like this:
+----------+ +-----------------+ +-----------------+
| | polls | | spawns | |
| Gitlab +<---------+ Gitlab Runner +--------->+ Gitlab Runner |
| | | (executor) | | (runner-pod) |
+----------+ | | | |
+-----+-------+---+ +-----------------+
| |
| reads |
v v
+---------+-+ +-+-----------+
| Secrets | | ConfigMap |
+-----------+ +-------------+
Registering a Runner
1. Install the gitlab-runner CLI
In order to define a gitlab-runner, you’ll need to install the gitlab-runner CLI locally.
Tip: For mac, you can use
brew install gitlab-runner
2. Register your runner
Next, you’ll need to register a runner that can communicate with Gitlab. Run gitlab-runner register
and answer the prompts.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.clearthehaze.comPlease enter the gitlab-ci token for this runner: [enter token found in gitlab project’s CI settings page]
[TOKEN]Please enter the gitlab-ci description for this runner:
Helpful description for runner, shown in GitlabPlease enter the gitlab-ci tags for this runner (comma separated):
Tags which will limit execution to stages which have these tagsPlease enter the executor: ssh, docker+machine, kubernetes, docker-ssh, parallels, virtualbox, docker-ssh+machine, docker, shell:
kubernetes
3. Get the token for your runner
When you deploy your runner to Kubernetes, you’ll need to know the token your runner is using.
Run gitlab-runner list
to see all registered runners, and find the token in for your runner.
Customizing the Docker Image
In order to run Gitlab CI in Kubernetes, you’ll need a Docker image. Gitlab offers an image already, and I based my custom image from it. I next needed to inject my custom certificates to enable TLS for my internal services, Gitlab and Kubernetes API Server.
FROM gitlab/gitlab-runner:v13.5.0 # cert for talking to gitlab SSL COPY gitlab.clearthehaze.com.crt /etc/gitlab-runner/certs/ # certs for talking to K8s API COPY kubernetes-apiserver-servers.crt /usr/local/share/ca-certificates/ # This command is already in the image and will # refresh the certs index for the runner to use RUN update-ca-certificates --fresh >/dev/null WORKDIR / RUN ln -s /etc/gitlab-runner .gitlab-runner
Deployment to Kubernetes
Secrets
I deployed multiple secrets to mount into the deployment (further below) and used within the ConfigMap (which follows this). This allowed me to separate the various credentials for npm, maven, etc, to support publishing artifacts.
ConfigMap
This ConfigMap will configure the gitlab runner by providing the config.toml file. There are a few items of note:
concurrent
sets the max number of runner pods to spawn while executing jobs, and provides a single value control for scaling CIurl
must point to your gitlab instancetoken
comes from registration abovepre_build_script
allows you to inject shell execution to run at the beginning of every job this executor processes. In this case, I wanted an executor which handles any kind of build job, so I configured a variety of libraries/tools in one place. If you wanted to have multiple executors for different tags, such as anode
orjava
runner, you could slim down the configuration in this script. But the cost would be maintaining multiple executor deployments rather than a single one.bearer_token
is a secret your executor will need to make authenticated requests to the Kubernetes API server.- All of the memory and cpu requests and limits I keep the same for the containers in the runner pod. The memory limit is one to keep an eye on. If you see
fatal: write error: Out of memory
then this needs to be higher.
apiVersion: v1 kind: ConfigMap metadata: namespace: myteam-ci name: gitlab-runner-config data: KUBERNETES_NAMESPACE: myteam-ci GET_SOURCES_ATTEMPTS: "3" config.toml: | concurrent = 4 [[runners]] name = "my-gitlab-runner" url = "https://gitlab.clearthehaze.com/" token = "1111111111111" executor = "kubernetes" output_limit = 409600 pre_build_script = """ echo 'Configuring environment' if [ -x "$(command -v npm)" ]; then npm config set _auth $(cat /npm-secrets/auth) npm config set email $(cat /npm-secrets/email) fi if [ -x "$(command -v ruby)" ]; then cp /gem-secrets/credentials ~/.gem fi if [ -x "$(command -v mvn)" ]; then mkdir -p ~/.m2 cp /mvn-secrets/settings ~/.m2/settings.xml fi if [ -x "$(command -v kubectl)" ]; then echo "Authorizing kubectl" kubectl config set-credentials $(cat /k8s-secrets/user) --token=$(cat /k8s-secrets/token) --certificate-authority={{ .Values.k8sca }} kubectl config set-context dc --cluster=dc --user=$(cat /k8s-secrets/user) kubectl config set-context jpn --cluster=jpn --user=$(cat /k8s-secrets/user) fi if [ -x "$(command -v git)" ]; then SSH_PRIVATE_KEY=$(cat /git-secrets/privateKey) SSH_ORIGIN="[email protected]:${CI_PROJECT_URL:35:${#CI_PROJECT_URL}}.git" which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y ) eval $(ssh-agent -s) mkdir -p ~/.ssh chmod 700 ~/.ssh ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d) echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config git config --global user.email [email protected] git config --global user.name [email protected] git config --global push.default simple fi """ [runners.cache] Type = "s3" BucketName = "my-ci-caches" Shared = true ServerAddress = "{{ .Values.cache.ServerAddress }}" AccessKey = "my-access-key" SecretKey = "secret" Insecure = true [runners.kubernetes] namespace = "myteam-ci" image = "busybox" privileged = false host = "https://kube-apiserver-dc.clearthehaze.com" service_account = "gitlab-runner" cpu_limit = "300" memory_limit = "1000Mi" service_cpu_limit = "300" service_memory_limit = "1000Mi" helper_cpu_limit = "300" helper_memory_limit = "1000Mi" cpu_request = "100m" memory_request = "200Mi" service_cpu_request = "100m" service_memory_request = "200Mi" helper_cpu_request = "100m" helper_memory_request = "200Mi" bearer_token = "thisisasecret" [[runners.kubernetes.volumes.secret]] name = "gitlab-npm-publish" mount_path = "/npm-secrets" read_only = true [[runners.kubernetes.volumes.secret]] name = "gitlab-gem-publish" mount_path = "/gem-secrets" read_only = true [[runners.kubernetes.volumes.secret]] name = "gitlab-mvn-publish" mount_path = "/mvn-secrets" read_only = true [[runners.kubernetes.volumes.secret]] name = "gitlab-k8s-deploy" mount_path = "/k8s-secrets" read_only = true [[runners.kubernetes.volumes.secret]] name = "gitlab-git" mount_path = "/git-secrets" read_only = true
Deployment
The few things to take note of here is:
- This deploys the executor, which makes requests to Gitlab to see if there’s any work for it to do
- Spawns runner pods to run the jobs to which it is assigned
- There only needs to be 1 replica, since it will asynchronously spawn runner pods
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-gitlab-runner spec: replicas: 1 revisionHistoryLimit: 0 selector: matchLabels: name: my-gitlab-runner template: metadata: labels: name: my-gitlab-runner spec: serviceAccountName: gitlab-runner automountServiceAccountToken: false securityContext: runAsUser: 1000 containers: - args: - run image: my-gitlab-runner-image:1.0.0 imagePullPolicy: Always name: gitlab-runner volumeMounts: - mountPath: /etc/gitlab-runner name: config # Since the deployment simply monitors and deploys new pods, the limits # here do not affect the limits of the pods running the build. # For more info: https://kubernetes.io/docs/user-guide/compute-resources/ resources: limits: cpu: "100m" memory: "100Mi" requests: cpu: "100m" memory: "100Mi" volumes: - configMap: name: gitlab-runner-config name: config
Using Your New Runner
In your project settings (Settings -> CI/CD), you’ll now see the runner attached to your project. You’ll have to edit the runner to deselect the lock on it, which will then allow you to use the runner on other projects.
![](http://clearthehaze.com/wp-content/uploads/2020/11/2020-11-locked-runner-half.png)