I’m a big proponent of Kubernetes and I figured how to run Gitlab runners in Kubernetes. This enables CI pipelines for my Gitlab instance and scales easily!
To accomplish this:
- I needed to use the gitlab-runner cli to register a runner
- I needed a custom Docker image to support SSL communication with Gitlab, Kubernetes, and the artifact store (Nexus or Artifactory)
- I needed to deploy the executor to Kubernetes
- I needed to configure the executor which would create runners as pipelines were needed
- I needed to handle the secrets referring to access to my artifact store or my Kubernetes cluster
In the end, I have an architecture like this:
+----------+ +-----------------+ +-----------------+
| | polls | | spawns | |
| Gitlab +<---------+ Gitlab Runner +--------->+ Gitlab Runner |
| | | (executor) | | (runner-pod) |
+----------+ | | | |
+-----+-------+---+ +-----------------+
| |
| reads |
v v
+---------+-+ +-+-----------+
| Secrets | | ConfigMap |
+-----------+ +-------------+
Registering a Runner
1. Install the gitlab-runner CLI
In order to define a gitlab-runner, you’ll need to install the gitlab-runner CLI locally.
Tip: For mac, you can use
brew install gitlab-runner
2. Register your runner
Next, you’ll need to register a runner that can communicate with Gitlab. Run gitlab-runner register and answer the prompts.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):https://gitlab.clearthehaze.comPlease enter the gitlab-ci token for this runner: [enter token found in gitlab project’s CI settings page][TOKEN]Please enter the gitlab-ci description for this runner:Helpful description for runner, shown in GitlabPlease enter the gitlab-ci tags for this runner (comma separated):Tags which will limit execution to stages which have these tagsPlease enter the executor: ssh, docker+machine, kubernetes, docker-ssh, parallels, virtualbox, docker-ssh+machine, docker, shell:kubernetes
3. Get the token for your runner
When you deploy your runner to Kubernetes, you’ll need to know the token your runner is using.
Run gitlab-runner list to see all registered runners, and find the token in for your runner.
Customizing the Docker Image
In order to run Gitlab CI in Kubernetes, you’ll need a Docker image. Gitlab offers an image already, and I based my custom image from it. I next needed to inject my custom certificates to enable TLS for my internal services, Gitlab and Kubernetes API Server.
FROM gitlab/gitlab-runner:v13.5.0 # cert for talking to gitlab SSL COPY gitlab.clearthehaze.com.crt /etc/gitlab-runner/certs/ # certs for talking to K8s API COPY kubernetes-apiserver-servers.crt /usr/local/share/ca-certificates/ # This command is already in the image and will # refresh the certs index for the runner to use RUN update-ca-certificates --fresh >/dev/null WORKDIR / RUN ln -s /etc/gitlab-runner .gitlab-runner
Deployment to Kubernetes
Secrets
I deployed multiple secrets to mount into the deployment (further below) and used within the ConfigMap (which follows this). This allowed me to separate the various credentials for npm, maven, etc, to support publishing artifacts.
ConfigMap
This ConfigMap will configure the gitlab runner by providing the config.toml file. There are a few items of note:
concurrentsets the max number of runner pods to spawn while executing jobs, and provides a single value control for scaling CIurlmust point to your gitlab instancetokencomes from registration abovepre_build_scriptallows you to inject shell execution to run at the beginning of every job this executor processes. In this case, I wanted an executor which handles any kind of build job, so I configured a variety of libraries/tools in one place. If you wanted to have multiple executors for different tags, such as anodeorjavarunner, you could slim down the configuration in this script. But the cost would be maintaining multiple executor deployments rather than a single one.bearer_tokenis a secret your executor will need to make authenticated requests to the Kubernetes API server.- All of the memory and cpu requests and limits I keep the same for the containers in the runner pod. The memory limit is one to keep an eye on. If you see
fatal: write error: Out of memorythen this needs to be higher.
apiVersion: v1
kind: ConfigMap
metadata:
namespace: myteam-ci
name: gitlab-runner-config
data:
KUBERNETES_NAMESPACE: myteam-ci
GET_SOURCES_ATTEMPTS: "3"
config.toml: |
concurrent = 4
[[runners]]
name = "my-gitlab-runner"
url = "https://gitlab.clearthehaze.com/"
token = "1111111111111"
executor = "kubernetes"
output_limit = 409600
pre_build_script = """
echo 'Configuring environment'
if [ -x "$(command -v npm)" ]; then
npm config set _auth $(cat /npm-secrets/auth)
npm config set email $(cat /npm-secrets/email)
fi
if [ -x "$(command -v ruby)" ]; then
cp /gem-secrets/credentials ~/.gem
fi
if [ -x "$(command -v mvn)" ]; then
mkdir -p ~/.m2
cp /mvn-secrets/settings ~/.m2/settings.xml
fi
if [ -x "$(command -v kubectl)" ]; then
echo "Authorizing kubectl"
kubectl config set-credentials $(cat /k8s-secrets/user) --token=$(cat /k8s-secrets/token) --certificate-authority={{ .Values.k8sca }}
kubectl config set-context dc --cluster=dc --user=$(cat /k8s-secrets/user)
kubectl config set-context jpn --cluster=jpn --user=$(cat /k8s-secrets/user)
fi
if [ -x "$(command -v git)" ]; then
SSH_PRIVATE_KEY=$(cat /git-secrets/privateKey)
SSH_ORIGIN="[email protected]:${CI_PROJECT_URL:35:${#CI_PROJECT_URL}}.git"
which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
eval $(ssh-agent -s)
mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
git config --global user.email [email protected]
git config --global user.name [email protected]
git config --global push.default simple
fi
"""
[runners.cache]
Type = "s3"
BucketName = "my-ci-caches"
Shared = true
ServerAddress = "{{ .Values.cache.ServerAddress }}"
AccessKey = "my-access-key"
SecretKey = "secret"
Insecure = true
[runners.kubernetes]
namespace = "myteam-ci"
image = "busybox"
privileged = false
host = "https://kube-apiserver-dc.clearthehaze.com"
service_account = "gitlab-runner"
cpu_limit = "300"
memory_limit = "1000Mi"
service_cpu_limit = "300"
service_memory_limit = "1000Mi"
helper_cpu_limit = "300"
helper_memory_limit = "1000Mi"
cpu_request = "100m"
memory_request = "200Mi"
service_cpu_request = "100m"
service_memory_request = "200Mi"
helper_cpu_request = "100m"
helper_memory_request = "200Mi"
bearer_token = "thisisasecret"
[[runners.kubernetes.volumes.secret]]
name = "gitlab-npm-publish"
mount_path = "/npm-secrets"
read_only = true
[[runners.kubernetes.volumes.secret]]
name = "gitlab-gem-publish"
mount_path = "/gem-secrets"
read_only = true
[[runners.kubernetes.volumes.secret]]
name = "gitlab-mvn-publish"
mount_path = "/mvn-secrets"
read_only = true
[[runners.kubernetes.volumes.secret]]
name = "gitlab-k8s-deploy"
mount_path = "/k8s-secrets"
read_only = true
[[runners.kubernetes.volumes.secret]]
name = "gitlab-git"
mount_path = "/git-secrets"
read_only = true
Deployment
The few things to take note of here is:
- This deploys the executor, which makes requests to Gitlab to see if there’s any work for it to do
- Spawns runner pods to run the jobs to which it is assigned
- There only needs to be 1 replica, since it will asynchronously spawn runner pods
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-gitlab-runner
spec:
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
name: my-gitlab-runner
template:
metadata:
labels:
name: my-gitlab-runner
spec:
serviceAccountName: gitlab-runner
automountServiceAccountToken: false
securityContext:
runAsUser: 1000
containers:
- args:
- run
image: my-gitlab-runner-image:1.0.0
imagePullPolicy: Always
name: gitlab-runner
volumeMounts:
- mountPath: /etc/gitlab-runner
name: config
# Since the deployment simply monitors and deploys new pods, the limits
# here do not affect the limits of the pods running the build.
# For more info: https://kubernetes.io/docs/user-guide/compute-resources/
resources:
limits:
cpu: "100m"
memory: "100Mi"
requests:
cpu: "100m"
memory: "100Mi"
volumes:
- configMap:
name: gitlab-runner-config
name: config
Using Your New Runner
In your project settings (Settings -> CI/CD), you’ll now see the runner attached to your project. You’ll have to edit the runner to deselect the lock on it, which will then allow you to use the runner on other projects.
