clear the haze

My adventures in software engineering

How to Deploy Gitlab CI Runners to Kubernetes

I’m a big proponent of Kubernetes and I figured how to run Gitlab runners in Kubernetes. This enables CI pipelines for my Gitlab instance and scales easily!

To accomplish this:

  • I needed to use the gitlab-runner cli to register a runner
  • I needed a custom Docker image to support SSL communication with Gitlab, Kubernetes, and the artifact store (Nexus or Artifactory)
  • I needed to deploy the executor to Kubernetes
  • I needed to configure the executor which would create runners as pipelines were needed
  • I needed to handle the secrets referring to access to my artifact store or my Kubernetes cluster

In the end, I have an architecture like this:

+----------+          +-----------------+          +-----------------+
|          |  polls   |                 |  spawns  |                 |
|  Gitlab  +<---------+  Gitlab Runner  +--------->+  Gitlab Runner  |
|          |          |   (executor)    |          |   (runner-pod)  |
+----------+          |                 |          |                 |
                      +-----+-------+---+          +-----------------+
                            |       |
                            | reads |
                            v       v
                  +---------+-+   +-+-----------+
                  |  Secrets  |   |  ConfigMap  |
                  +-----------+   +-------------+

Registering a Runner

1. Install the gitlab-runner CLI

In order to define a gitlab-runner, you’ll need to install the gitlab-runner CLI locally.

Tip: For mac, you can use brew install gitlab-runner

2. Register your runner

Next, you’ll need to register a runner that can communicate with Gitlab. Run gitlab-runner register and answer the prompts.

  • Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://gitlab.clearthehaze.com
  • Please enter the gitlab-ci token for this runner: [enter token found in gitlab project’s CI settings page] [TOKEN]
  • Please enter the gitlab-ci description for this runner: Helpful description for runner, shown in Gitlab
  • Please enter the gitlab-ci tags for this runner (comma separated): Tags which will limit execution to stages which have these tags
  • Please enter the executor: ssh, docker+machine, kubernetes, docker-ssh, parallels, virtualbox, docker-ssh+machine, docker, shell: kubernetes

3. Get the token for your runner

When you deploy your runner to Kubernetes, you’ll need to know the token your runner is using.

Run gitlab-runner list to see all registered runners, and find the token in for your runner.

Customizing the Docker Image

In order to run Gitlab CI in Kubernetes, you’ll need a Docker image. Gitlab offers an image already, and I based my custom image from it. I next needed to inject my custom certificates to enable TLS for my internal services, Gitlab and Kubernetes API Server.

FROM gitlab/gitlab-runner:v13.5.0

# cert for talking to gitlab SSL
COPY gitlab.clearthehaze.com.crt /etc/gitlab-runner/certs/

# certs for talking to K8s API
COPY kubernetes-apiserver-servers.crt /usr/local/share/ca-certificates/

# This command is already in the image and will 
# refresh the certs index for the runner to use
RUN update-ca-certificates --fresh >/dev/null

WORKDIR /
RUN ln -s /etc/gitlab-runner .gitlab-runner

Deployment to Kubernetes

Secrets

I deployed multiple secrets to mount into the deployment (further below) and used within the ConfigMap (which follows this). This allowed me to separate the various credentials for npm, maven, etc, to support publishing artifacts.

ConfigMap

This ConfigMap will configure the gitlab runner by providing the config.toml file. There are a few items of note:

  • concurrent sets the max number of runner pods to spawn while executing jobs, and provides a single value control for scaling CI
  • url must point to your gitlab instance
  • token comes from registration above
  • pre_build_script allows you to inject shell execution to run at the beginning of every job this executor processes. In this case, I wanted an executor which handles any kind of build job, so I configured a variety of libraries/tools in one place. If you wanted to have multiple executors for different tags, such as a node or java runner, you could slim down the configuration in this script. But the cost would be maintaining multiple executor deployments rather than a single one.
  • bearer_token is a secret your executor will need to make authenticated requests to the Kubernetes API server.
  • All of the memory and cpu requests and limits I keep the same for the containers in the runner pod. The memory limit is one to keep an eye on. If you see fatal: write error: Out of memory then this needs to be higher.
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: myteam-ci
  name: gitlab-runner-config
data:
  KUBERNETES_NAMESPACE: myteam-ci
  GET_SOURCES_ATTEMPTS: "3"

  config.toml: |
    concurrent = 4

    [[runners]]
      name = "my-gitlab-runner"
      url = "https://gitlab.clearthehaze.com/"
      token = "1111111111111"
      executor = "kubernetes"
      output_limit = 409600
      pre_build_script = """
        echo 'Configuring environment'
        if [ -x "$(command -v npm)" ]; then
          npm config set _auth $(cat /npm-secrets/auth)
          npm config set email $(cat /npm-secrets/email)
        fi
        if [ -x "$(command -v ruby)" ]; then
          cp /gem-secrets/credentials ~/.gem
        fi
        if [ -x "$(command -v mvn)" ]; then
          mkdir -p ~/.m2
          cp /mvn-secrets/settings ~/.m2/settings.xml
        fi
        if [ -x "$(command -v kubectl)" ]; then
          echo "Authorizing kubectl"
          kubectl config set-credentials $(cat /k8s-secrets/user) --token=$(cat /k8s-secrets/token) --certificate-authority={{ .Values.k8sca }}
          kubectl config set-context dc --cluster=dc --user=$(cat /k8s-secrets/user)
          kubectl config set-context jpn --cluster=jpn --user=$(cat /k8s-secrets/user)
        fi
        if [ -x "$(command -v git)" ]; then 
          SSH_PRIVATE_KEY=$(cat /git-secrets/privateKey)
          SSH_ORIGIN="[email protected]:${CI_PROJECT_URL:35:${#CI_PROJECT_URL}}.git"
          which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
          eval $(ssh-agent -s)
          mkdir -p ~/.ssh
          chmod 700 ~/.ssh
          ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
          echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
          git config --global user.email [email protected]
          git config --global user.name [email protected]
          git config --global push.default simple
        fi
      """
      [runners.cache]
        Type = "s3"
        BucketName = "my-ci-caches"
        Shared = true
        ServerAddress = "{{ .Values.cache.ServerAddress }}"
        AccessKey = "my-access-key"
        SecretKey = "secret"
        Insecure = true
      [runners.kubernetes]
        namespace = "myteam-ci"
        image = "busybox"
        privileged = false
        host = "https://kube-apiserver-dc.clearthehaze.com"
        service_account = "gitlab-runner"
        cpu_limit = "300"
        memory_limit = "1000Mi"
        service_cpu_limit = "300"
        service_memory_limit = "1000Mi"
        helper_cpu_limit = "300"
        helper_memory_limit = "1000Mi"
        cpu_request = "100m"
        memory_request = "200Mi"
        service_cpu_request = "100m"
        service_memory_request = "200Mi"
        helper_cpu_request = "100m"
        helper_memory_request = "200Mi"
        bearer_token = "thisisasecret"
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-npm-publish"
          mount_path = "/npm-secrets"
          read_only = true
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-gem-publish"
          mount_path = "/gem-secrets"
          read_only = true
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-mvn-publish"
          mount_path = "/mvn-secrets"
          read_only = true
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-k8s-deploy"
          mount_path = "/k8s-secrets"
          read_only = true
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-git"
          mount_path = "/git-secrets"
          read_only = true

Deployment

The few things to take note of here is:

  • This deploys the executor, which makes requests to Gitlab to see if there’s any work for it to do
  • Spawns runner pods to run the jobs to which it is assigned
  • There only needs to be 1 replica, since it will asynchronously spawn runner pods
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-gitlab-runner
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      name: my-gitlab-runner
  template:
    metadata:
      labels:
        name: my-gitlab-runner
    spec:
      serviceAccountName: gitlab-runner
      automountServiceAccountToken: false
      securityContext: 
        runAsUser: 1000
      containers:
      - args:
        - run
        image: my-gitlab-runner-image:1.0.0
        imagePullPolicy: Always
        name: gitlab-runner
        volumeMounts:
        - mountPath: /etc/gitlab-runner
          name: config
        # Since the deployment simply monitors and deploys new pods, the limits
        # here do not affect the limits of the pods running the build.
        # For more info: https://kubernetes.io/docs/user-guide/compute-resources/
        resources:
         limits:
           cpu: "100m"
           memory: "100Mi"
         requests:
           cpu: "100m"
           memory: "100Mi"
      volumes:
      - configMap:
          name: gitlab-runner-config
        name: config

Using Your New Runner

In your project settings (Settings -> CI/CD), you’ll now see the runner attached to your project. You’ll have to edit the runner to deselect the lock on it, which will then allow you to use the runner on other projects.

Leave a Reply

Your email address will not be published. Required fields are marked *