Multiple runners

[gbonnefille]

I'm actively looking for a way to have a single gilab-runner instance running multiple «runners» in order to create a single shared runner for dedicated groups/projects (cf. https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/230) while controlling the number of total jobs via a single concurrent property.

Do you think it make sense to improve your operator with a new CRD (for example SharedRunner) allowing to define multiple registrations while keeping a single gitlab-runner?

[alekc]

Is it even supported by the official kubernetes executor? On top of my head (I am on the phone ATM) its limited to one reg,thus requiring multiple runners if the split in reg/tags/config is needed.

…

On Tue, Oct 4, 2022, 17:21 Guilhem Bonnefille ***@***.***> wrote: I'm actively looking for a way to have a single gilab-runner instance running multiple «runners» in order to create a single shared runner for dedicated groups/projects (cf. https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/230) while controlling the number of total jobs via a single concurrent property. Do you think it make sense to improve your operator with a new CRD (for example SharedRunner) allowing to define multiple registrations while keeping a single gitlab-runner? — Reply to this email directly, view it on GitHub <#17>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACJ5V4EZTBL7IGFBQOFPWTWBRKQPANCNFSM6AAAAAAQ4XSQP4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[alekc]

@gbonnefille
I've checked the docs and it looks like you are after this scenario: https://docs.gitlab.com/runner/fleet_scaling/#intermediate-configuration-one-runner-multiple-workers

concurrent = 3

[[runners]]
  name = "instance_level_shell_001"
  url = ""
  token = ""
  executor = "shell"

[[runners]]
  name = "instance_level_shell_002"
  url = ""
  token = ""
  executor = "shell"

[[runners]]
  name = "instance_level_shell_003"
  url = ""
  token = ""
  executor = "shell"

It might be achievable with something like

apiVersion: gitlab.k8s.alekc.dev/v1beta1
kind: MultiRunner
metadata:
  name: multi-runner
spec:
  runners:
    - registration_config:
        token: "xxx"
        tag_list:
          - group1
    - log_level: debug
      executor_config:
        image: "debian:slim"
        memory_limit: "150Mi"
        memory_request: "150Mi"
        volumes:
          config_map:
            - mount_path: /cm/
              name: test-config
          secret:
            - mount_path: /secrets/1/
              name: test-secret
            - mount_path: /secrets/2/
              name: test-secret-2
      registration_config:
        token: "xxx"
        tag_list:
          - advanced-runner

I can definitely see advantages to this, however I need to think about it (due to a lack of personal free time atm sadly).
Biggest catches to look for would be:

• dealing with a situation where one of the regs is not succesful (since the registrations would have to be done one by one)
• removal of multiple runners on crd deletion.

[gbonnefille]

I can definitely see advantages to this, however I need to think about it (due to a lack of personal free time atm sadly).

No problem, the most important is that the idea is acceptable, so I can start looking how I can contribute to that.

[gbonnefille]

Here are some though after reading code.

The most challenging part is, of course, to take into account any changes on the config.

The gitlab-runner is able to reload automatically its config if a change occurs. So, no need to look for change in the executor_config, simply rewrite again and again the configuration file.

The gitlab-runner-controler has to detect changes on registration_config. As a registration token is certainly uniq, it could serves as a key for an internal map. Base on it, it would be possible to detect addition, change or deletion of runner. Such events trigger new registration/unregistration on the gitlab.
Note: if the user simply change the registration token, keeping everything else, this would be processed as a deletion+addition.

[gbonnefille]

@alekc I also started playing with the code (nothing functional yet).
Finally, I started looking at a different solution: extending the current CRD with a runners: array.
#19

One idea is to stay as close as the Gitlab Runner configuration.

Another idea is to reuse the « root » configuration as a default configuration. When I set multiple configurations in runners: tree, I can share some details by declaring a global registration_config.

What do you think about such implementation direction?

[alekc]

I am doing some work on it as well (you can see it in PR). I had to split Runner and Multirunner in 2 different CRD/controllers.

Main reason is: as right now, Runner saves the information about latest registration tag, registration and auth tokens as a single field in the status (they are needed because otherwise we do not know if the reg spec has been changed or not). That obviously won't work with the multiple runners approach because we need to keep track of this data for every runner specified in the multirunner section.

WHile I am at it, I am doing some minor/major refactoring around components and making them as generic as possible. You can expect a working testing demo in couple of days top.

I'd rather not to touch too much existing runner because its simple and already used in some environments, so I would like to keep the change as small as possible for legacy (well, I am going to drop the secret containing token, but thats another story).

[gbonnefille]

I understand you don't want to risk breaking the legacy.
But I have the feeling that we can create a solution keeping compatibility with previous logic. I will try to work on this.

[alekc]

Hey @gbonnefille , sorry for the delay. Past days were a bit more challenging that expected.

You can try to build from the latest branch mentioned in the pr above.
so far in my tests given this object

apiVersion: gitlab.k8s.alekc.dev/v1beta1
kind: MultiRunner
metadata:
  name: multirunner-simple
spec:
  concurrent: 10
  gitlab_instance_url: https://gitlab.com/
  entries:
    - name: runner 1
      registration_config:
        tag_list:
          - dev1
          - test-gitlab-runner
        token: xxx
        description: "Runner 1desc"
      executor_config:
        memory_request: "50Mi"
    - name: runner 2
      registration_config:
        tag_list:
          - dev2
          - runner2
        token: xxx
        description: "Runner 2 desc"

A single runner (on kubernetes) is built and launched having following config:

apiVersion: v1
data:
  config.toml: |
    listen_address = ":9090"
    concurrent = 10
    check_interval = 3
    log_level = "info"

    [[runners]]
      name = "runner 1"
      limit = 10
      url = "https://gitlab.com/"
      token = "mSMjXSd-x"
      executor = "kubernetes"
      [runners.kubernetes]
        host = ""
        namespace = "default"

    [[runners]]
      name = "runner 2"
      limit = 10
      url = "https://gitlab.com/"
      token = "xx"
      executor = "kubernetes"
      [runners.kubernetes]
        host = ""
        namespace = "default"
kind: ConfigMap
metadata:
  creationTimestamp: "2022-10-19T19:49:50Z"
  name: gitlab-mrunner-multirunner-simple
  namespace: default
  ownerReferences:
  - apiVersion: gitlab.k8s.alekc.dev/v1beta1
    controller: true
    kind: MultirRunner
    name: multirunner-simple
    uid: 97867551-b586-4b4f-b2a3-249a00b17698
  resourceVersion: "84100"
  uid: 1ba25c67-0b60-4210-9dd0-402b658ff363

There are still a quiet bit of work to do before this sees the end release but, so far looks good for most aspects.

Todo list:

• Check if there is a bug related to the generation of the config, in the example above, 50Mi should have been stored in the config somewhere fixed
• Due to how the logic is structured, multiple entries need to have unique tag. Otherwise there is a registration loop present. This has to be mentioned in the docs, and checked on admission controller/logic check version
• Since multirunner and single runner now share the same Business logic, they need to be unified
• Tests
• Documentation
• Examples