Skip to content

Support for custom Kubernetes PodSpec

Georgi N. Georgiev requested to merge spike-kubernetes-custom-pod-spec into main

SPIKE for gitlab#336161 (closed).

Introduction

This MR explores the possibility to pass a PodSpec in either JSON or YAML format inside the config.toml of GitLab Runner. This will allow power users to modify the final PodObject that is sent to the Kubernetes master for scheduling. This also circumvents the need for us to implement all possible options supported by the Kubernetes API, e.g. Support for Pods Affinity and Anti-Affinity which we recently merged.

Implementation

I went with using the same approach that is used with kubectl to patch objects.

In short, in config.toml users can specify a patch that will be applied on top of the final PodObject, allowing any part of the object to be rewritten.

Simple example

The following will apply the hostname field to the final PodSpec object.

  [runners.kubernetes]
    [[runners.kubernetes.pod_spec]]
      patch = '''
        hostname: "custom-pod-hostname"
      '''

Simple example with JSON

The patch can be a valid JSON or YAML string, since the end result is always converted to JSON.

  [runners.kubernetes]
    [[runners.kubernetes.pod_spec]]
      patch = '''
        {"hostname": "custom-pod-hostname"}
      '''
Note: We can also have the patch pointing to a file, relative to the config.toml file.

Merge strategies

There are 3 merge strategies

  • merge: Applies a simple key-value replacement. RFC here
  • json: Follows the JSON Patch specification. Allows for fine-grained control over objects and arrays. RFC here
  • strategic merge (default): Each field that is part of the Kubernetes API objects has a patchStrategy annotation that denotes how changes to this object will be handled. For example, the initContainers slice will be merged instead of replaced when a new container is specified in the patch.

Example with adding a new initContainer

    [[runners.kubernetes.pod_spec]]
      patch = '''
        initContainers:
         - command:
           - sh
           name: new-init-container
      '''
      patch_type = "strategic"

Multiple patches

Multiple patches can be specified and will be applied in order. This makes sense in case one syntax makes sense over the other for specific fields. For example, strategic merge will be the best strategy most of the time, but maybe json patch can be utilised to completely replace a slice field which is marked for merge.

[[runners]]
  [runners.kubernetes]
    [[runners.kubernetes.pod_spec]]
      patch = '''
        hostname: "custom-pod-hostname"
      '''
      patch_type = "merge"
    [[runners.kubernetes.pod_spec]]
      patch = '''
        subdomain: "subdomain"
      '''
      patch_type = "strategic"
    [[runners.kubernetes.pod_spec]]
      patch = '''
        [{"op": "replace", "path": "/terminationGracePeriodSeconds", "value": 60}]
      '''
      patch_type = "json"

Thoughts

I personally like the solution and approach. The technical parts are clean and easy to understand enough that as long as we are OK with giving away the power in and of itself I see no other reason to not implement this feature.

Suggestions

Namings and config format are not final, feel free to suggest alternatives.

What are the relevant issue numbers?

Edited by Darren Eastman

Merge request reports