This repository has been deprecated and archived on Jun 25th, 2024. There is replacement available, see https://github.com/kyma-project/lifecycle-manager.

With the Kyma 2.0 release, the lifecycle management of a Kyma installation is centralised in a dedicated reconciler component.

Goal of the reconciler is to install, update, and delete Kyma installations.

Besides managing the lifecycle, the reconciler is also responsible for reverting unexpected changes applied on Kyma-managed resources in Kubernetes. This is especially relevant since Kyma 2.0, which allows customers to have full admin access to the cluster or to bring their own cluster.

Non-functional requirements are:

• Reconciliation must be lightweight, fast, and easy scalable.
• Focus of the reconciler is on the lifecycle management of Kyma-related components. It does not need to support everything.
• The reconciler must expose monitoring metrics and log data, and be integrated into the existing observability stack.
• There is an administration interface (for example, a console) and it can be used by, or embedded in, existing administrative tools.
• Log messages must be expressive and reconciliation-related logs must indicate to which cluster they belong.

Top five main quality goals are:

This section summarizes requirements that constrain software architects in their freedom of design and implementation decisions or decision about the development process.

• Postgres must be used as RDBMS system because it is already used by KEB (Kyma Environment Broker) in KCP (Kyma Control Plane).
• The software must run as Pod in Kubernetes clusters (KCP), but also as standalone process (main program).
• It must be possible to use the reconciliation logic within the Kyma CLI.
• Introducing component-specific reconciliation logic must be supported.
• Any reconciliation must be idempotent.
• Reconciliation must be scalable. Parallel accesses on data (race conditions) is expected.
• Integration into the existing observability and security scanner stack in SAP must be possible.
• It must be possible to use the Kyma CI/CD system (Prow), and, wherever possible, to reuse or adapt of existing pipelines.
• It must use the HELM templating and installation logic, because it's a mainstream library that doesn't use hooks. Be aware about HELM's drawbacks.

The reconciler runs in two business contexts:

1. microservice inside the KCP ecosystem
2. embedded as library into another application (for example, Kyma CLI)

{diagram}

Within the KCP landscape, the reconciler is running in a microservice setup, which consists of two components:

1. Mothership reconciler

   The mothership reconciler is responsible for coordinating the reconciliation processes applied on Kyma clusters by interacting with the component reconcilers. The mothership reconciler is the leader of all reconciliation-related data.

2. Component reconcilers

   The component reconcilers are responsible for rendering particular Kyma component charts and applying them on Kubernetes clusters. The mothership reconciler is the only client of component reconcilers.

Surrounding services are:

{diagram}

Other applications can reuse the reconciler logic to offer lifecycle management capabilities for Kyma as well.

The Kyma CLI is one of the most important consumers of the reconciler. It includes the reconciler library.

The main difference of the embedded setup compared to micoservices is the missing REST API, and that a cluster reconciliation happens just once instead in an interval loop.

In the embedded setup, the mothership and component reconcilers interact over Go function calls with each other instead of HTTP REST calls.

The reconciler interfaces (mothership and component reconcilers) are based on REST as it's currently only used by technical clients.

Two OpenAPI specifications are established:

• Internal API

  Defines the interface contract between the mothership and the component reconcilers.

• External API

  Includes the interface contract of the REST API exposed to a 3rd party system (for example, KEB, KCP CLI etc.) by the mothership reconciler. Most of the communication between the mothership reconciler and 3rd party systems is based on a pull approach: The external system asks the reconciler for particular data or triggers an action.

No reconciler initiates a communication to an external system. Outgoing communication happens just between the mothership and component reconciler:

• The mothership calls the component reconciler and informs it about a required reconciliation of a Kyma component on a remote cluster.
• The component reconciler replies with a regular heartbeat message to the mothership reconciler about the progress of the requested reconciliation.

Following the single responsibility principle (SRP), the decomposition of the reconciler happend based on two major concerns:

• A data-leading unit that administrates reconciliation processes

2. An executing unit that assembles and applies Kyma component charts

The mothership reconciler controls the management of the Kyma clusters, and when and how they must be reconciled. Depending on the configuration of a Kyma cluster, the reconciliation of a Kyma installation consists of multiple Kyma components (normally packaged as Helm chart).

The mothership reconciler is responsible for managing all reconciliation-related data and coordinating that Kyma clusters are reconciled either because their configuration was changed, or the last reconciliation was too long ago.

It is a pure management unit, which doesn't interact with any managed Kyma cluster.

Component reconcilers handle any actions that are needed to apply Kubernetes resources, respectively resources defined within a Kyma component.

The mothership delegates the reconciliation of a particular Kyma component to a component reconciler, which can be either a generalist capable to handle multiple different Kyma components or specialised for reconciling just a particular component.

Component reconcilers take the workload of a reconciliation and have the following characteristics:

• They're stateless.
• They don't need a database.
• They're quickly and easily scalable.

The runtime view shows the most important regularly executed tasks that the mothership reconciler performs, and how the involved software elements are interacting.

The reconciler follows the recommended approach for consistent logging in Kyma.

ZAP library is used and generating log messages. Default output channel is STDERR.

The interaction between microservices in KCP is only allowed over secured channels (TLS encrypted connections).

Authentication is also mandatory for any communication; it's ensured by one of the following options:

• mTLS (primarily used for Kubernetes-internal communication between services via the Istio service mesh)
• SSL Key authentication (used when communicating to the Kubernetes cluster)
• Protocol-specific authentication used by the Postgres DB
• OAuth2 (used by the KCP CLI to authenticate developers or SRE engineers)

Monitoring is only needed when the reconciler runs in microservice mode. Because the reconciler is deployed in KCP landscapes, the available Kyma observability stack in KCP is used.

Custom metrics are exposed to the central Prometheus monitoring system.

Postgres is already used in KCP by other services (KEB and provisioner).

To avoid adding another storage system, the storage layer of the reconciler also uses Postgres as RDBMS.

Modules in the reconciler architecture are decoupled units with particular responsibilities or tasks, respectively.

Each module must perform its task regularly. In the initial setup, each module is triggered in intervals (using Go Tickers).

Later, these time-based triggers will be replaced by an event-driven approach: A module will be called just in time when it must perform its task.

In Kyma release 1.x, particular components (called "prerequisites") were installed first. Any common Kyma component can expect that these prerequisites are available when it is deployed. To avoid major refactoring for Kyma 2.x, the reconciler takes care of this task and installs the prerequisites before common Kyma components are deployed.

Pre-requisites are:

• CRDs
• Component: cluster essential
• The Istio component
• Certificates

The list of prerequisite components is defined in the reconciler configuration.

The code structure of the different reconciler modules follows a layered model:

.

Each software layer is allowed to access layers on lower levels, but it is not allowed to access functionalities that are part of a higher layer.

1. Storages

   Persistent data sinks (such as Github, PostgresDB, cloud storages).

2. Persistency

   Abstraction layer to access lower-level data storages.

3. Business model

   Including business models.

4. Repository

   Manages business model entities.

5. Business Components

   Business entities offering enhanced business functions and processes.

6. Endpoints

   Exposed API interfaces (such as HTTP endpoints).

7. Client

   Clients (such as a microservice) accessing exposed API interfaces.

The reconciler uses Postgres to persist business model entities.

Access to the database happens by a custom ORM implementation. This decision was taken to offer optimized access and functionalities to the data structure underneath. This database layer is still in an early state and lacks many features of the common mainstream ORMs (such as GORM).

This increases maintenance efforts, introduces potential security risks caused by a lower maturity level, and can reduce the flexibility of the developers caused by missing or incomplete functionalities.

Any access to data models must go through the ORM and be processed by the RDBMs system. This makes the system a potential bottleneck that scales proportionally; and an increasing amount of data can lead to longer query times. Query optimization is challenging because of non-optimized data schema structures; and sometimes not possible at all.