Start with a Template for a popular service like Airflow, Kafka, or PostgreSQL and CNDI will help you deploy it on your own infrastructure - just as easily as you can sign up for a cloud-based Platform as a Service.

You can also develop your own Templates to provide fill-in-the-blanks style wizards to your team.

Once your cluster is set up, manage the infrastructure and applications with ease using GitOps and Infrastructure as Code.

If you'd like to see an overview of the tool followed by a walkthrough for setting up an Airflow cluster using CNDI, you can find both here:

Install the latest CNDI release on your system using one of the commands below:

Shell (Mac, Linux):

curl -fsSL https://raw.githubusercontent.com/polyseam/cndi/main/install.sh | sh

PowerShell (Windows):

irm https://raw.githubusercontent.com/polyseam/cndi/main/install.ps1 | iex

For more information check out our install guide.

Use CNDI to deploy GitOps enabled Kubernetes application clusters on any platform, as quickly and easily as possible. Every CNDI project starts with a Template.

the best way to bootstrap a CNDI project is by using cndi create. It's best because it is easy and interactive, to some folks it may even feel like too much magic 🪄 but fear not! We will explain everything soon.

We start by picking a CNDI Template, and airflow is one of our favourites.

Let's run with that:

# cndi create johnstonmatt/my-airflow --template airflow && cd my-airflow
cndi create <owner>/<repo> -t airflow && cd <repo>

Airflow is one of the Templates bundled with CNDI, so we can call it out by name, but Templates are just YAML, so they can also be loaded from a URL or file path!

When in interactive mode, cndi will prompt you for the information a Template needs step-by-step.

The first prompt cndi create displays asks you where you want to store your project on disk, defaulting to a new folder called <repo> in the current working directory.

There will be a few more prompts that are asked for every new CNDI project, including asking for GitHub credentials and where you want to deploy your new cluster.

These prompts are called core prompts, and depending how you answer that first set, you'll be shown more prompts. One of the more important decisions is to choose your cloud provider and cndi natively supports 4 today: aws, gcp, azure, and dev for deploying experiments locally.

CNDI shines brightest when deploying to the cloud, so we encourage that if you have access!

When you are asked for credentials to your deployment target, you can follow the corresponding setup guide for AWS, GCP, or Azure.

The last set of questions relate directly to the Template you selected, and generally it is fine to accept the defaults if you aren't completely familiar with the stack you are deploying.

Once you've answered all of the prompts, CNDI will create a number of project files and we have a dedicated section that covers the CNDI Project Structure, but there a couple things that are important to know now:

1. cndi create will create a GitHub repo on your behalf and push the project (files and folders) it generates for you to that repo
2. All of your responses that are sensitive are written to a .env file, and that file will not be included in your repo. Instead, CNDI will push these to GitHub for use in automation by using the GitHub Secrets API.
3. CNDI created a file called .github/workflows/cndi-run.yaml which will trigger any time code is pushed to your repo

The CLI will provide info about the files it is creating for you, then provide a link to the new repo where you can watch the GitHub Action deployment progress!

After a Template has been initialized with either cndi init or cndi create , CNDI projects are managed from only 2 files:

.env : adheres to the dotenv pattern, its .gitignored, it contains all secret values which must not be included in source control

cndi_config.yaml : this file is the center of your CNDI project, it provides an abstraction which unifies infrastructure, applications, and configuration.

Though these files are the only ones required to manage your project as a user, they are not natively understood by Infrastructure as Code tools, and they are not understood by Kubernetes natively either.

You manage your cluster using cndi_config.yaml, but you must transform it into code that can be processed by Terraform, and manifests which can be processed by Kubernetes through ArgoCD.

The cndi overwrite command is responsible for taking both your .env file and your cndi_config.yaml file and transforming them into a set of Kubernetes manifests, and Terraform objects.

The cndi_config.yaml file is structured into 4 main sections:

• metadata - eg. project_name, cndi_version, provider, distribution
• applications - used to define Helm Charts which should be installed and configured
• cluster_manifests - used for arbitrary Kubernetes configuration
• infrastructure - used to define infrastructure CNDI exposes and optionally raw Terraform objects

The workflow is simple, modify your cndi_config.yaml fields and call cndi ow, it will transform your configuration into files which can be processed by ArgoCD and Terraform, and it will output the resulting files to the ./cndi folder at the root of your repo.

Feel free to examine the result of calling cndi ow to better understand how it works. This can be especially insightful when examining a git diff, but remember, those files are for machines and you can be productive with CNDI without ever reading or understanding them - that's by design.

Once you've made changes using cndi ow, the next step is to git push a commit so that those changes to infrastructure and applications can be applied. The .github/workflows/cndi-run.yaml file we've generated for you is very simple.

The workflow checks to ensure there is no active instance of the workflow in progress to prevent corruption, it loads the credentials which are required for the deployment of infrastructure, and then it calls cndi run.

The run command is responsible for calling Terraform against all of the objects in the ./cndi/terraform/ folder, then encrypting and persisting Terraform state in git.

Calling cndi run doesn't do anything directly with the other half of your config ./cndi/cluster_manifests, and that is because those manifests are instead pulled into your cluster by ArgoCD when it has been successfully deployed by Terraform.

The last CNDI command you should know about is cndi destroy. This command takes no arguments, and it is responsible for pulling and decrypting Terraform state, then calling terraform destroy for you under the hood, which will blast away every resource that CNDI has created. Once the command exits successfully, you can safely delete your git repo or achive it for reference later.

We've got a few walkthroughs you can follow if you'd like, one for each deployment target. These walkthroughs hold your hand through the process of deploying Airflow to the provider and distribution of your choice. They include info about how to get credentials, explanations about prompts, screenshots, and more.

• eks/airflow - AWS EKS
• gke/airflow - GCP GKE
• aks/airflow - Azure AKS
• dev/airflow - Local Development

If you are interested in using CNDI, these walkthroughs will be entirely transferrable to other applications beyond Airflow!

If you understand a cndi_config.yaml file, you will be successful in using CNDI. The file enables configuring existing systems like cert-manager for TLS certs, external-dns, and ingress. It also enables adding arbitrary Kubernetes Manifests and Terraform objects, yielding endless possibilities. To learn about all the configuration options, check out the CNDI Config Guide and accompanying jsonschema file.

There are a few other files beyond cndi_config.yaml which all play a part in your CNDI project. To learn more about each file cndi create generated, check out CNDI Project Structure Guide.

CNDI has support for deploying serverless functions to Kubernetes, to learn more checkout the CNDI Functions Guide.

Sometimes you need to connect to your cluster in order to debug applications, especially if you have not exposed your ArgoCD instance to the internet. To learn more about how to connect to your cluster, check out Connecting Remotely to CNDI Clusters Guide.

Templates are YAML files which define a relationship between prompts and file outputs. To learn more about building new CNDI Templates and distributing them to your team, checkout the CNDI Template Authoring Guide. To request new Templates or bump existing requests, check out our Template wishlist

If you're hoping to contribute to CNDI, please reach out to johnstonmatt! To learn more about setting up your development environment and other contributor info, check out CNDI Contributor Guide.

CNDI is in active development and there may be bugs, rough edges, or missing docs. We are commited to help you succeed with the project. If you need help reach out however you can.

We aim to maintain a discussion post for every possible exception here, so those are a great place to start, but don't hesitate to create an issue if you aren't able to find a discussion thread for your error.

We'd love to see you in the Polyseam Discord Channel too, for help or just to hang out.