GitOps state for my cluster using flux v2

K3S multi-arch highly available cluster installed via Ansible on Proxmox VMs.

The cluster is designed to allow tearing it completely without any data lost.

Stack is ordered in multiple layers (Flux kustomizations) depending on the lower one (example apps depend on infrasteructure).

• Applications
  • default
  • kube-system
  • podinfo
  • vpn
• Core
  • ceph
  • ceph-rbd
  • cert-manager
  • flux-system
  • kube-system
  • monitoring
  • nginx
  • postgres
  • redis
  • vpn
  • vpn-gateway
• Custom Resource Definitions
  • cert-manager
  • multus
  • postgres
• Settings and Secrets
  • global settings
  • global secrets
  • production global settings
  • production global secrets
  • staging global settings
  • staging global secrets
• Base
  • sources
  • namespaces
  • flux-system
• Clusters:
  • production
  • staging
• Persistance:
  • Cluster configuration:
    • flux2 - Keep cluster in sync with this repo
  • Secrets - see Secret Management::
    • Ansible Vault - Ansible, Deployment
    • SOPS - Flux, K8S GitOps
  • Files:
    • Fast but depending on Sinology NAS: nfs
    • Slower but replicated: Ceph in Promox
  • Databases:
    • postgres: 3 instances deployed via Zalando´s Postgres Operator

• 3x Intel NUC 11 vPro (NUC11TNHv5) with:
  • 11th Gen Intel® Core™ i5-1145G7 @ 2.60GHz
  • 32 GB DDR4
  • 250 GB Sata SSD for local disks - 2x Samsung SSD 850 EVO, 1x CT240BX500SSD1
  • 500 GB NVME for Ceph - WDC WDS500G1B0C-00S6U0
  • 2 Thunderbolt 4/3 connected as network mesh for ceph:
    • Node 1, Port 1 <-> Node 2, Port 1
    • Node 1, Port 2 <-> Node 3, Port 1
    • Node 2, Port 2 <-> Node 2, Port 2

Installed via Ansible. It creates the VMs for the 3 nodes

The cluster is designed to allow tearing the cluster completly without any data lost.

Master secret is stored in Ansible Vault.

Kubernetes passwords and secrets encrypted with mozilla SOPS which it is supported out of the box in Flux2.

GPG key is deployed via Ansible. Its hash must be kept in sync with .sops.yaml.

Based on Vaskozl I use a pre-commit hook to ensure that secrets are never pushed unencrypted. The hook is deployed by running cd scripts; ./install_git_hooks.sh

To encrypt files with secrets use:

sops -e -i my-secret.yaml # Initial encrypt
sops my-secret.yaml # To edit it directly in you $EDITOR

• Delete stuck objects (PVs, PVCs)

  kubectl patch <object type> <object name> -p '{"metadata":{"finalizers": []}}' --type=merge
  

• Delete stuck NSs

  NAMESPACE=your-rogue-namespace
  kubectl proxy &
  kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >/tmp/patch.json
  curl -k -H "Content-Type: application/json" -X PUT --data-binary @/tmp/patch.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
  

This cluster in inspired by the work of others shared at awesome-home-kubernetes. These projects can be searched in GitHub. They include the k8s-at-home topic.

There is also an active k8s@home Discord for this community.