Define sleep & wake up cycles for your Kubernetes resources. Automatically schedule to shutdown Deployments, CronJobs, StatefulSets and HorizontalPodAutoscalers that occupy resources in your cluster and wake them up only when you need them; in that way you can:
- schedule resource-hungry workloads (migrations, synchronizations, replications) in hours that do not impact your daily business
- depressurize your cluster
- decrease your costs
- reduce your power consumption
- lower you carbon footprint
You’ll need a Kubernetes cluster to run against. You can use KIND or K3D to get a local cluster for testing, or run against a remote cluster.
Under config/samples
you will find a set manifests that you can use to test this sleepcycles on your cluster:
- core_v1alpha1_sleepcycle_app_x.yaml, manifests to deploy 2
SleepCycle
resources in namespacesapp-1
andapp-2
apiVersion: core.rekuberate.io/v1alpha1
kind: SleepCycle
metadata:
name: sleepcycle-app-1
namespace: app-1
spec:
shutdown: "1/2 * * * *"
shutdownTimeZone: "Europe/Athens"
wakeup: "*/2 * * * *"
wakeupTimeZone: "Europe/Dublin"
enabled: true
Note
The cron expressions of the samples are tailored so you perform a quick demo. The shutdown
expression schedules
the deployment to scale down on odd minutes and the wakeup
schedule to scale up on even minutes.
Every SleepCycle
has the following mandatory properties:
shutdown
: cron expression for your shutdown scheduleenabled
: whether this sleepcycle policy is enabled
and the following non-mandatory properties:
shutdownTimeZone
: the timezone for your shutdown schedule, defaults toUTC
wakeup
: cron expression for your wake-up schedulewakeupTimeZone
: the timezone for your wake-up schedule, defaults toUTC
successfulJobsHistoryLimit
: how many completed CronJob Runner Pods to retain for debugging reasons, defaults to1
failedJobsHistoryLimit
: how many failed CronJob Runner Pods to retain for debugging reasons, defaults to1
runnerImage
: the image to use when spawn CronJob Runner pods, defaults toakyriako78/rekuberate-io-sleepcycles-runners
Important
DO NOT ADD seconds or timezone information to you cron expressions.
- whoami-app-1_x-deployment.yaml, manifests to deploy 2
Deployment
that provisions traefik/whoami in namespaceapp-1
- whoami-app-2_x-deployment.yaml, manifests to deploy a
Deployment
that provisions traefik/whoami in namespaceapp-2
- apache-hpa.yaml, manifest to deploy an
HorizontalPodAutoscaler
for a PHP application in namespaceapp-2
- nginx-statefulset.yaml, manifest to deploy a
Statefulset
in namespaceapp-2
- busybox-cronjob.yaml, manifest to deploy a
Statefulset
in namespaceapp-1
SleepCycle
is a namespace-scoped custom resource; the controller will monitor all the resources in that namespace that
are marked with a Label
that has as key rekuberate.io/sleepcycle:
and as value the name
of the manifest you created:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-2
namespace: app-2
labels:
app: app-2
rekuberate.io/sleepcycle: sleepcycle-app-2
spec:
replicas: 9
selector:
matchLabels:
app: app-2
template:
metadata:
name: app-2
labels:
app: app-2
spec:
containers:
- name: app-2
image: traefik/whoami
imagePullPolicy: IfNotPresent
Important
Any workload in namespace kube-system
marked with rekuberate.io/sleepcycle
will be ignored by the controller by design.
The diagram below describes how rekuberate.io/sleepcycles
are dealing with scheduling a Deployment
:
- The
sleepcycle-controller
watches periodically, every 1min, all theSleepCycle
custom resources for changes (in all namespaces). - The controller, for every
SleepCycle
resource within the namespaceapp-1
, collects all the resources that have been marked with the labelrekuberate.io/sleepcycle: sleepcycle-app1
. - It provisions, for every workload - in this case deployment
deployment-app1
aCronJob
for the shutdown schedule and optionally a secondCronJob
if a wake-up schedule is provided. - It provisions a
ServiceAccount
, aRole
and aRoleBinding
per namespace, in order to make possible for runner-pods to update resources' specs. - The
Runner
pods will be created automatically by the cron jobs and are responsible for scaling the resources up or down.
Note
In the diagram it was depicted how rekuberate.io/sleepcycles
scales Deployment
. The same steps count for a
StatefulSet
and a HorizontalPodAutoscaler
. There are two exception though:
- a
HorizontalPodAutoscaler
will scale down to1
replica and not to0
as for aDeployment
or aStatefulset
. - a
CronJob
has no replicas to scale up or down, it is going to be enabled or suspended respectively.
You can combine rekuberate.io/sleepcycles
with applications provisioned with ArgoCD,
as long as you disable self-healing
when an automatic sync policy is enabled, otherwise ArgoCD's sync mechanism will operate antagonistically towards the
shutdown and wakeup cronjobs. In this case ArgoCD will always automatically revert to the state described in the
git manifests and practically will cancel the effect of the sleepcycle schedule. An ArgoCD application with Manual or
Automatic sync policy without self-healing will work as expected.
Tip
There is a git repository in place, with all the necessary artifacts to deploy via ArgoCD an nginx application and a preconfigured sleepcycle. You can find the git repo here.
- Build and push your image to the location specified by
IMG
inMakefile
:
# Image URL to use all building/pushing image targets
DOCKER_HUB_NAME ?= $(shell docker info | sed '/Username:/!d;s/.* //')
# sleepcycles
IMG_TAG ?= $(shell git rev-parse --short HEAD)
IMG_NAME ?= rekuberate-io-sleepcycles
IMG ?= $(DOCKER_HUB_NAME)/$(IMG_NAME):$(IMG_TAG)
# runners
RUNNERS_IMG_NAME ?= rekuberate-io-sleepcycles-runners
RUNNERS_IMG ?= $(DOCKER_HUB_NAME)/$(RUNNERS_IMG_NAME)
# targets
PLATFORMS ?= linux/arm64,linux/amd64,linux/s390x,linux/ppc64le
# CONTAINER_TOOL defines the container tool to be used for building images.
# Be aware that the target commands are only tested with Docker which is
# scaffolded by default. However, you might want to replace it to use other
# tools. (i.e. podman)
CONTAINER_TOOL ?= docker
make docker-buildx
- Deploy the controller to the cluster using the image defined in
IMG
:
make install && make deploy
and then deploy the samples:
kubectl create namespace app-1
kubectl create namespace app-2
kubectl apply -f config/samples
make undeploy
If you are on a development environment, you can quickly test & deploy the controller to the cluster
using a Helm chart directly from config/helm
:
helm install rekuberate-io-sleepcycles config/helm/ -n <namespace> --create-namespace
and then deploy the samples:
kubectl create namespace app-1
kubectl create namespace app-2
kubectl apply -f config/samples
helm uninstall rekuberate-io-sleepcycles -n <namespace>
On the other hand if you are deploying on a production environment, it is highly recommended to deploy the controller to the cluster using a Helm chart from its repo:
helm repo add sleepcycles https://rekuberate-io.github.io/sleepcycles/
helm repo update
helm upgrade --install sleepcycles sleepcycles/sleepcycles -n rekuberate-system --create-namespace
and then deploy the samples:
kubectl create namespace app-1
kubectl create namespace app-2
kubectl apply -f config/samples
helm uninstall rekuberate-io-sleepcycles -n <namespace>
This project aims to follow the Kubernetes Operator pattern. It uses Controllers which provides a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
make generate
make manifests
then install the CRDs in the cluster with:
make install
Tip
You can debug the controller in the IDE of your choice by hooking to the main.go
or you can start
the controller without debugging with:
make run
Tip
Run make --help
for more information on all potential make
targets
More information can be found via the Kubebuilder Documentation
You always need to build a new docker container and push it to your repository:
make docker-buildx
Important
In this case you will need to adjust your Helm chart values to use your repository and container image.
Adjust the image and container registry name in Makefile
:
# runners
RUNNERS_IMG_NAME ?= rekuberate-io-sleepcycles-runners
RUNNERS_IMG ?= $(DOCKER_HUB_NAME)/$(RUNNERS_IMG_NAME)
and then build and push the new image to the registry:
make docker-buildx-runner
Important
In this case you will need to adjust the runnerImage
of your SleepCycle
manifest to use your own Runner image,
otherwise it defaults always to akyriako78/rekuberate-io-sleepcycles-runners
To delete the CRDs from the cluster:
make uninstall