It is an easier way to get started with Bhojpur.NET Platform - Self-Hosted installation

• A machine running Linux/MacOS and Docker
• Kubectl installed
• A Kubernetes cluster configured
• A TLS certificate

Or,

The process to install Bhojpur.NET Platform is described below:

1. generate a base config set
2. amend the config to suit your own use-case
3. validate configuration
4. render the Kubernetes YAML
5. kubectl apply

docker create -ti --name installer us-west2-docker.pkg.dev/bhojpur/platform/build/installer:$TAG
docker cp installer:/app/installer ./installer
docker rm -f installer

# Get the versions
docker run -it --rm us-west2-docker.pkg.dev/bhojpur/platform/build/versions:$TAG cat /versions.yaml > versions.yaml

For every command below, replace installer with go run . --debug-version-file ./versions.yaml.

./installer init > bhojpur.config.yaml

There are many things you can change in your config, which can be found in the Configuration Guide.

For the purposes of a quickstart, just change the domain to one of your own.

# Checks the validity of the configuration YAML
./installer validate config --config bhojpur.config.yaml

Any errors here must be fixed before deploying. See Config for more details.

# Checks that your cluster is ready to install the Bhojpur.NET Platform
./installer validate cluster --kubeconfig ~/.kube/config --config bhojpur.config.yaml

Any errors here must be fixed before deploying. See Cluster Dependencies for more details.

./installer render --config bhojpur.config.yaml > bhojpur.yaml

kubectl apply -f bhojpur.yaml

After a few minutes, your Bhojpur.NET Platform installation will be available on the specified domain.

The Installer generates a ConfigMap with the metadata of every Kubernetes object generated by the Installer. This can be retrieved to remove Bhojpur.NET Platform from your cluster.

kubectl get configmaps bhojpur-app -o jsonpath='{.data.app\.yaml}' \
  | kubectl delete -f - # Piping to this will delete automatically

Important. This may leave certain objects still in your Kubernetes cluster. This will include Secrets generated from internal Certificates and PersistentVolumeClaims. These will need to be manually deleted.

• All Bhojpur.NET Platform components
• Container registry*
• MySQL database*
• Jaeger operator*
• RabbitMQ
• Minio object storage*

* By default, these dependencies are installed if the inCluster setting is true. External dependencies can be used in their place

Not every parameter is discussed in this table, just ones that are likely to need changing. The full config structure is available in config.go.

The Bhojpur.NET Platform must be connected to a Git provider. This can be done via the dashboard on first load, or by providing authProviders configuration.

authProviders:
  - id: Public-GitHub
    host: github.com
    type: GitHub
    oauth:
      clientId: xxx
      clientSecret: xxx
      callBackUrl: https://$DOMAIN/auth/github.com/callback
      settingsUrl: xxx

The Bhojpur.NET Platform requires certain services for it to function correctly. The Installer provides all of these in-cluster, but they can be configured to use services external to the cluster.

To use the in-cluster dependency, set inCluster to be true.

containerRegistry:
  inCluster: false
  external:
    url: <url of registry>
    certificate:
      kind: secret
      name: container-registry-token

The container-registry-token secret must contain a .dockerconfigjson key - this can be created by using the kubectl create secret docker-registry command.

The Bhojpur.NET Platform requires an instance of MySQL 5.7 for data storage.

The default encryption keys are [{"name":"general","version":1,"primary":true,"material":"7uGh3q8y2DYryJwrVMHs5kWXJlqvHWWt/KJuNi25edI="}]

If using a GCP SQL instance, a Cloud SQL Proxy connection can be used.

database:
  inCluster: false
  cloudSQL:
    instance: <PROJECT_ID>:<REGION>:<INSTANCE>
    serviceAccount:
      kind: secret
      name: cloudsql-token

The cloudsql-token secret must contain the following key/value pairs:

• credentials.json - GCP Service Account key with roles/cloudsql.client role
• encryptionKeys - database encryption key. Use default value as above if unsure
• password - database password
• username - database username

For all other connections, use an external database configuration.

database:
  inCluster: false
  external:
    certificate:
      kind: secret
      name: database-token

The database-token secret must contain the following key/value pairs:

• encryptionKeys - database encryption key. Use default value as above if unsure
• host - IP or URL of the database
• password - database password
• port - database port, usually 3306
• username - database username

The Bhojpur.NET Platform supports the following object storage providers:

metadata:
  region: <gcp-region-code, e.g. asia-south1-a>
objectStorage:
  inCluster: false
  cloudStorage:
    project: <PROJECT_ID>
    serviceAccount:
      kind: secret
      name: gcp-storage-token

The gcp-storage-token secret must contain the following key/value pairs:

• service-account.json - GCP Service Account key with roles/storage.admin and roles/storage.objectAdmin roles

Azure Blob Storage is not S3 compatible. This uses Minio Gateway to provide an AWS S3 interface

metadata:
  region: <azure-region-code, e.g. centralindia>
objectStorage:
  inCluster: false
  azure:
    certificate:
      kind: secret
      name: az-storage-token

The az-storage-token secret must contain the following key/value pairs:

• accountName - the globally-unique storage account name
• accountKey - access key for the storage account

This is currently only tested with AWS. Other AWS S3-compatible providers should work but there may compatability issues - please raise a ticket if you have issues with other providers

metadata:
  region: <aws-region-code, e.g. ap-south-1>
objectStorage:
  inCluster: false
  s3:
    endpoint: s3.amazonaws.com
    credentials:
      kind: secret
      name: s3-storage-token

The s3-storage-token secret must contain the following key/value pairs:

• accessKeyId - username that has access to S3 account
• secretAccessKey - password that has access to S3 account

In AWS, the accessKeyId/secretAccessKey are an IAM user's credentials with AmazonS3FullAccess policy

In order for the deployment to work successfully, there are certain dependencies that need to be installed.

Your Kubernetes nodes must have the Linux kernel v5.4.0 or above and have a containerd runtime.

Your Kubernetes nodes must have the following labels applied to them:

• bhojpur.net/workload_meta
• bhojpur.net/workload_apps
• bhojpur.net/workload_workspace_services
• bhojpur.net/workload_workspace_regular
• bhojpur.net/workload_workspace_headless

It is recommended to have a minimum of two node pools, grouping the meta and apps nodes together and the application nodes together.

It is a requirement that a certificate secret exists, named as per certificate.name in your config YAML with tls.crt and tls.key in the secret data. How this certificate is generated is entirely your choice - we suggest cert-manager for simplicity, however any certificate authority can be used by creating a Kubernetes secret.

The certificate must be associated with the following domains (where $DOMAIN is the value in config domain):

• $DOMAIN
• *.$DOMAIN
• *.ws.$DOMAIN

See FAQs for help with creating a TLS certificate using cert-manager.

cert-manager MUST be installed to your cluster. In order to secure communication between the various components, the application creates internally which are created using the cert-manager Certificate and Issuer Custom Resource Definitions.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade \
    --atomic \
    --cleanup-on-fail \
    --create-namespace \
    --install \
    --namespace='cert-manager' \
    --reset-values \
    --set installCRDs=true \
    --set 'extraArgs={--dns01-recursive-nameservers-only=true,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}' \
    --wait \
    cert-manager \
    jetstack/cert-manager

The Bhojpur.NET Platform Installer is a complete replacement for our Helm charts. Over time, this had grown to be too complex to effectively support and was a barrier to entry for new users - the base config was many hundreds of lines long and did not have effective validation on it. By contrast, the Installer's config is < 50 lines long and can be fully validated before running.

Also, by baking-in the container image versions to each release of the Bhojpur.NET Platform Installer, we reduce the potential for variance making it easier to support the community.

Please see cert-manager.io for full documentation. This should be considered a quickstart guide.

There are two steps to creating a public TLS certificate using cert-manager.

As the certificate is a wildcard certificate, you must use the DNS01 Challenge Provider. Please consult their documentation for instructions. This can be either an Issuer or ClusterIssuer.

Replace $DOMAIN with your domain. This example assumes you have created a ClusterIssuer called bhojpur-issuer - please change this if necessary.

This certificate is called https-certificates - please use that in your Bhojpur.NET Platform installer config.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: https-certificates
spec:
  secretName: https-certificates
  issuerRef:
    name: bhojpur-issuer
    kind: ClusterIssuer
  dnsNames:
    - $DOMAIN
    - "*.$DOMAIN"
    - "*.ws.$DOMAIN"

If you don't wish to use cert-manager to create a TLS certificate with a public certificate authority, you can bring your own.

To do this, generate your certificate as you would normally and then create a secret with the CRT set to tls.crt and the Key set to tls.key.

The DNS names must be $DOMAIN, *.$DOMAIN and *.ws.$DOMAIN, where $DOMAIN is your domain.

apiVersion: v1
kind: Secret
metadata:
  name: https-certificates
data:
  tls.crt: xxx
  tls.key: xxx

By default, Bhojpur.NET Platform will be installed to the default Kubernetes namespace. To install to a different namespace, pass a namespace flag to the render command.

./installer render --config bhojpur.config.yaml --namespace bhojpur > bhojpur.yaml

IMPORTANT: this does not create the namespace, so you will need to create that separately. This is so that uninstallation of Bhojpur.NET Platform does not remove any Kubernetes objects, such as your TLS certificate or connection secrets.

kubectl create namespace bhojpur

PRs/comments welcome