Before installing Fusion, you need to choose a Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.

NOTE: All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.

Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.

Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you’re using, you need to install helm as it is required to install Fusion for any K8s platform. On MacOS, you can do:

If you already have helm installed, make sure you’re using the latest version:

For other OS, please refer to the Helm installation docs: https://helm.sh/docs/using_helm/

The Fusion helm chart requires that helm is greater than version 3.0.0; check your Helm version by running helm version --short.

You should clone this repo from github as you’ll need to run the scripts on your local workstation:

You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.

Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: https://github.com/lucidworks/fusion-cloud-native/archive/master.zip. Once downloaded, extract the zip and cd into the fusion-cloud-native-master directory.

If you’ve already installed the gcloud command-line tools, you can skip to Create a Fusion cluster in GKE.

These steps set up your local Google Cloud SDK environment so that you’re ready to use the command-line tools to manage your Fusion deployment.

Usually, you only need to perform these setup steps once. After that, you’re ready to create a cluster.

For a nice getting started tutorial for GKE, see: Deploy an app to a GKE cluster.

1. Enable the Kubernetes Engine API.

2. Log in to Google Cloud: gcloud auth login

3. Set up the Google Cloud SDK:

   1. gcloud config set compute/zone <zone-name>

      If you are working with regional clusters instead of zone clusters, use gcloud config set compute/region <region-name> instead.

   2. gcloud config set core/account <email address>

   3. New GKE projects only: gcloud projects create <new-project-name>

      If you have already created a project, for example in the Google Cloud Platform console, then skip to the next step.

   4. gcloud config set project <project-name>

Make sure you install the Kubernetes command-line tool kubectl using:

Run the setup_f5_gke.sh script to install Fusion 5.x in a GKE cluster. To create a new, single-node demo cluster and install Fusion, simply do:

Use the --help option to see script usage. If you want the script to create a cluster for you, then you need to pass the --create option with either demo or multi_az. If you don’t want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the -c parameter.

If you pass --create demo to the script, then we create a single node GKE cluster (defaults to using n1-standard-8 node type). The minimum node type you’ll need for a 1 node cluster is an n1-standard-8 (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads.

You can change the instance type using the -i parameter; see: https://cloud.google.com/compute/docs/regions-zones/#available for an list of which machine types are available in your desired region.

Note: If not provided the script generates a custom values file named gke_<cluster>_<namespace>_fusion_values.yaml which you can use to customize the Fusion chart.

WARNING If using Helm V2, the setup_f5_gke.sh script installs Helm’s tiller component into your GKE cluster with the cluster admin role. If you don’t want this, then please upgrade to Helm v3.

If you see an error similar to the following, then wait a few seconds and try running the setup_f5_gke.sh script again with the same arguments as this is usually a transient issue:

After running the setup_f5_gke.sh script, proceed to the Verifying the Fusion Installation section below.

When you’re ready to deploy Fusion to a production-like environment, see more information at Fusion 5 Survival Guide.

With a three-node regional cluster, nodes are deployed across three separate availability zones.

In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.

When running in a multi-zone cluster, each Solr node has the solr_zone system property set to the zone it is running in, such as -Dsolr_zone=us-west1-a.

After running the setup_f5_gke.sh script, proceed to the Verifying the Fusion Installation section below.

When you’re ready to deploy Fusion to a production-like environment, see more information at Fusion 5 Survival Guide.

Before you begin, please consult the Migration Guide.

During installation, the setup script generates a file named gke_<cluster>_<release>_fusion_values.yaml; use this file to customize Fusion settings.

In addition, the setup script creates a helper upgrade script to streamline the upgrade process. Look in the directory where you ran the setup script initially for a file named:

where <release> is typically the same as your namespace unless you overrode the default value using the -r option.

After running the upgrade, use kubectl get pods to see the changes being applied to your cluster. It may take several minutes to perform the upgrade as new Docker images need to be pulled from DockerHub. To see the versions of running pods, do:

A user must grant permissions to the Google service account so the ML Model Service can use Google Cloud Storage. This way you can always reference your model even if nodes are created or destroyed as part of cluster scaling.

Grant the default service account read/write access to a GCS bucket by upgrading with these changes:

To get the service account, do:

In the values.yaml, provide:

Before launching an EKS cluster, you need to install and configure kubectl, aws, eksctl, aws-iam-authenticator using the links provided below:

1. kubectl: Install kubectl

2. aws: Installing the AWS CLI

3. eksctl: Getting Started with eksctl

4. aws-iam-authenticator: AWS IAM Authenticator for Kubernetes

Run aws configure to configure a profile for authenticating to AWS. You’ll use the profile name you configure in this step, which defaults to default, as the -p argument to the setup_f5_eks.sh script in the next section.

To create a cluster in EKS the following IAM policies are required:

• AmazonEC2FullAccess

• AWSCloudFormationFullAccess

Download and run the setup_f5_eks.sh script to install Fusion 5.x in an EKS cluster.

• To create a new cluster and install Fusion, run the following command:

  ./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo

  • Replace my-eks-cluster, profile-name, and fusion-namespace with your cluster, profile, and namespace values.

  • Pass the --create option with either demo or multi_az.

• To use an existing cluster and install Fusion, run the following command:

  ./setup_f5_eks.sh -c cluster-name -p profile-name

  • Replace cluster-name with the name of the cluster you already created.

  • Replace profile-name with the name of your profile.

The profile is automatically set to default if you ran the AWS configure command without giving the profile a name.

Use the --help option to see full script usage.

After running the setup_f5_eks.sh script, proceed to the Verifying the Fusion Installation section below.

Before you begin, please consult the Migration Guide.

To make things easier for you, our setup script creates an upgrade script you can use to perform upgrades, see:

Initially, only the user that created the Amazon EKS cluster has system:masters permissions to configure the cluster. In order to extend the permissions, a ConfigMap should be created to allow access to IAM users or roles.

For providing these permissions, use the following yaml file as a template, replacing the required values:

aws-auth.yaml

Use the following command for applying the yaml file: kubectl apply -f aws-auth.yaml

In case you have deployed an ALB ingress controller, you would need to remove the policy that was created for managing the ALB before removing the cluster. You can use the following command for it:

Also you can remove it manually using the AWS IAM console, searching for eksctl-<cluster-name>-alb-policy.

After that you should remove the ALB with helm delete, list the current releases with helm list.

The EKS cluster is created using Cloudformation stacks so you need to remove them to delete the cluster, you can check them in the AWS Cloudformation Console, check for the following stacks:

• eksctl-<cluster-name>-nodegroup-standard-workers

• eksctl-<cluster-name>-cluster

The eksctl-<cluster-name>-nodegroup-standard-workers stack should be the first to be removed. After that we can remove the eksctl-<cluster-name>-cluster stack.

Also you can use the following commands>:

Before launching an AKS cluster, you need to install and configure kubectl and az using the links provided below:

1. kubectl: Install kubectl

2. az: Installing the Azure CLI

To confirm your account access and command-line tools are set up correctly, run the az login command (az login –help to see available options).

To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure. For more information about resource groups, see https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups.

You also need to choose a location where you want to spin up your AKS cluster, such as westus2. For a list of locations you can choose, see https://azure.microsoft.com/en-us/global-infrastructure/locations/.

Use the Azure console in your browser to create a resource group, or simply do:

1. Azure Account set up.

2. azure-cli (az) command-line tools installed.

3. az login working.

4. Created an Azure Resource Group and selected a location to launch the cluster.

Download and run the setup_f5_aks.sh script to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do:

If you don’t want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the -c parameter.

Use the --help option to see full script usage.

By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.

You can override the namespace using the -n option. In addition, our script uses f5 for the Helm release name; you can customize this using the -r option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only.

You can also pass the --preview option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see https://docs.microsoft.com/en-us/azure/aks/availability-zones.

It takes a while for AKS to spin up the new cluster. The cluster will have three Standard_D4_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the az aks create command.

After running the setup_f5_aks.sh script, proceed to Verifying the Fusion Installation.

Before you begin, please consult the Migration Guide.

To make things easier for you, our setup script creates an upgrade script you can use to perform upgrades, see:

You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters, ensure Helm V3 is ahead of Helm V2 in your working shell’s PATH before proceeding.

Before you begin, please consult the Migration Guide.

To update an existing installation, do:

Except for Zookeeper, all K8s deployments and statefulsets use a RollingUpdate update policy:

Zookeeper instances use OnDelete to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:

Alternatively, you can set the updateStrategy under the zookeeper section in your "${MY_VALUES}" file:

We can deploy Fusion in an existing OpenShift cluster. This cluster should be created using OpenShift Infrastructure Provider. A Red Hat Customer Portal account is required. OpenShift Online services are not supported.

The easiest way to install on OpenShift is to run the setup_f5_k8s.sh script for your existing cluster; use the --help option to see script usage. For instance, the following command will install Fusion 5 into the specified namespace (-n) and OpenShift cluster (-c):

Tip: kubectl should work with your OpenShift cluster (see: https://docs.openshift.com/container-platform/4.1/cli_reference/usage-oc-kubectl.html) and Lucidworks recommends installing the latest kubectl for your workstation instead of using oc for installing Fusion 5. However, if you do not have kubectl installed, then you’ll need to update the upgrade script created by setup_f5_k8s.sh to use oc instead of kubectl (search and replace on the BASH script using a text editor).

When you’re ready to deploy Fusion to a production-like environment, see more information at Fusion 5 Survival Guide.

Lucidworks recommends using Helm v3, but in case Tiller is required for Helm v2, the cluster security needs to be relaxed to allow images to run with different UIDs:

Here is a list of tools we found useful for improving your command-line experience with Kubernetes:

• krew (kubectl plugin mgr): https://github.com/kubernetes-sigs/krew/

• kube-ps1 (show current context on command line prompt): https://github.com/jonmosco/kube-ps1

• kubectx / kubens (switch between clusters / namespaces): https://github.com/ahmetb/kubectx

kubectl reference: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

Set the namespace for kubectl if not using the default:

This saves you from having to pass -n with every command.

Get a list of running pods: k get pods

Get logs for a pod using a label: k logs –l app.kubernetes.io/component=query-pipeline

Get pod deployment spec and details: k get pods <pod_id> -o yaml

Get details about a pod events: k describe po <pod_id>

Port forward to a specific pod: k port-forward <pod_id> 8983:8983

SSH into a pod: k exec -it <pod_id> — /bin/bash

CPU/Memory usage report for pods: k top pods

Forcefully kill a pod: k delete po <pod_id> --force --grace-period 0

Scale up (or down) a deployment: k scale deployment.v1.apps/<id> --replicas=N

Get a list of pod versions: k get po -o jsonpath='{..image}' | tr -s '' '\n' | sort | uniq

Once the install script completes, you can check that all pods and services are available using:

If all goes well, you should see a list of pods similar to:

The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file. Also, don’t worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use kubectl logs <pod_id> to see the logs for that pod; to see the logs for previous versions of a pod, use: kubectl logs <pod_id> -p. You can also look at the actions Kubernetes performed on the pod using kubectl describe po <pod_id>.

To see a list of Fusion services, do:

For an overview of the various Fusion 5 microservices, see: Fusion microservices.

Once you’re ready to build a Fusion cluster for production, please see see more information at Fusion 5 Survival Guide.

One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.

When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.

Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.

To upgrade your cluster at any time, use the --upgrade option with our setup scripts in this repo.

The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. To see what would be upgraded, you can pass the --dry-run option to the script.

Get the initial Grafana password from a K8s secret by doing:

With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer. To define a LoadBalancer, do (replace ${RELEASE} with your Helm release label):

You can use kubectl get services --namespace <namespace> to determine when the load balancer is setup and its IP address. Direct your browser to http://<GrafanaIP>:3000 and enter the username admin@localhost and the password that was returned in the previous step.

This will log you into the application. It is recommended that you create another administrative user with a more desirable password.

The dashboards and datasoure will be setup for you in grafana, simply navigate to Dashboards → Manage to view the vailable dashboards