Clustercontrol (CC) v2.3.0 is now live, featuring new enterprise binaries support for Percona Server Pro clusters and support for MySQL v8.4, Elasticsearch v8.15, and Valkey v8.0. Let’s dive into these key updates and discover what’s new!

New enterprise binaries with compatibility for Percona Server Pro

We have upgraded our enterprise binaries with Percona Server Pro packages. Percona Pro users can now easily deploy and import Percona Server Pro (replication) clusters and benefit from FIPS-140-2 compliance for enhanced security.

Support for major and minor database versions

• MySQL v8.4 LTS (Percona, Oracle): MySQL v8.4 LTS brings significant updates to performance, security, and developer usability, designed to meet the needs of modern applications. For a complete breakdown, see MySQL v8.4 release notes.
• Elasticsearch v8.15: For users trialing Elasticsearch and looking for the latest LTS version, v8.15 offers notable advancements, introducing LogsDB to optimize log storage, vector search enhancements for efficient memory use, and streamlined integration for AI models. For more info, check out Elasticsearch’s v8.15 release highlights.
• Valkey v8.0 (Alma/Rocky/RHEL9): Valkey v8.0 provides several optimizations, delivering performance up to three times faster than the original Redis open-source code. It also includes considerable memory efficiency improvements for this in-memory data store. Read more in the Valkey v8.0 announcement.

Discover more updates in Clustercontrol v2.3.0

Clustercontrol’s v2 UI also reintroduces the cluster load dashboard for MySQL Replication, Galera, and PostgreSQL. But these are just a few of the updates; for a complete list of what’s new, refer to our documentation.

Wrapping up

Clustercontrol v2.3.0 extends enterprise binaries and adds support for MySQL v8.4, Elasticsesearch v8.15, and Valkey v8.0 to improve the performance of critical workloads and applications. Look out for our next release, featuring PostgreSQL bi-directional logical replication, major database version support, and much more!

New to Clustercontrol? Try our Enterprise edition free for 30 days with technical support to guide you through the setup and beyond.

The post Clustercontrol adds Percona Server Pro, and support for MySQL v8.4, Elasticsearch v8.15, Valkey v8.0 in latest release appeared first on Severalnines.

We recently attended and spoke at the 2024 Data Community Conference in Bern. Leading voices in IT and database management came together for one day to discuss wide-ranging topics, covering everything from improving sustainability within the sector and employee mental well-being to the practical use of AI and ensuring a proactive cybersecurity posture.

Our CEO, Vinay Joosery, presented Sovereign DBaaS: A Practical Vision for Self-Implementation of DBaaS, which aimed to chart the evolution of DBaaS through to now and offered a practical reconceptualization of it as an implementation concept as opposed to a commercial one.

Before we jump into the others, let’s go ahead and dive into his highlights.

Sovereign DBaaS: A Vision for Database Autonomy

Vinay introduced Sovereign DBaaS as a solution for organizations seeking greater autonomy in managing their database operations. He started by charting DBaaS’s evolution across three phases:

1. Hyperscalers: Early providers like AWS offered convenience but limited flexibility.

2. Database Independent vendors: They reduced lock-in by packaging open-source databases as legitimate services, but still posed environmental and provider lock-in.

3. Sovereign DBaaS: A new concept that provides workload access and portability by decoupling the entire stack, resulting in efficient and reliable automation at scale without forcing organizations to sacrifice operational control of their data stacks.

His talk highlighted some of the benefits organizations could experience by rethinking DBaaS:

• Operational efficiency: Maintain control over their data and still benefit from automation.
• Workload portability: Deploy and move workloads to on-premises, cloud, or hybrid environments to suit evolving operational needs.
• Security and compliance: Address data residency, governance, and security with enhanced workload access and platform capabilities.
• Cost efficiency and control: Leverage open-source databases to avoid expensive licenses and be able to port their workloads to where they make the most sense.

These are just a few highlights from his session; he also provides a case study with a larger FINBIN customer. So make sure you check out the full presentation here. Let’s move on!

Other Key Themes from the Conference

Here is the rundown of other featured talks on the critical factors shaping IT strategies today:

1. Green IT strategies

Ivan Mariblanca Flinch discussed the growing issue of digital pollution, which accounts for up to 40% of a service company’s carbon footprint. He emphasized the need for sustainable IT certifications, like the global Sustainable IT label, to align digital operations with environmental goals.

2. Mental health in the workplace

André Delafontaine explored systematic approaches to mental health, providing tools to assess and enhance workplace well-being. His presentation highlighted the importance of addressing productivity and motivation through structured interventions.

3. AI in database administration

Gianni Ceresa explored the role of AI tools like large language models (LLMs) in solving SQL and database challenges. While AI is a powerful tool, he stressed that it cannot replace human expertise and works best in collaboration with skilled professionals.

4. Cybersecurity in IT

Christian Das Neves highlighted the sophisticated tactics used by malicious actors and underscored the importance of proactive defense and continuous education to counter evolving threats.

Learn more about Sovereign DBaaS

Interestingly, implementing a Sovereign DBaaS concept can be used to address most of the key topics above due to its focus on helping organizations enhance operational control over their data through improving workload access and portability by decoupling the data stack.

Explore the possibilities of Sovereign DBaaS, see it in action through real-world applications, and discover how you can achieve true, “database ops your way” by visiting our Sovereign DBaaS page and joining the movement by becoming a member of our Slack community.

The post Recapping Data Community Conference 2024: sustainability, AI, security and Sovereign DBaaS appeared first on Severalnines.

The siren song of the cloud is undeniable, promising scalability, resilience, and reduced operational overhead. However, the reality can be different. Factors like vendor lock-in, limited control, and unexpected costs, mean that organizations can still benefit from on-premises (on-prem) infrastructure to deploy and manage their systems.

Kubernetes—the de facto container orchestration tool in the cloud native ecosystem today—has enabled countless organizations to maximize the cloud’s possibilities. But there’s a question from on-premises users…“Can we deploy Kubernetes on-premises?” The short answer is “YES!!” and this article will explain why and how.

In this article, I will start by discussing what Kubernetes on-premises is, why organizations would want to deploy Kubernetes on-premises and considerations for Kubernetes on-prem. Towards the end, I’ll show you how you can get started with Kubernetes on-premises the provider agnostic way.

What is Kubernetes on-premises?

Kubernetes on-premises refers to running a Kubernetes cluster on your own infrastructure, within your own data center, rather than in a cloud environment offered by a cloud service provider (CSP) like AWS.

Kubernetes is platform-agnostic, which means it can run anywhere— from your local development environment using tools like minikube to high-scale production environments in the cloud and, critically, on-premises data centers.

Deploying and running Kubernetes on-premises is not the same as doing it in the cloud. It requires navigating specific challenges and leveraging the unique advantages of your private infrastructure.

Why run Kubernetes on-premises?

Seeing that Kubernetes on-prem has its unique constraints that can vary amongst organizations, why would one deploy Kubernetes on-prem? These are the most common reasons why organizations need on-premises infrastructure.

Environment standardization

Today you can implement Serverless, WebAssembly, blockchain, etc. on Kubernetes. Seeing the ability to run Kubernetes anywhere, organizations that have on-premises infrastructure can easily innovate with new and powerful technologies using the same/similar tools and processes used in the cloud without the wholesale shift.

Essentially, Kubernetes acts as a great environment equalizer, allowing these cutting-edge technologies to run seamlessly on any infrastructure.

Compliance & data privacy

Many industries operate under strict regulations regarding data storage and access. On-premises Kubernetes deployments provide complete control over data location, ensuring compliance with industry-specific mandates like GDPR, HIPAA, or financial regulations.

Furthermore, sensitive data might require heightened security measures that are best met within a privately managed environment. Deploying on-premises allows for granular control over security protocols and minimizes exposure to external threats.

Integration with Legacy systems

Legacy systems are still the backbone of our society today, especially in industries like banking, healthcare, and insurance. Many organizations in these industries are modernizing some parts of their systems and some decide to use Kubernetes as they containerize them.

What’s the best way to manage containerized systems deployed on-prem? Kubernetes on-prem.

This approach enables organizations to modernize their infrastructure incrementally, avoiding disruptions to critical operations. By running workloads on-premises, they can minimize latency by keeping data processing close to the source, rather than relying on cloud-based solutions.

Avoid Lock-in

One of the main challenges organizations today face with CSPs is vendor lock-in. Yes, you can transfer data from one cloud to another today with tools like GCP Storage Transfer Service or AWS DataSync, but that doesn’t mean you can easily move infrastructure. Especially infrastructure that has gone through multiple iterations over the years.

While Kubernetes is platform agnostic, managed Kubernetes offerings on CSPs have key differences due to their underlying cloud platforms. Control plane architecture, networking, features, ecosystem integrations etc differences that can make migration a challenge.

Long-term cost of the cloud service providers

Aside from vendor lock-in, the long-term cost of cloud services is a major challenge for organizations, especially at scale — and you know Kubernetes is designed for scale.

While initial setup costs may be higher, on-premises Kubernetes can offer a more predictable cost model over the long term, especially for organizations with consistent and predictable workloads. On-premises deployments allow for fine-grained control over resource allocation, maximizing efficiency and potentially reducing costs compared to cloud provider pricing models.

Considerations for Kubernetes on-premises

Before deploying Kubernetes on-prem, it’s crucial to approach your deployment with a clear understanding of the unique challenges it presents. Beyond the initial excitement of controlling your own infrastructure, several factors demand careful consideration to ensure a successful and sustainable Kubernetes environment.

Location

When deploying Kubernetes on-premises, the location of your infrastructure—whether in your own data center, a co-located facility, or at the edge—plays a crucial role in shaping performance, control, and cost. Owning a data center provides maximum control, but requires more configurations.

Co-locating in a third-party data center would reduce upfront costs and some maintenance overhead, but post limitations around physical access and potentially data governance.

In cases where you want to deploy on the edge, network connectivity, security, and cooling are concerns you should think about.

Hardware management

If you own/manage your data centers, hardware management is a critical consideration for running Kubernetes on-premises, as it significantly impacts the performance, reliability, and scalability of the environment.

Unlike cloud environments where infrastructure is abstracted and hardware maintenance is managed by the provider, on-premises deployments place the responsibility entirely on the organization’s IT and DevOps teams. This requires careful planning around hardware selection, capacity, maintenance, and compatibility to ensure Kubernetes can run effectively on local resources.

A common challenge is tackling over-provisioning. For example, if you’re running Kubernetes on-premises and it’s utilizing only 30% of the available CPU, you are underutilizing your CapEx investment. To make better use of this upfront cost, consider deploying or moving additional applications to the same server to fully leverage the existing infrastructure.

Network complexity

Deploying Kubernetes on-premises does not come with the built-in, automated networking features that cloud providers offer, making it essential for organizations to design, implement, and maintain their own networking infrastructure to support the cluster’s needs. Internal networking within an on-premises Kubernetes environment requires careful configuration.

Kubernetes uses a flat networking model that assumes all pods within a cluster can communicate directly with one another across nodes. Setting this up on-premises involves selecting and configuring a network overlay solution, such as Cilium, Calico, Flannel, etc. to enable seamless pod-to-pod communication while respecting network policies for security and isolation. Each of these tools brings specific configuration and maintenance needs, and they must be integrated with the existing network setup.

Storage concerns

Storage is a fundamental consideration when deploying Kubernetes on-premises, as managing data in a containerized environment presents unique challenges that require careful planning.

Yes, Kubernetes wasn’t originally designed for stateless workloads, but organizations do deploy stateful applications on Kubernetes today, from ML workloads to databases of different flavors.

Kubernetes supports a variety of storage solutions, including local storage, Network File System (NFS), and more sophisticated dynamic provisioning options through the Container Storage Interface (CSI).

In on-premises environments, persistent storage requires careful management to ensure that stateful applications have the reliability they need; this often involves integrating with existing SAN/NAS solutions or using distributed storage systems.

Managed Kubernetes services? The key to Kubernetes on-prem?

Yes, you can manually deploy and manage Kubernetes on-premises — Doing Kubernetes The Hard Way. But in production, you wouldn’t want to do that. You would want to automate as many of the deployment processes as possible with tools like kOps – Kubernetes Operations, Kubeadm, and Kubespray.

With these tools, you specify your configuration and they handle most of the details of setting up and bootstrapping the clusters for you. But this approach in itself can be challenging when issues arise. Due to this, there’s been a growing ecosystem of service providers to support on-premises data centers.

The Kubernetes on-premises landscape is supported by specialized organizations such as Rancher by SUSE, VMware vSphere and Red Hat OpenShift, and smaller, agnostic organizations like Sidero Labs, Platform9, Giant Swarm, etc also contribute to the ecosystem, offering fully managed experiences tailored to on-premises needs.

Additionally, major CSPs also offer on-premises Kubernetes solutions, namely, Amazon EKS Anywhere, Google Cloud Anthos and Azure Arc. These solutions are very attractive for organizations that have some applications running on the CSP’s Kubernetes on the cloud and want to extend it to on-premises, effectively creating a hybrid cloud solution. However, these will not be the best solutions for air-gapped environments.

Agnostic guide to setting up Kubernetes on-premises

As mentioned earlier, there are several ways to deploy Kubernetes on-prem. You can choose to do “Kubernetes The Hard Way”, automate some processes with the likes of kOps, or use a managed service or an agnostic tool.

On the Severalnines blog, over the past decade, we’ve been supporting organizations all over the world in implementing a Sovereign DBaaS model that lets them have control not just of their data, but the infrastructure and tooling that surrounds it through cloud-agnostic tools like ClusterControl. So we will also let you know ways to have more control over your environments with no manual overhead.

If you google Kubernetes on-premises, aside from the specialized providers, you would frequently see a tool called Talos Linux among the results you would get on Reddit threads.

Talos Linux is a modern, open-source Linux distribution built for Kubernetes. Talos can help you deploy a secure, immutable, and minimal Kubernetes cluster on any environment including bare metal, virtualized, and single-board platforms.

I got to try out Talos running on a single-board computer at the SREDay London Conference 2024, and it was indeed a great experience.

Overview

In this demo, I will introduce you to Talos by taking you through the steps to create a KVM-based cluster of 2 control plane nodes and 1 worker node with Vagrant and its libvirt plugin as outlined in the Talos documentation. KVM is ideal for building a Kubernetes cluster on your local system or in a private data center, as it offers flexibility, performance, and low overhead.

For this, we’ll mount the Talos ISO onto the VMs using a virtual CD-ROM and configure the VMs to prioritize booting from the disk, with the CD-ROM as a fallback option.

Prerequisites

To follow along with the demo in this article, you need to have the following prerequisites.

• Linux OS — For this article, we will be using an Ubuntu 22.04 Server of 8GB RAM and 160GB Disk.
• Vagrant installed — If you are using Vagrant version greater than 2.4.1-1, you might have issues installing plugins due to Logger dependency conflict with Vagrant 2.4.2, you should downgrade to 2.4.1-1.
• vagrant-libvirt plugin installed
• talosctl installed
• kubectl installed

Setting up Talos Linux on virtualized platforms with Vagrant & Libvirt

We will start by downloading the latest Talos ISO image — metal-amd64.iso — from GitHub releases into the /tmp directory with

wget --timestamping curl https://factory.talos.dev/image/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba/v1.8.0/metal-amd64.iso -O /tmp/metal-amd64.iso

Next create the demo Vagrant file with the following configuration:

Vagrant.configure("2") do |config|

  config.vm.define "control-plane-node-1" do |vm|
    vm.vm.provider :libvirt do |domain|
      domain.cpus = 2
      domain.memory = 2048
      domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-1.log"}
      domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
      domain.storage :file, :size => '4G', :type => 'raw'
      domain.boot 'hd'
      domain.boot 'cdrom'
    end
  end

  config.vm.define "control-plane-node-2" do |vm|
    vm.vm.provider :libvirt do |domain|
      domain.cpus = 2
      domain.memory = 2048
      domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-2.log"}
      domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
      domain.storage :file, :size => '4G', :type => 'raw'
      domain.boot 'hd'
      domain.boot 'cdrom'
    end
  end

  config.vm.define "worker-node-1" do |vm|
    vm.vm.provider :libvirt do |domain|
      domain.cpus = 1
      domain.memory = 1024
      domain.serial :type => "file", :source => {:path => "/tmp/worker-node-1.log"}
      domain.storage :file, :device => :cdrom, :path => "/tmp/metal-amd64.iso"
      domain.storage :file, :size => '4G', :type => 'raw'
      domain.boot 'hd'
      domain.boot 'cdrom'
    end
  end

end

The above configuration creates a small cluster of three VMs:

• Two control-plane nodes (each with 2 CPUs and 2GB of RAM).
• One worker node (with 1 CPU and 1GB of RAM).

Each VM has a bootable ISO mounted as a virtual CD-ROM and a 4GB primary disk, and they attempt to boot from the disk with a fallback to the CD-ROM. Serial output for each VM is logged to a corresponding file in /tmp.

Next, check the status of the node with vagrant status and you will see the VMs in “not created” state:

Now spin up the vagrant environment with the following command:

vagrant up --provider=libvirt

After a few seconds run vagrant status again and you see the VMs in running state.

You can find out the IP addresses assigned by the libvirt DHCP by running:

virsh list | grep root | awk '{print $2}' | xargs -t -L1 virsh domifaddr

The above command grep’s the string root because by default Vagrant creates the VMs prefixing your user. If, for example, your user is demo, edit the above command to demo.

For this demo, the control plane nodes have the following IPs:

• 192.168.121.232
• 192.168.121.68

And worker node IP:

• 192.168.121.150

Note the IP addresses as we will use them further in the workflow. Also, we should now be able to interact (maintenance mode) with the nodes with the talosctl through the following command:

talosctl -n 192.168.121.232 disks --insecure

You should get an output similar to the image below.

Installing Talos

Before installing Talos, we first need to generate a “machine configuration”. A machine configuration is a YAML file you’d use to configure a Talos Linux machine.

We will start by picking an endpoint IP in the vagrant-libvirt subnet that’s not used by any nodes, e.g. — 192.168.121.100.

Run the following command to generate the machine configuration:

talosctl gen config my-cluster https://192.168.121.100:6443 --install-disk /dev/vda

The above generates YAML configuration files for a cluster named “mycluster”. We will use these files to configure Talos nodes by applying them with talosctl commands, allowing each node to join and operate as part of the Kubernetes cluster.

Edit controlplane.yaml to add the virtual IP 192.168.121.100 to a network interface under .machine.network.interfaces:

machine:
network:
interfaces:
- interface: eth0
dhcp: true
vip:
ip: 192.168.121.100

Apply this configuration to one of the control plane nodes:

talosctl -n 192.168.121.232 apply-config --insecure --file controlplane.yaml

After applying the above config, we will enable our shell to use the generated talosconfig and configure its endpoints (the IPs of the control plane nodes):

export TALOSCONFIG=$(realpath ./talosconfig)

talosctl config endpoint 192.168.121.232 192.168.121.68

Now we can bootstrap the Kubernetes cluster from the first control plane node:

talosctl -n 192.168.121.232 bootstrap

The bootstrap command sets up the initial etcd database (used for storing cluster state) on that node becoming the initial leader in the control plane, and once the bootstrap is complete, additional control plane and worker nodes can join the cluster.

Now apply the machine configurations to the remaining nodes:

talosctl -n 192.168.121.68 apply-config --insecure --file controlplane.yaml

talosctl -n 192.168.121.150 apply-config --insecure --file worker.yaml

After a while, we can see that all the members have joined by running the following command:

talosctl -n 192.168.121.232 get members

Interacting with the Kubernetes cluster

To interact with the Kubernetes cluster we need to first retrieve the kubeconfig from the cluster using the following command:

talosctl -n 192.168.121.232 kubeconfig ./kubeconfig

Now we can view all the nodes in the cluster with the kubectl get node command:

kubectl --kubeconfig ./kubeconfig get node -owide

You will see an output similar to the image below:

Now you have a highly available Talos cluster running!!!

Troubleshooting

While walking through the demo, you might encounter a “Call to virStoragePoolLookupByName failed” fail error as shown in the image below.

The error message indicates that libvirt (via virsh) is trying to access a storage pool named ‘default’, but it can’t find it. This typically occurs when the default storage pool has either been deleted, is not defined, or has not been started.

You can check if the default storage exists with the following command:

virsh pool-list --all

If the default storage pool does not exist, you can define with the following command:

virsh pool-define-as default dir --target /var/lib/libvirt/images

Cleaning up

We can destroy the vagrant environment and remove the ISO image by running the following commands:

vagrant destroy -f

sudo rm -f /tmp/metal-amd64.iso

Kubernetes Anywhere: The future of Kubernetes?

In this article, we’ve explored what Kubernetes on-premises is, the reasons why organizations might choose to deploy it in their own infrastructure, and the key considerations involved in such a deployment.

Finally, we’ve provided a guide on how to get started with Kubernetes on-premises in a provider-agnostic manner, helping you take the first steps toward a successful on-prem deployment.

Kubernetes has evolved significantly over the years, and its growth shows no signs of slowing down, driven by one key factor: its vibrant and ever-expanding community. Today we see Kubernetes used in many industries across the world, tomorrow, we might see Kubernetes more in our homes as smart IoT technologies grow.

At Severalnines, we’ve been contributing to the database on Kubernetes community over the years, building CCX Cloud and CCX for CSPs, Kubernetes-based DBaaS implementations for end-users and Cloud Service Providers, respectively.

In our next post, we will dive into the world of Container Storage Interfaces (CSIs) and walk you through how they help make Kubernetes run stateful applications at scale. In the meantime, don’t forget to follow us on LinkedIn and X or sign up for our newsletter to stay informed of the latest in the database world.

The post Kubernetes on-premises: the why, what and how to implement appeared first on Severalnines.

From Conf42 to SREdays, the brothers Mark and Miko Pawlikowski bring developers, DevOps engineers, analysts, executives, etc from all over the world to learn from each other. Conf42 is a series of online tech events that happen throughout the year, while SREday is an in-person event in major cities worldwide.

In June, Severalnines spoke at Conf42 Observability on how ClusterControl can help organizations simplify database troubleshooting. In September we gave an in person presentation at the SREday London.

This post will summarize our presentation, and share highlights from our experience – from mind-blowing talks to eureka moments.

Database Operations (Ops) Your Way

Databases are the foundation of software applications. They serve as the organized repository for data, enabling applications to store, retrieve, and manipulate information efficiently. We all know there are different types of data models, different types of databases, and different ways to host them.

Most conversations around databases today have gone beyond deciding what type of database to use, to “how best to deploy the database for scalability and cost efficiency”. And a recurring debate is whether to deploy on Kubernetes or stick with VMs and bare metal.

At SREday London 2024, we presented on how cloud native technologies are the building blocks of modern database software, showing how organizations are increasingly deploying and scaling their databases on Kubernetes with the help of Operators and with Kubernetes using platforms such as CCX.

During the event, some attendees we chatted with were excited about databases on Kubernetes, while some shared that they wouldn’t let Kubernetes near their databases.

The choice to deploy on Kubernetes, with Kubernetes, or any other means depends on your needs and capacity. What is more important is that you are doing your “Database Operations Your Way”, hence the importance of Sovereign DBaaS.

A Sovereign DBaaS model empowers you to tailor your database operations to your exact needs. Choose the databases, tools, and infrastructure that best suit your requirements, whether you prefer open-source, proprietary, or a combination.

The level of control is entirely up to you. From coding operations from scratch to leveraging off-the-shelf solutions, and from on-premises to hybrid cloud deployments, the possibilities are endless. The only question is, how much control do you value and require for your data stack?

Kubernetes at the center stage in SRE and DevOps

As you’ve already noticed in the past few years, almost everyone is either going “Cloud Native” or talking about it. At SREday London, there was a good number of talks on Kubernetes. The following is brief summary of some of those talks.

Platform engineering and Kubernetes maturity

The K8s talks began by discussing the challenges and opportunities of maturing platform engineering initiatives using Kubernetes. Speakers emphasized the importance of balancing people, process, policy, and technology to achieve a sustainable path forward. The CNCF platform maturity model was introduced to guide organizations through this process.

Scaling, optimizing, and upgrading Kubernetes

Several talks focused on leveraging Kubernetes to improve infrastructure efficiency and scalability. Karpenter, a tool for managing node provisioning, was highlighted as a powerful solution for automating these tasks.

In optimizing Kubernetes cluster resources, the approach “Virtual Kubernetes cluster” was discussed. Virtual clusters are fully functional Kubernetes clusters nested inside a physical host cluster providing better isolation and flexibility to support multi-tenancy.

Additionally, speakers explored strategies for deploying and managing microservices across multiple Kubernetes environments.

Debugging and Troubleshooting Kubernetes

The conference also addressed the challenges of debugging and troubleshooting Kubernetes clusters. Speakers shared techniques for diagnosing and resolving common issues, such as memory leaks and network problems. eBPF tools like Cilium were presented as valuable aids for troubleshooting complex Kubernetes environments.

Will you promote AI to a senior engineer?

The first Keynote discussed automated incident management, where the speaker shared their experience adopting AI in their incident management workflow. The overall discussion can be summarized as, “AI is a good junior engineer.”

The distinction between a junior and senior engineer primarily lies in their level of experience, knowledge, and responsibilities. Promotion from junior to senior engineer requires a combination of factors. Experience is crucial, as consistent and successful performance over several years is often expected.

With the advancement of AI, and the creation of new AI tooling, both assisted and independent (autonomous), would you ever give AI full control? As of today…

…most likely not!!

Other AI and automation talks discussed where you would use AI more, and one is monitoring and observability. With AI’s ability to analyze vast amounts of data from monitoring tools, it can identify anomalies, predict failures, and provide proactive recommendations. This helps teams stay ahead of potential issues and optimize system performance.

Automated reporting and alerts can sometimes lead to “noise” and “alert fatigue” which results in decreased efficiency and increased stress among engineers. Another talk at the event delved into ways to group alerts based on content similarity and demonstrated methods to filter events based on various conditions.

Tackling industry challenges: teamwork, discipline, and strategy

Success in the technology space often hinges on the ability to navigate and overcome many challenges. Whether it’s facing economic downturns, technological disruptions, or intense competition, organizations and individuals alike must possess a strategic mindset and a collaborative approach to thrive.

A soothing talk at SREday London by Steve Wade who shared his experience going from professional football to technology due to a life changing injury shined a bright light on the mindset we should have today. As things get tough, it should not put us down, but cause us to innovate and grow.

Catch up on past SREday events

SREday London 2024 was a great event, hosting over 200 attendees and 50 speakers who delivered insightful talks on different areas of infrastructure engineering today. One of the highlights we got from an attendee was how great the venue was in enabling networking and collaboration.

To watch a replay of the talks I highlighted in this blog post and past talks from other SREday events, head over to the SREday YouTube channel.

At Severalnines, for over a decade, we have powered hundreds of companies to take control of their data through our products, i.e. ClusterControl and CCX.

Check out our Sovereign DBaaS Decoded podcast, where we interview industry leaders about implementing open-source databases, creating cloud vendor-neutral environments, and mitigating business risks, including data sovereignty and compliance requirements.

You can also join our “Sovereign DBaaS Community.” Whether you’re a seasoned database operations pro or a novice looking to connect, learn, and grow – join now!

Follow us on LinkedIn and X for more great content in the coming weeks. Stay tuned!

The post SREday London 2024: Our conference highlights appeared first on Severalnines.

Supported by the growing maturity of operators, many organizations have graduated from using Kubernetes to deploy and manage stateless applications to adopting it for all kinds of stateful workload deployment in the last 4 years.

But, running databases on Kubernetes is much more than deployment and can include Day 2 operations like backup and recovery, which are critical in any database lifecycle, regardless of where they are deployed.

For high availability, you would store your backups externally outside of your database node and a very common place organizations store them is an s3 compatible object storage service — let’s start there.

I will walk you through the steps involved in setting up backup and recovery to Amazon s3 for your MySQL database clusters deployed on Kubernetes with the “MySQL Operator for Kubernetes”.

Prerequisites

To follow along with this article, ensure you have the following prerequisites in place:

• A Kubernetes cluster. The demo in this article was done on a DigitalOcean cluster, but you can use minikube — a local single Node Kubernetes cluster.
• The kubectl command-line tool configured to communicate with the cluster.
• An AWS account for S3 access. You can create a free account if you don’t have one.

In your AWS account, create an S3 bucket with a choice name. Be sure to remember the bucket name and region you created it in, as you will use these details in the operator’s configuration.

The demo in this blog post will use the following:

• Bucket name: k8s-mysql-dev-bucket
• Region: us-east-1

Installing MySQL Operator for Kubernetes

There are several ways to install the MySQL Operator for Kubernetes. The documentation highlights two methods: using Helm or manually applying manifests using kubectl. Throughout this post, we will apply manifests using kubectl for better clarity and understanding.

To install the operator, first deploy the Custom Resource Definition (CRDs) it uses with the following command:

kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-crds.yaml

Note: “trunk” in the URL references the latest MySQL Operator for Kubernetes release on Github.

The above CRDs create the required custom objects (MySQLBackup, InnoDB clusters, etc.) that tell Kubernetes how to handle the special operator’s resources.

After running the command, the output would be similar to the image below.

Next, deploy the operator, which also includes RBAC definitions, as noted in the output below.

kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-operator.yaml

You can switch the kubectl context to the mysql-operator namespace for future commands.

kubectl config set-context --current --namespace=mysql-operator

Next, you will deploy a MySQL InnoDB Cluster. The backups you will create will be of the cluster.

Deploying a MySQL InnoDB Cluster on Kubernetes

To create a MySQL InnoDB Cluster, first you need to create a secret with credentials for your intended MySQL root user to use in performing administrative tasks in the cluster.

kubectl create secret generic mypwds \
  --from-literal=rootUser=root \
  --from-literal=rootHost=% \
  --from-literal=rootPassword="test"

The above generic K8s secret creates a root user root and password test for demo purposes.

Now define your MySQL InnoDB Cluster, with a reference to the mypwds secret. For this demo, you can create a file mysql-backup-cluster.yaml :

apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
  name: mysqlcluster
spec:
  secretName: mypwds
  tlsUseSelfSigned: true
  instances: 3
  router:
    instances: 1

The above configuration creates an InnoDB Cluster with three MySQL Server instances and one MySQL Router instance. Apply the configuration with the following command:

kubectl apply -f mysql-backup-cluster.yaml

You can watch the cluster creation process with the kubectl --watch flag

After the cluster is created, you can connect to it with the InnoDB Cluster Service by using the MySQL shell or port forwarding. To use port fowarding, run the following command:

kubectl port-forward service/mysqlcluster mysql

The above port forwarding creates a redirection from your local machine to use a MySQL client, such as mysql. With the port forward running, open another terminal and run the following command to open up the MySQL shell: 

mysql -h127.0.0.1 -P3306 -uroot -p

With the shell running, to test the recovery of the backup later on in this demo, create a test database k8_mysql_backup_test_db and populate it with some test data using the SQL queries below.

CREATE DATABASE k8_mysql_backup_test_db;
USE k8_mysql_backup_test_db;
CREATE TABLE users (
    id INT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(50),
    email VARCHAR(100)
);
INSERT INTO users (name, email)
SELECT CONCAT('User', LPAD(ROW_NUMBER() OVER(), 3, '0')),
       CONCAT('user', LPAD(ROW_NUMBER() OVER(), 3, '0'), '@example.com')
FROM information_schema.columns
LIMIT 100;



SHOW DATABASES;
SELECT * FROM users LIMIT 5;

Verify that the data has been created with the SHOW DATABASES; and SELECT * FROM users LIMIT 5; queries as in the image below.

You can see the database and test users in the screenshot above. Note them as you will use it to verify the backup.

Setting up MySQL backups on Kubernetes to AWS S3

Now that you have an InnoDB cluster running, you can take backups and send them to S3 for storage.

The MySQL operator for Kubernetes requires two components to successfully create a backup:

• Backup profile: Configures the general backup structure, from storage location to MySQL export options.
• Backup schedule: Defines when the backup will run. You specify this as a cron expression for regular backups, or with no schedule when performing one-off backups.

You create backup profiles and schedules independently of each other. This lets you run multiple backups on different schedules using the same profile.

Each profile and schedule is associated with a specific database cluster. They’re created as nested resources within the InnoDBCluster objects. Each database you create with the operator needs its own backup configuration.

Before you create a backup profile and schedule, create a Kubernetes secret that contains your AWS credentials so that the operator can upload your backups straight to S3 object storage.

Note: There is another main object related to MySQL backups, “Backup request”. With this object, requesting a backup initiates a new object that creates a new pod to perform the backup. To learn more about “Backup request” and other MySQLBackup resource options, see its documentation.

Creating a Kubernetes secret for AWS s3

Create a YAML file s3-secret.yaml, add the following configuration, replacing the placeholders with your credentials and region:

apiVersion: v1
kind: Secret
metadata:l
  name: s3-secret
type: Opaque
stringData:
  credentials: |
    [default]
    aws_access_key_id = <YOUR-ACCESS-KEY-ID without encoding>
    aws_secret_access_key = <YOUR-SECRET-KEY without encoding>
  config: |
    [default]
    region=<YOUR-REGION>

Then, use kubectl to create the secret:

kubectl apply -f s3-secret.yaml

Creating a backup profile for the MySQL operator for Kubernetes

You define backup profiles and schedules in your database’s spec.backupProfiles and spec.backupSchedules fields respectively.

To create a backup profile for this demo, copy and paste the following YAML configuration into your mysql-backup-cluster.yaml file spec.

backupProfiles:
    - name: s3-backup
      dumpInstance:
        dumpOptions:
        storage:
          s3:
            bucketName: k8s-mysql-backup-bucket 
            prefix: /mysql-backup 
            config: s3-secret 
            profile: default 
            endpoint: https://s3.us-east-1.amazonaws.com 

In the above backup profile:

• name defines the name of the backup profile which the backup schedules would reference.
• dumpInstance is the parent spec for the dumpOptions and storage configuration.
• dumpOptions, if left blank, performs a MySQL Shell Dump by default.
• S3 storage configuration:
  • bucketName defines the name of the S3 bucket where the dump will be stored, and this must be an existing bucket as earlier highlighted in this blog.
  • prefix defines the path in the bucket where the dump files are stored.
  • config defines the name of the secret with S3 configuration and credentials.
  • profile defines the profile being used in configuration files.
  • endpoint is an override endpoint URL to specify the AWS region where the bucket is hosted.

Next, we will define a backup schedule that references the backup profile to create backups.

Creating a backup schedule for the MySQL operator for Kubernetes

As mentioned earlier, you create backup schedules in spec.backupSchedules field. The following YAML configuration defines a backup schedule referencing the s3-backup profile you defined earlier.

backupSchedules:
    - name: test-s3-backup-schedule
      schedule: "* * * * *" 
      backupProfileName: s3-backup 
      enabled: true 

In the above schedule:

• The name defines the name of the schedule
• schedule defines the cron expression for the backup. The above expression schedules the every minute. You can edit this to per day, bi-weekly, month, etc.
• backupProfileName references the desired backup profiles’s name.
• enabled let’s you enable or disable a backup schedule.

Putting it all together, the full configuration of mysql-backup-cluster.yaml will be:

apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
  name: mysqlcluster
spec:
  secretName: mypwds
  tlsUseSelfSigned: true
  instances: 3
  router:
    instances: 1
  backupProfiles:
    - name: s3-backup
      dumpInstance:
        dumpOptions:
        storage:
          s3:
            bucketName: k8s-mysql-backup-bucket 
            prefix: /mysql-backup 
            config: s3-secret 
            profile: default 
            endpoint: https://s3.us-east-1.amazonaws.com 
  backupSchedules:
    - name: test-s3-backup-schedule
      schedule: "* * * * *" 
      backupProfileName: s3-backup 
      enabled: true 

Apply this configuration to tell the operator to create the backups based on the schedule (every 1 minute).

kubectl apply -f mysql-backup-cluster.yaml

After applying the configuration, you will see that backup creation jobs (pods) are created every minute per backup, as in the image below.

Verifying the backup creation

To verify that the backups are done according to the schedule and sent to Amazon s3, head over to your AWS console and click on the s3 bucket you set; in the mysql-backup directory you set as the prefix, you will see all the backups the operator creates as in the image below.

Looking closely in the image above, you will notice the timestamp at the end of each folder name. You can compare the three folders and see that their timestamps are one minute apart, as specified in the backup schedule.

We will now validate the backup by performing a test restore.

Restoring a backup with the MySQL operator for Kubernetes

Using the spec.initDB, the operator can initialize new database clusters using previously created files from the dumpInstance object. This allows you to restore your backups straight into your Kubernetes cluster, which is useful in validating your backup, in recovery situations or when migrating an existing database to Kubernetes.

As mentioned, we will perform a restore in the context of validating our backup. To validate the backup, you can create a new Kubernetes cluster, install the operator with the same steps as above and create an InnoDB cluster telling Kubernetes to initialize the database cluster with the backup.

apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
  name: mysql-recovery-cluster
spec:
  secretName: mypwds
  tlsUseSelfSigned: true
  instances: 3
  router:
    instances: 1
  initDB:
    dump:
      name: initdb-dump
      storage:
        s3:
          prefix: /mysql-backup/mysqlcluster-test-s3-backup-schedule241007114206
          config: s3-secret
          bucketName: k8s-mysql-backup-bucket
          profile: default
          endpoint: https://s3.us-east-1.amazonaws.com

In the above configuration, the prefix indicates the path to the dump folder from which to recover the backup. Also recall the timestamp so you will be sure to recover with the exact backup you need.

Assuming you save the configuration in a file named recovery.yaml, apply it with the following command:

kubectl apply -f recovery.yaml

Verifying the recovery

With all the pods running, using the same port forwarding methods as earlier explained in the article you can connect to the cluster and verify that the recovery was successful.

In the above image, you can see that the database and its dataset were added to the new cluster upon creation.

Note: It is important to note that sometimes the InnoDB Cluster pod can be stuck in a terminating state due to issues preventing graceful termination of the pod, such as “node deletion.” You can resolve this by removing the finalizer from the pod’s metadata, and the pod will be in a ready state again. For example:

kubectl patch pod <mysql-pod-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Wrapping up

This walkthrough showed you the steps in setting up backup and recovery to S3-compatible storage using the “MySQL Operator for Kubernetes.” Beyond backup and recovery, there are other critical Day 2 operations, such as scaling up/out, upgrades, security, etc., that go into managing a database on Kubernetes.

In most production use cases, you won’t want to do this manually and/or need to provide a seamless user interface to your database end users who may not be familiar with how Kubernetes works. In these cases, you can use CCX, a Kubernetes-based DBaaS that can be implemented by organizations who want to provide DBaaS internally, e.g., platform engineering teams, or externally, e.g., Cloud Service Providers (CSPs).

To learn more about CCX, explore our CCX for CSP page, or reach out today to discover how CCX can elevate your business to new heights!

The post A MySQL backup and recovery tutorial on K8s using the MySQL Operator for Kubernetes appeared first on Severalnines.

Data security is crucial in today’s world. Personally Identifiable Information (PII) is everywhere – emails, names, addresses, personal ID numbers and so on. There are regulations in place that organizations managing PII have to comply with and fines for those who fail to comply are significant. This is a known threat and organizations are taking steps to ensure the access to the critical data is limited and monitored.

What about other threats, though? What if it is not a cyber security threat? What if it is a good old hardware failure? Say, on a big scale, just to mix things up? Can your organization survive losing most or all of its data? Your customer data is gone. Not just stolen, which would still be a huge problem, but literally gone. Puff! It does not exist anymore on your database servers. Maybe even those servers do not exist anymore as well? What now? Are you prepared for such a catastrophic event?

The 3-2-1-1-0 Golden Rule for backups

You may have heard about a backup Golden Rule: 3-2-1-1-0. If not, let’s quickly explain what it means.

3 – Keep at least three copies of your data

This one is quite simple — you want to create three copies of your data. In case there are problems with any one of them, you still have two more copies to work with.

2 – Store the backups on at least two different storage mediums

You don’t want to use the same storage medium to keep all of the backups. If you store the data on the disk array, disks may fail. If you upload the data to some sort of cloud storage, you rely on internet access to download that data. What if, during the incident, your internet access is unreliable?

1 – Store at least one copy in an offsite location

This is quite self-explanatory — you don’t want to put all eggs in one basket. We have seen whole data centers affected by failures. You want to keep at least one copy of your data in some other place, in case you would need to rebuild your infrastructure from scratch, using a different ISP.

1 – Store at least one copy offline

Storing data offsite is not good enough to cover all of the bases. If your infrastructure has been compromised, anything that is online may, potentially, be accessed and affected. To put it bluntly, someone who gained full access to the system can just destroy a VM where you store the backup, even if the VM is stopped. You want to use storage that can be “unplugged” from the infrastructure. External disk drives, tapes etc. Sure, it feels like the 90’s here but this is really secure. Think of it, if tape is out from the drive, no one, no matter which server or laptop he gained access to, can access it in any way other than physically.

0 – Test your backups to verify there’s zero errors during restore

This is again, self-explanatory. You may have heard about Schrodinger’s backup – each backup is in an unknown state, it may work or may not and you won’t know until you attempt to restore it. Therefore, testing each backup is critical – only then you can verify that a backup can be restored correctly and it can be reliably used if there’s a need for it.

How ClusterControl can help to implement 3-2-1-1-0 rule

ClusterControl is a database ops orchestration platform that, among others, allows you to automate backup handling and; as such, it can be used, with a little bit of additional work, to implement the 3-2-1-1-0 rule. Let’s see how we can do that.

First of all, let’s create a backup. You can either create an ad hoc backup or schedule one.

Here, we want to schedule a backup.

Now, this is going to be a full backup so we are picking am xtrabackup (full) as a backup method. The list of backup methods depends on the cluster type. In this case we have a Percona Server cluster of a primary node and a couple of replicas. We can pick a replica to take the backup from, to minimize the impact on the primary. Let’s not forget about enabling a cloud upload – ClusterControl will upload the backup to the cloud (to a S3 compatible storage) after the backup completes.

Here, there are 3 settings we want to pick:

1. Enable encryption – as we are uploading data to S3, we want it to be encrypted. 

2. Enable the option to failover if the node is down. What does it mean? At the previous step we picked a backup host – one of the replicas, 192.168.10.185. That will be the node where the backup will be executed. Still, if that node, for whatever reason, would be unavailable at the time the backup is scheduled, this option makes ClusterControl use another node from the cluster. In this case we decided to let CC pick any node that is available (Auto select) but you can also pick a particular one, let’s say, another replica. 

3. Enable backup verification.

To verify the backup we have to have a node up and running with SSH access configured in a way that ClusterControl can connect to it. After the backup is completed, a given node (in our case 192.168.10.151) will have database software provisioned and then the backup will be copied to it and restored. 

N.B. You can configure the delay after which the restore process will start and can decide whether the server will remain up and running or stop after the test restore is completed. If the restore process finishes successfully, the backup is assumed to be working.

Now we want to configure the local storage part. Backups can be kept on the backup node, which may have its pros and cons, or transferred over to the controller host. Here you can pick a location where the backup should be kept and; quite often, it is an NFS or some other mount from a “backup” server that has been configured with a large amount of storage.

The next step is to configure cloud access. If you have already set up the integration, you can pick it by its name. If you haven’t done that yet, you can create new credentials.

ClusterControl supports AWS, GCP, Azure and any S3 compatible storage, for example from OpenStack.

For this example, we’ll use AWS which requires you to define a name, pass credentials and pick the region.

Once the credentials are created, you need to define the name of the S3 bucket. If needed, you can change the retention period. You can also keep the backup only in the cloud by enabling the “Delete after upload” option, but we don’t want to do that. We want to keep the data in multiple places, as per 3-2-1-1-0 rule.

We can now define when the backup is supposed to be executed – in our case it’s 3:00am UTC every day.

Finally, review the configuration and proceed with creating the schedule.

It will become visible in the “Schedules” tab.

Ok, that part is done but what exactly did we do and how does it compare with the 3-2-1-1-0 golden backup rule?

1. We have created a backup that is stored in multiple locations:

• a) Cloud — S3 bucket

• b) Via controller host — on a backup server

• c) Technically we can argue that the backup verification server also stores the backup, just in a restored form

2. We store the data in two different mediums:

• a) Disks for the backup server and backup verification server

• b) S3 bucket

3. We store one copy of the backup offsite

• a) Cloud provider via S3 bucket

4. We ensure that the backup has zero errors during restore via verifying each backup after it is created

Out of the 3-2-1-1-0 rule we are missing the only one — we do not store data offline. We have a very good reason for that though: you cannot really automate it from a host inside of your network (where ClusterControl instance is located) because that would kind of defeat the purpose of ensuring that data is totally offline. There are many ways in which you can easily implement this point. Let’s write down some options that would work quite well:

1. You can have a computer that is started manually on a daily basis, downloads the backup from either S3 bucket or through rsyncing it from the controller host. Once this is done, you would stop the box and physically unplug it from the power source.
2. A backup server may have a tape drive and a script that executes once the backup is completed. It would copy the latest backup to the tape and then, let’s say, umount it. You would then swap the tape and put in an empty one, ready for the next day’s backup.

As you can see, ClusterControl gets you across all of the aspects of the 3-2-1-1-0 rule that can be automated, making the implementation of that best practice a breeze. We always hope that we will never have to use our backups; but, should that time come, ClusterControl has you covered and ensures that you can recover even from the most dire scenarios.

To try it out for yourself by signing up to ClusterControl free for 30-days (no credit card required). In the meantime, stay abreast of the latest in open-source database operations best practices by following us on Twitter or LinkedIn or signing up to our monthly newsletter.

The post Implementing the golden backup rule for your databases appeared first on Severalnines.

Clustercontrol (CC) v2.2.0 is now live, offering a new Redis alternative in response to recent licensing changes, along with improved support for MariaDB 11.4 and Ubuntu 24.04. Let’s explore the key updates and see what’s new in Clustercontrol v2.2.0!

Redis shifts to source-available licensing

Beginning with v7.4, Redis will adopt a dual-licensing model under the Redis Source Available License (RSALv2) and the Server Side Public License (SSPLv1). Consequently, Redis will no longer be distributed under the three-clause Berkeley Software Distribution (BSD) license.

Enter Valkey, a Redis alternative

Valkey is a powerful, open-source (BSD-licensed) key/value datastore designed for high-performance workloads. Whether you need caching, message queues, or a reliable primary database, Valkey delivers the versatility to handle it all.

Clustercontrol streamlines Valkey day 2 ops with essential features:

• Deployment & importing database clusters
• System and database dashboards
• Cluster topology visualization
• Configuration file management
• Backup and restore (local storage and retention settings)
• Scaling (add/remove primaries or replicas)

Learn how to create a Valkey database cluster, import an existing cluster, perform backups, and manage cluster/node actions in Clustercontrol.

Support for MariaDB 11.4 LTS

MariaDB Community Server 11.4 marks the first long-term support (LTS) release in the 11.x series, providing five years of bug fixes. Discover how to upgrade from version 10.11 to 11.4, and see this MariaDB resource for more details.

Support for Ubuntu 24.04 LTS

Ubuntu 24.04 LTS will be the first to offer 12 years of support, with Canonical planning to include Linux Kernel 6.8, which improves hardware support, particularly for CPUs and GPUs. For more information, check out Canonical’s release.

Explore more updates in Clustercontrol v2.2.0

Clustercontrol v2.2.0 includes support for FIPS-140-2, the ability to import Elasticsearch GA 7.x and 8.x clusters, and expanded backup options for Redis. Additionally, enhancements have been made to MongoDB, Vault integration, HAProxy, and the Clustercontrol Operations Center (CC-Ops-C). For all the details, refer to our changelog.

Wrapping up

Clustercontrol 2.2.0 introduces Valkey and new features tailored for open-source databases, helping you optimize and scale modern workloads. Stay tuned for our next release, featuring enhancements for PostgreSQL, MySQL, and more!

New to Clustercontrol? Try our Enterprise edition free for 30 days, complete with technical support to guide you every step of the way.

The post Clustercontrol adds Valkey in latest release appeared first on Severalnines.

While Kubernetes (K8s) was originally designed for stateless workloads, advancements over the years, including enhanced support for Persistent Volumes, StatefulSets, and Operators, have made it possible to run stateful applications on Kubernetes. 

These new features and Kubernetes’ inherent scalability and resilience have made it an attractive platform for deploying databases. As a result, many organizations are already operating their databases on Kubernetes, and many more are eager to follow suit. However, some organizations are skeptical about running databases on K8s in production. 

This blog will start by walking you through the history of database deployments. After that, it will discuss running databases on Kubernetes in 2024 and running databases with Kubernetes using the Severalnines’ CCX for CSP. 

A brief history of database deployments: from mainframe to the cloud and container revolution

The evolution of database deployment reflects the ever-changing landscape of IT infrastructure and the growing demands of modern applications. From humble beginnings with mainframe computers, we’ve witnessed a shift towards greater abstraction, automation, and flexibility.

Early days of database deployments

In the early days of databases — 1960s, vast amounts of data were stored in the world’s new mainframe computers—many of them IBM System/360 machines. While revolutionary at the time, this approach came with limitations in terms of cost and rigid data models (a network model called CODASYL and a hierarchical model called IMS). 

To solve the problem of rigid data models, Oxford-educated mathematician Edgar “Ted” Codd, published a paper in 1970 that introduced the relational database, and his ideas changed the way people thought about databases. 

Several advancements in technology led to the creation of smaller servers over the years to reduce computer costs, maintenance, accessibility, etc. 

Pre-Cloud Era

Before cloud computing, organizations had to buy and manage their own hardware and software for databases, which was expensive and inflexible, especially for smaller and newer companies. This paved the way for cloud computing, which promised easier and more affordable database solutions.

It’s worth noting that on-premises database deployments remain relevant and continue to grow in today’s Cloud-First Age. Industries and organizations with specific requirements such as data sovereignty, regulatory compliance, low-latency access, and other use cases, drive continued growth and innovation in on-premises deployments. 

The Cloud Era

The rise of cloud computing marked a significant turning point in database deployment in the late 2000s and early 2010s. Cloud-managed databases, offered by large cloud providers (hyperscalers), simplified the provisioning and management of database infrastructure. The heavy lifting of hardware procurement, maintenance, and scaling was offloaded to the cloud provider, allowing organizations to focus on their applications.  

Database-as-a-service (DBaaS) solutions further abstracted away the underlying complexities. DBaaS offerings provided on-demand, scalable databases with pay-as-you-go pricing models. This democratized access to databases, making them more accessible and cost-effective for businesses of all sizes.  

The Container Revolution

With Docker as the leading platform, Container technologies revolutionized application deployment, including databases. Containers package applications and their dependencies into isolated, portable units, ensuring consistent behavior across different environments. This facilitated easier and faster deployment of databases across various platforms, from development to production.  

In the late 2010s, Kubernetes emerged as the de facto standard for orchestrating and managing containerized applications. Kubernetes provides powerful automation and scaling capabilities, enabling the deployment, management, and scaling of complex systems across large clusters of machines.  

The road to databases on Kubernetes started in its early days with the introduction of robust support for StatefulSets (initially known as “PetSet”) in version 1.5, released in 2016.

Running Databases on Kubernetes in 2024

Databases on Kubernetes have come a long way in the past five years. In February 2018, Kelsey Hightower, an early contributor and evangelist of Kubernetes, made a X thread on why he wouldn’t run stateful workloads on Kubernetes. Fast-forward to February 2023, he made another thread saying “ You can run databases on Kubernetes because it’s fundamentally the same as running a database on a VM.”

This shows how Kubernetes data workloads have come a long way since the introduction of StatefulSets.

The improvements and adoption of databases on Kubernetes wouldn’t have been possible without the Data on Kubernetes Community(DoKC). The DoKC was founded in 2020 to advance the use of Kubernetes for stateful and data workloads, and unite the ecosystem around data on Kubernetes (DoK). 

Since its inception, the DoKC has helped hundreds of organizations and individuals deploy databases on Kubernetes through its resources, meetups, reports and events. 

How to run databases on Kubernetes

How you run databases on Kubernetes today depends on your organization, the databases you want to run, and your team’s technical expertise. 

Running databases on Kubernetes involves the separation of the control and data plane layers. By default, Kubernetes has this separation with the Kubernetes API server acting as the interface for its data plane to request computing resources, while the control plane takes care of mapping those requests to the underlying IaaS platform.

This default Kubernetes separation can be applied to database management on Kubernetes with Operators taking the role of the control plane.  Operators are agents in Kubernetes that have the responsibility to reconcile the current state with the desired state. 

Learning to deploy databases on Kubernetes manually and then worrying about performing Day 2 operations such as scale up/out, upgrades, security, etc. introduces operational overhead in your organization and does not truly unlock the value of using Kubernetes.

With operators, you can utilize constructs like Custom Resource Definitions (CRDs) and the resulting Custom Resource (CRs) to simplify database deployments on Kubernetes.

Image source: Data on Kubernetes whitepaper

More detailed information can be found in the whitepaper created by the Cloud Native Computing Foundation (CNCF) Storage TAG and the DoKC, Data on Kubernetes – Database Patterns Whitepaper. The whitepaper describes the patterns of running databases on Kubernetes. 

The central message is clear: the decision to run databases on Kubernetes should align with your overall infrastructure strategy. For example, if the majority of your applications and services are not running on Kubernetes, then deploying your databases on Kubernetes may not yield the optimal benefits. And this is true because users won’t be able to provision full environments.

3 reasons for running databases on Kubernetes

Now that you have a brief overview of running databases on Kubernetes; now the question is, “Why would I want to run databases on Kubernetes? What is the gain?” Here are 3 key reasons:

• To break vendor lock-in: When organizations start using a cloud DBaaS, they slowly become ensnared in the single vendor’s ecosystem and face several challenges, such as limited flexibility, reduced bargaining power, higher costs etc. By running databases on Kubernetes, organizations can gain the flexibility to move their workloads across different cloud providers or on-premises environments without significant changes.

• Scalability and elasticity: Kubernetes makes it easy to scale database resources up or down dynamically based on demand, ensuring they can handle varying workloads without over-provisioning resources or experiencing performance bottlenecks. 

Yes, Kubernetes tops out at 5000 nodes per cluster and 110 pods per node, but with micro-clusters, the bottleneck is removed enabling an enormous scale. 

• High availability and resilience: Kubernetes provides built-in mechanisms for self-healing and failover, ensuring that your databases remain available even in the event of node failures or other disruptions. 

For more reasons as to why you should consider running databases on Kubernetes,  check out this Sovereign DBaaS Decoded podcast episode – Mad world – creating a new stateful world with Kubernetes – with Eddie Wassef, VP and Cloud Architect for Vonage.

Concerns with running databases on Kubernetes

Running databases on Kubernetes provides great benefits, and many organizations are championing this approach. However, just like there was a time when people felt uncomfortable running a database on virtual machines, organizations, and individuals have concerns today. The following are two general concerns: 

Complexity of Operators?

About a year ago in a conversation on LinkedIn, a DevOps lead shared an experience with a Postgres operator, highlighting issues from image policy, maintaining the database cluster to fine-tuning the database which resulted in migrating DBs back to VMs and bare-metal servers. 

One can understand the overall cost of the experience and why they resulted in returning to VMs.

Beyond technical issues, the additional abstraction layer makes it difficult to understand the underlying processes and behaviors, especially for users who are not intimately familiar with Kubernetes or using closed source operators. 

Expertise and documentation

One can argue that the complexity of operators is due to a shortage of expertise and issues with documentation. The 2022 Data on Kubernetes report showed that lack of talent was one of the top challenges facing the adoption of data on Kubernetes. Kubernetes has “many moving parts,” and good understanding, experience, and detailed documentation are critical to properly managing databases on Kubernetes.

It is important to note that every case is different, and every organization has different database needs and capacity. Some organizations run absolutely massive stateful workloads on Kubernetes and are happy about it, while others run bare metal or managed (think RDS) databases and are unhappy. And vice versa.

But then again, you may ask, “Can I enjoy the benefits of Kubernetes without having to move my workload from VMs?” We say, “Yes!!”, which brings us to the next section of this blog. 

Running Databases with Kubernetes

When we say “Running Databases with Kubernetes,” we mean using Kubernetes to manage data on VMs where the databases themselves aren’t directly deployed on Kubernetes. 

There are several ways to use Kubernetes to deploy and manage your database infrastructure. At Severalnines, we built CCX for CSPs(Cloud Service Providers), a Kubernetes-based DBaaS that helps CSPs enhance their customers’ optionality for where their data stack is located and provision high-availability databases on OpenStack infrastructure resources and other VM providers.

CCX could also be implemented by teams who want to provide DBaaS internally e.g. platform engineering teams can implement CCX to take control of their data resources while providing the same end-user experience to their development teams as traditional DBaaS, like AWS RDS. 

CCX Architecture

Built on the industry-leading ClusterControl(CC) platform by a team with extensive database automation, monitoring, and management expertise, CCX is based on a loosely coupled architecture, which means it is highly customizable — you can choose to use a different DNS service, metrics server, etc. — and gives you control over how you deploy and manage your database infrastructure.  

As you can see in the above architecture diagram, you would deploy CCX on Kubernetes. Kubernetes acts as the control plane, running CCX core backend services, including CCX UI, backup services, authentication services, CCX observability stack, etc. 

Dataplane

With CCX deployed on Kubernetes, users can deploy vanilla open-source and proprietary databases on OpenStack infrastructures, WMware, AWS EC2 instances, and GCP VMs. They can also automate the lifecycle and Day 2 Ops of these databases just as in a regular cloud DBaaS. 

CCX currently supports MySQL, MariaDB, Redis, PostgreSQL, and MS SQL Server, with support for more databases coming soon.

Interface

CCX provides a simple-to-use interface for your end users with access to powerful features. With the Admin interface, you can monitor users and manage thousands of deployed databases across your cloud.

The Terraform provider for CCX enables DevOps teams to deploy and manage database infrastructure with Terraform. The REST-based API also lets you easily integrate CCX directly with your existing workflows.

Key features of CCX

With a focus on sovereignty and control, CCX gives you the features you would want in any DBaaS plus more including:

• Effortless deployment: Empower your end users to provision high-availability databases on your infrastructure resources with just a few clicks.
  
• Open-source focus: CCX supports popular open-source databases, such as MySQL, MariaDB, and PostgreSQL, providing users with flexibility and avoiding vendor lock-in.
  
• Seamless failover and auto-recovery: Enable your end users to monitor their databases’ health and proactively address potential issues, minimizing downtime.

• Powerful backups and disaster recovery: Equip your end users with robust tools to implement sophisticated backup and disaster recovery strategies, ensuring minimal disruption to ops.
  
• Dynamic scaling options: With CCX end users can easily scale up, out, down or in, adjusting replicas and storage to meet their needs.

• Observability dashboards: With observability dashboards, your end users can gain insight into their data stacks’ performance, enabling them to pinpoint and address any issues.

To see CCX features in action, watch the CCX comprehensive demo here:

Why use CCX?

Organizations like Elastx and Lintasarta already use CCX and enjoy the following value propositions. 

Value proposition for organizations: 

• Implement a full-fledged, premium service quickly instead of committing a ton of R&D resources and time to build one from scratch.
• Provide greater value by offering a premium, DBaaS solution as an extension of your Cloud Services.

Fully customize the platform to meet your specific needs.

What’s in it for your end user?

• An intuitive end user portal to deploy, manage, and monitor database clusters.
• Advanced database knowledge is not required to leverage powerful features.
• True sovereignty over their data stack with a cloud-native experience.
• They can concentrate on their core business while maintaining compliance.

Conclusion

This blog post discussed how to run a database on Kubernetes and how you can run databases with Kubernetes using CCX. Just like in most aspects of a software development lifecycle, there are different approaches. Your choice of deploying on Kubernetes, with Kubernetes or any other means, depends on your needs and capacity. 

The journey of database deployment is far from over. As technology continues to evolve, so will the ways in which we deploy and manage our data. 

To learn more about CCX, explore our CCX for CSP page, or reach out today to discover how CCX can elevate your business to new heights!

The post An overview of running your databases on and with Kubernetes appeared first on Severalnines.

Cloud computing is a core part of today’s world with more and more organizations moving their critical applications to the cloud. This shift highlights the need to manage and protect data, especially with the growing focus on data privacy.

Take the EU-U.S. Data Privacy Framework, for example. It was created to restore data transfers between the EU and the U.S. after the EU’s top court struck down the Privacy Shield in 2020. It is likely that this framework will also be challenged in court and other countries will continue to adopt more stringent privacy legislation.

In this post, we’ll explore one concept to help organizations understand and determine whether a Sovereign Cloud solution is right for them. We’ll look at what it is, why they Sovereign Cloud’s are needed and their adoption is rising, the requirements to be considered one, their key operational characteristics and benefits, and even how you can extend the concept of sovereignty to your database layer by implementing a Sovereign DBaaS concept on top of your infrastructure.

What the Sovereign Cloud concept is

A Sovereign Cloud is a specialized cloud environment designed to comply with the data laws of specific countries and regions. It ensures that data remains where it is produced, minimizing risks associated with foreign jurisdiction overreach and enhancing trust by making sure sensitive data is under the purview of local law.

VMware popularized the concept in 2021, emphasizing its importance in addressing social and governance values, political challenges, cybersecurity threats, data protection, portability, and control. In short, the increasing complexity of global data privacy laws has shined a light on these solutions’ necessity — let’s explore deeper.

Why Sovereign Clouds are needed

As enterprises have increasingly migrated their applications to public clouds, the cloud itself is increasingly considered critical infrastructure, akin to roads, bridges, etc. Consequently, governments are thinking about what happens to the public and private data stored in the servers forming the “cloud”, and protecting the rights to it.

At the end of the day, sovereignty is a question of control; and in this case, it is a question of who has control over the data they generate. This question is evermore present in today’s increasingly competitive, fractured geopolitical landscape.

Sovereign Cloud solutions help enterprises build customer trust and mitigate these risks by ensuring there are no questions around compliance with the laws and regulations in their operating regions. These solutions protect consumer and organizational data, including personally identifiable information (PII), intellectual property (IP), including the likes of software, trade secrets, and financial information.

Why Sovereign Cloud solution adoption is rising

As regulations tighten, organizations are finding it essential to prioritize data sovereignty and compliance. In Europe, the General Data Protection Regulation (GDPR) has made it clear that firms with European customers using U.S.-based hyperscalers face significant risks due to the U.S. CLOUD Act, which allows U.S. federal law enforcement access to user data collected by U.S. domained cloud service providers, regardless of where their servers are physically located. 

This issue, highlighted in multiple Schrems judgements, has pushed many companies to seek alternatives that ensure compliance. But, this is not just an EU block movement — its adoption is further fueled by major technology companies recognizing its strategic importance to national governments and the opportunities that come with. Oracle’s emphasis on national and state government applications underscores the growing demand for sovereign regions. Oracle founder Larry Ellison has pointed out that “pretty much every government” is seeking Sovereign Cloud solutions that will allow them to secure their data within national borders. 

At the time of this post, there are 162 national data privacy laws already in place. Even in the U.S., there are quasi-GDPR regulations on the books, starting with the California Consumer Privacy Act (CCPA) in 2018, continuing with 19 additional states enacting similar laws by mid-2024, and likely ending with every state enacting its own; but, it’s not inconceivable the Federal government will enact one.

Then there are national and sector-specific regulations prompting shifts to these providers. For example, Prorenata, a Swedish Healthcare SaaS firm, moved from AWS to a local provider to ensure compliance with national and EU regulations. Jens Alm, founder of Prorenata, emphasized that hosting healthcare data on a U.S. hyperscaler could compromise compliance, a sentiment shared by many of his customers. This trend extends beyond healthcare to sectors like banking, education, and research, where data sovereignty is increasingly seen as non-negotiable.

Sovereign Cloud’s fundamental requirements

To be considered a Sovereign Cloud solution, it must deliver three results: 

Data Sovereignty

Data sovereignty ensures that data remains under the jurisdiction of the country or region where it’s created, complying with local laws and providing enhanced protection against cyberattacks and unauthorized access, e.g. even when the infrastructure is managed by a third party, they don’t have access to the data itself.

Operational Sovereignty

Operational sovereignty keeps critical infrastructure for critical applications available, accessible, and migratable, helping enterprises achieve control.

Digital Sovereignty

Digital sovereignty gives organizations control over their digital assets, including data, software, content, and digital infrastructure. It involves governance and transparency, allowing enterprises to manage access and audit their processes. This way, the assets are managed according to enforceable policies, like policy-as-code.

Sovereign Cloud’s key operational characteristics

Sovereign Cloud solutions offer a reliable way for organizations to ensure data sovereignty and compliance through a number of related feature themes, such as: 

• Access restrictions: These restrictions limit usage to specific users, software, systems, and services belonging to a particular organization, geographic region, or individuals with specific citizenship or security clearances.
• Organizational control: Defines where the Sovereign Cloud is located – whether in a service provider’s data center or within the customer’s premises.
• Compliance: Ensures adherence to governmental, regulatory, or industry requirements through specific legal, contractual, and business practices.
• Operational support: Cloud service providers must meet high standards for their employees’ security clearances, citizenship, and residency.
• Dedicated network capacity: Ranges from secure VPNs to air-gapped implementations completely isolated from the internet.
• Sophisticated encryption: Employs customer-maintained encryption keys or provider-managed keys that comply with stringent security protocols.

Benefits of Sovereign Cloud

According to Accenture, Sovereign Cloud solutions offer numerous advantages for organizations of all sizes:

• Firstly, control. Companies perceive Sovereign Cloud as a way to maintain tight control over their data, not just within their own ops, but also across their broader network of partners, employees, governments, and other stakeholders.
• Then there’s regulatory compliance. Sovereign Cloud can help organizations navigate and stay on the right side of increasingly tough data protection and privacy regulations, keep their data safe and protect their intellectual property.
• Reputation is another big one. With its strict security requirements, Sovereign Cloud can help companies build customer trust, improve their retention, and serve as a unique value proposition for prospective ones.
• Business leaders are also turning to Sovereign Cloud for the flexibility it offers in combining different cloud providers. This approach allows them to customize solutions that precisely align with their compliance requirements and business objectives for different workloads.

Sovereign Clouds offer significant benefits, but meaningful challenges:

• Control paradox: Cloud service providers still retain some control over the infrastructure. For example, your data stays within national borders but your data management practices and policies could clash with your provider’s.
• Integrability and portability: The lack of standardized protocols can staunch integration with your existing and multi-vendor architectures and migration.
• Increased costs: The specialized infrastructure and strict operational requirements the concept poses are cost drivers.
• Upstack service paucity: The concept’s ecosystem is small and upstack services like DBaaS aren’t guaranteed.

“Sovereign Cloud” solutions from U.S. hyperscalers

While each U.S. cloud provider – AWS, Microsoft Azure, and Google Cloud – now claim to offer solutions tailored for cloud sovereignty, it’s important to note that they are still subject to the US CLOUD Act, the core incongruence with GDPR. 

For example, AWS’s investment in a European Sovereign Cloud and Azure’s “Cloud for Sovereignty” solutions are designed to reassure customers about data control and compliance. However, the reality is that these measures still leave data subject to U.S. jurisdictional overreach, undermining the very sovereignty they promise, despite their advanced security features and regional customization options.

At the end of the day, while these initiatives are steps in the right direction, and also demonstrate the market’s demand for sovereign solutions, they do not solve the core issue that underlie the legal challenges like those of the Schrem’s series.

So long as the US CLOUD Act remains unchanged, these efforts do and will not satisfy the strict data sovereignty requirements mandated by regulations like GDPR.

Extending sovereignty across the stack with Sovereign DBaaS

Our Sovereign DBaaS implementation concept builds on the idea of Sovereign Cloud by extending control and compliance to every aspect of the data stack. So now you have enhanced control and security over your infrastructure, but what about the actual databases working with and storing the data? Should you have to sacrifice database orchestration solutions that ensure operational continuity? No.

With such an implementation, you can choose solutions that decouple the orchestration layer from the database, enabling you to assert control over how and where your databases operate through greater access and portability. This ensures that you can satisfy regulations while enjoying the benefits of core services.

One of our solutions, ClusterControl, provides the efficiency, reliability and scale benefits associated with traditional DBaaS for your database operations without sacrificing workload access or locking you into a particular environment. The end result is that you have sovereignty over your infrastructure and database layers.

Wrapping up

Sovereign Cloud solutions offer a strong option for organizations focused on data sovereignty and regulatory compliance. However, it’s important to recognize its limitations. Organizations need to conduct thorough due diligence and strategic planning to ensure that Sovereign Cloud solutions fit their business posture.

By understanding these challenges and finding ways to address them, organizations can make informed decisions while the ecosystem evolves to mitigate them; for instance, Sovereign DBaaS implementation concept and solutions. 

The important point to remember is that whatever the categorical label sovereignty is about control, which is a spectrum; and, you will fall on it depending on what your current and future business requirements demand. For instance, you don’t have to choose a certified “this or that” tool or service. There are many tools, services and concepts that you can implement to increase your sovereignty. 

Nonetheless, a well-planned data stack strategy that leans more toward operational sovereignty / control is a non-negotiable for your organization to ensure agility, stay compliant and retain customer trust and loyalty in today’s world.

Follow us on LinkedIn and Twitter for more great content in the coming weeks. Stay tuned!

The post The Sovereign Cloud brief appeared first on Severalnines.

In the early days of Elasticsearch, security features were less robust than those of the latest versions. For instance, versions prior to version 6.3 did not include the open-source X-Pack extension. As of version 8.0, X-Pack has been deprecated, with its features now integrated into the Elasticsearch API.

With Elasticsearch 8.0 and above, security features are enabled by default. This evolution reflects modern data protection approaches, providing sophisticated security solutions to meet large organizations’ demands.

Security becomes crucial when handling confidential and vital data within your Elasticsearch dataset. Data breaches often stem from neglect and a poor understanding of security requirements, making robust security essential to your technology strategy. Implementing security involves costs in CPU, memory, hardware, and expertise. However, with decreasing hardware costs, Elasticsearch’s default security features highlight the necessity of strong data protection.

1. Install Elasticsearch with security enabled

When you first install Elasticsearch, security is automatically configured. This includes encryption for communication between nodes and clients. The system generates private and public certificates and a strong, randomly generated password. Below is an excerpt from the installation log of a successful Elasticsearch installation on Ubuntu Linux:

……..

The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 184 not upgraded.
Need to get 596 MB of archives.
After this operation, 1234 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.7.0 [596 MB]
Fetched 596 MB in 13s (46.7 MB/s)
Selecting previously unselected package elasticsearch.
(Reading database ... 72764 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.7.0_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.7.0) ...
Setting up elasticsearch (8.7.0) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : JeZrDh=R5vQlIfH_49d1

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
……..

It installs the Elasticsearch package and sets up security autoconfiguration for the following:

• Sets up TLS for transport and HTTP layers for communication with clients.
• Enables TLS/SSL on the transport layer for node-to-node communication.
• Sets token enrollment for authentication, such as in Kibana.
• Generates a strong password for the superuser.

If you want to change any of these autoconfiguration values, you can check your /etc/elasticsearch/elasticsearch.yml configuration file.

Elasticsearch uses X-Pack for encrypted communication. X-Pack is an Elastic Stack extension that provides security, alerting, monitoring, reporting, machine learning, and many other capabilities.  

As of Elasticsearch version 8.0, X-Pack is bundled and integrated by default, ensuring that security features are enabled out-of-the-box unless explicitly disabled. 

X-Pack is free with Elasticsearch, and starting with version 8.0, security features are inherently part of the Elasticsearch ecosystem. You can confirm if security is enabled during a fresh installation by checking the configuration:

# grep 'xpack.security.enabled' /etc/elasticsearch/elasticsearch.yml
xpack.security.enabled: true

If xpack.security.enabled is set to true, security is active.

The X-Pack extension also handles the transport and communication encryption security layer in Elasticsearch, and this is verified in the configuration file as shown below:

# grep 'xpack.security.' /etc/elasticsearch/elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

Once you set xpack.security.enabled to false, you disable the security features of your Elasticsearch installation.

2. Make sure you restrict administrative access

During the autoconfiguration process or installation of Elasticsearch on Linux, a system user named elasticsearch is created. To prevent mistakes and exposure to data breaches, ensure that this user is solely responsible for any Elasticsearch files and processes.

During installation, a superuser or built-in user named elastic, as shown previously with a generated password, is created. This user has administrator privileges for the overall management of your Elasticsearch. It’s crucial to keep this password secure and share it only with limited users who have the capability and expertise to protect and manage your Elasticsearch nodes. To safely store your passwords, you can use tools such as HashiCorp Vault, 1Password, KeePass, NordPass, or similar technologies.

3. Set strong passwords

If you already have strong passwords and prefer to use them instead of the randomly generated ones, you can optionally reset the password of your choice. Elasticsearch provides a utility tool to reset passwords. The binaries and utilities are located in /usr/share/elasticsearch/bin/. Resetting the password is straightforward, as shown in the example below:

# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i
This tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:
Re-enter password for [elastic]:
Password for the [elastic] user successfully reset.

4. Limit access to your Elasticsearch nodes

You can reset the password using Elasticsearch’s utility tools. However, consider the potential risks if a server is breached and taken over by an intruder or hacker, leading to data being stolen, exploited, or held for ransom.

To prevent such scenarios, ensure that access to your Elasticsearch nodes is restricted to dedicated personnel. Only DevOps engineers or Elasticsearch DBAs should have access to detailed information such as passwords and filesystem permissions. The directory and OS user should be accessible only to individuals with admin or superuser privileges.

Additionally, files containing credentials, such as usernames and passwords, should not be globally readable or exposed. For example, the Kibana configuration file (/etc/kibana/kibana.yml) should set its filesystem permissions so that only the owner and appropriate group can read it. This minimizes the risk of unauthorized access.

5. Using enrollment tokens to interact with Elasticsearch nodes

One great feature of Elasticsearch is the ability to use tokens for authentication when joining and communicating with an existing Elasticsearch cluster with enabled security features. The Elastic Stack security features authenticate users using realms and one or more token-based authentication services. These services manage and authenticate tokens. 

You can attach these tokens to requests sent to Elasticsearch and use them as credentials. When Elasticsearch receives a request that requires authentication, it consults the token-based authentication services first and then the realm chain.

For an Elasticsearch cluster, using enrollment tokens with a fresh install of Elasticsearch where autoconfiguration is enabled employs this token-based approach. This method of node authentication for communication is unique compared to other security measures offered by popular mainstream databases.

Moreover, token-based authentication can also be used when authenticating users, such as when Kibana communicates with Elasticsearch. You can use tokens instead of specifying username/password credentials in /etc/kibana/kibana.yml. 

However, based on my experience, this is a more advanced approach to security, and I recommend getting familiar with it. It’s fascinating how you can incorporate tokens to authenticate and allow Elasticsearch to manage these tokens through their internal API, ensuring secure login and communication attempts.

For detailed instructions on enrolling nodes in Elasticsearch or Kibana, refer to the official documentation.

6. Enable password protection

In the Elastic Stack, security features include built-in user credentials to help you get up and running. It is highly recommended that you familiarize yourself with these built-in users. For example, Elasticsearch includes the elastic user, while Kibana has the kibana_system user.

These users should be password-protected. Unlike the elastic user, which is automatically configured during installation, you may need to set or reset passwords for other users. If you need to reset the password for the elastic user, refer to our earlier discussion on this process.

Passwords should be updated regularly. It is best practice to change passwords at least every three months. Additionally, if personnel who have access to your Elasticsearch nodes and underlying servers change, make sure to update the passwords promptly to prevent any malicious intent.

7. User Authentication with LDAP or Active Directory

Many organizations utilize their existing authentication systems, such as LDAP or Active Directory, to authenticate users. The Elastic Stack features support communication with LDAP and Active Directory, allowing you to seamlessly integrate your security authentication with your Elasticsearch cluster.

Although this might seem complex initially, once it is set up, it ensures robust security and safeguards. Additionally, this integration means you don’t need to manage Elasticsearch security in a separate realm, streamlining your overall security infrastructure.

8. Limit the users by role

In Elasticsearch, built-in roles are provided that you can explicitly assign to users. By default, Elastic Stack security features roles for all users, including anonymous users. The default role allows users to access the authentication endpoint, change their own password, and obtain information about themselves.

Elasticsearch offers a Role-Based Access Control (RBAC) mechanism, which enables you to authorize users by assigning privileges to roles and then assigning those roles to users or groups. Using roles, you can assign security privileges limiting users’ functions when accessing Elasticsearch or the Elastic Stack itself. This ensures that users have only the necessary permissions for their tasks, enhancing overall security.

9. Keep it private

Keeping your Elasticsearch isolated from internet access significantly enhances security and safeguards against data breaches or intrusions with malicious intent. Ideally, security implementations for databases involve accessing them through a bastion host and a series of jump hosts. Others use VPNs or secure tunneling to access their servers, providing additional protection and complexity against intruders.

If exposing your Elasticsearch ports to the world is necessary, ensure you:

1. Use a port other than the default 9200.
2. Assign roles that limit user permissions and avoid anonymous access.
3. Implement Role-Based Access Control (RBAC) to effectively manage and restrict user access.

These measures will help maintain a secure Elasticsearch environment even when external access is required.

10. Enable audit logging

Although this security feature is not available to Elasticsearch’s free license users, enabling audit logging if you plan to subscribe to a license can significantly enhance your ability to trace and track audit trails, preventing potential damage or harm to your data. 

You can log security-related events such as authentication failures and refused connections to monitor your cluster for suspicious activity, including data access authorization and user security configuration changes.Audit logging also provides forensic evidence in the event of an attack. By default, this feature is disabled, but enabling it is straightforward. To enable audit logging, set the following parameter to true in your /etc/elasticsearch/elasticsearch.yml configuration file. Set the following parameter as below:

xpack.security.audit.enabled: true

Then restart Elasticsearch…

systemctl restart elasticsearch.service

11. Always backup your data

In case of data breaches or ransom attacks, it is crucial to always have backup data for safekeeping. Elasticsearch provides the capability to create backups through snapshots. This method is the most reliable and highly recommended by Elasticsearch for backing up and restoring your data when necessary.

Snapshots allow you to capture the state of your cluster at a specific point in time, including indices and their settings. To ensure data safety, regularly schedule snapshots and store them in a secure location. This practice will help you quickly recover from any data loss incidents.

12. Disabling security in Elasticsearch

While disabling security is not a best practice, there might be specific cases where you might consider it. However, this should be approached with caution, especially if you are dealing with confidential or sensitive data. 

Security should always be a priority and should be implemented with appropriate safeguards. That said, there are scenarios where data can be regenerated and does not pose a risk if publicly accessed. In such cases, prioritizing speed and simplicity over security might be acceptable, particularly for proof-of-concept demonstrations.

For example, you might temporarily disable security if you need to quickly showcase a proof-of-concept scenario to demonstrate how data is processed and the potential outcomes.

To disable security in Elasticsearch, you can deactivate the X-Pack security plugin by modifying your elasticsearch.yml configuration file and setting the following:

xpack.security.enabled: false

13. Allow anonymous access

Another aspect to consider is that Elasticsearch also handles anonymous incoming requests, meaning no authentication token can be extracted from the incoming request. By default, anonymous requests are rejected and returned with a status code 401.

To enable anonymous access, you assign one or more roles to anonymous users in the elasticsearch.yml configuration file. For example, the following configuration assigns anonymous users the roles role1 and role2:

xpack.security.authc:
  anonymous:
    username: anonymous_user 
    roles: role1, role2 
    authz_exception: true 

What do the following configuration settings for anonymous users in Elasticsearch mean:

• principal: The username/principal of the anonymous user. Defaults to _es_anonymous_user if not specified.
• roles: The roles to associate with the anonymous user. If no roles are specified, anonymous access is disabled, meaning anonymous requests will be rejected and return an authentication error.

• authz_exception: When set to true, a 403 HTTP status code is returned if the anonymous user does not have the necessary permissions to perform the requested action, and the user will not be prompted to provide credentials to access the requested resource. When set to false, a 401 HTTP status code is returned if the anonymous user does not have the necessary permissions, and the user is prompted for credentials to access the requested resource. If you are using anonymous access in combination with HTTP, you should set authz_exception to false if your client does not support pre-emptive basic authentication – defaults to true.

Using Clustercontrol with Elasticsearch automatic security

Clustercontrol deploys Elasticsearch with security enabled by default, making it the standard for any Elasticsearch cluster deployments. The difference between manually installing Elasticsearch with security autoconfiguration and using Clustercontrol is that the latter provides transport and client security in place, resulting in hassle-free and error-resistant configurations.

Certificates, including private and public certs, are generated, and the client certificate is stored in /var/lib/cmon/ca/elastic/cluster_CLUSTER_ID/https/kibana/elasticsearch-ca.pem. You can use this certificate to securely communicate from the client to any Elasticsearch node in the cluster.

Additionally, repository and backup configurations are automatically set up for you, eliminating the need for manual configuration. Once an Elasticsearch cluster is deployed fresh, you can create and schedule backups, which will be stored in the designated repository path specified during deployment.

Here’s an example of the designated repository path for reference:

Once your deployed cluster is set up, you can manage backups by navigating to the Backups tab. Here, you can perform various backup-related actions. Alternatively, you can create a backup and store it in any S3-compatible destination.

For example, to set up an S3-compatible backup destination, you might configure it as follows:

Wrapping up

Elasticsearch security is an ever-evolving, state-of-the-art implementation. This means it continually improves and adapts to meet the most demanding data protection and security requirements. You can explore subscription options under the Elastic Stack security group for enhanced security features.

When it comes to hardening the security of your hardware and environment, firewalls and safeguards are crucial, especially if you manage large amounts of confidential data. Without these protections, your data could be vulnerable to ransom attacks or leaks for profit, which you want to avoid.

Security should always be a top priority. Learning the basics of Elasticsearch security will provide essential protection against catastrophic outcomes. Investing in robust security measures and staying updated with the latest security practices will help ensure the safety and integrity of your data.

Follow us on LinkedIn and Twitter for more great content in the coming weeks. Stay tuned!

The post Best practices for Elasticsearch security appeared first on Severalnines.

Large Language Models (LLMs) have shown incredible potential, but their reliance on static pre-trained knowledge has been a major limitation. To address this, Retrieval Augmented Generation (RAG) was developed as a solution to enhance LLMs by allowing them to access and incorporate relevant information from external sources in real-time.

However, relying solely on traditional retrieval methods, which often prioritize semantic matching, can be limiting when dealing with the nuances of language and meaning. This is where hybrid search comes in, offering a solution to improve the accuracy and comprehensiveness of information retrieval within RAG. 

This blog will explore creating a RAG application that uses hybrid search to address the limitations of Semantic search-based RAG. We will walk you through building a chatbot assistant for an e-commerce website that uses hybrid search to provide accurate and informative responses to user queries about store items. 

The role of semantic search in RAG 

Semantic search plays a crucial role in the retrieval phase of RAG. Unlike traditional keyword-based search methods, semantic search aims to understand the context and intent behind a query, allowing for more nuanced and relevant information retrieval. It leverages advanced natural language processing techniques and vector representations of text to identify conceptually similar content. 

This means that semantic search can:

• Understand synonyms and related concepts
• Interpret the context of words in a query
• Match queries with relevant information even when the exact keywords are not present

For instance, a semantic search for “affordable lightweight laptop” might return results about “budget-friendly ultrabooks” even if the exact terms of the search weren’t used in the product descriptions. Semantic Search plays an important role in the retrieval process and is at the heart of the RAG system.

Limitations of using semantic search for RAG

While Semantic Search has its advantages, it comes with its limitations, in situations where exact terminology is required like e-commerce; when a user searches for product names or technical specifications, a semantic search-based engine would simply try to find the most conceptually similar matches.

For queries requiring exact phrase matches or specific factual information, semantic search is unlikely to produce the necessary level of precision.

Consider a scenario where a customer searches for a specific product model, say “iPhone 13 Pro Max 256GB Sierra Blue“. A purely semantic search might return results for various high-end smartphones or even other iPhone models, missing the exact match the customer needs.

Hybrid search as a solution

To address these limitations, we turn to hybrid search, a powerful combination of keyword-based search and semantic search methodologies. Hybrid search aims to leverage the strengths of both approaches. Keyword-based search produces precision and exact matches, and semantic search provides an understanding of context and intent.

In the context of RAG, hybrid search enhances retrieval by ensuring that an LLM is provided with factually precise (keyword search) and conceptually relevant information (Semantic Search) for a particular query. By combining the contextual understanding of semantic search with the precision of keyword matching, we can create a more robust and accurate retrieval system for our RAG-powered chatbot.

Building an e-commerce chatbot with hybrid search for improved RAG

The e-commerce domain presents unique challenges for AI-powered customer support systems. An effective chatbot must handle a wide variety of queries, from general product information to specific technical details. 

Let’s consider some of the challenges an e-commerce chatbot might face:

• Diverse product catalog: The store may offer thousands of products across multiple categories, each with its attributes and specifications.
• Technical specificity: Many products, especially electronics, have precise technical specifications that customers may inquire about.
• Natural language variations: Customers might ask about the same product or feature using different phrases or terms.
• Intent understanding: The chatbot needs to discern whether a customer is looking for general information, comparing products, or seeking a specific item.

In this section, we’ll dive into the technical implementation of our hybrid search system, showing how to combine PostgreSQL’s pgvector for semantic search and Elasticsearch for keyword search to create a powerful, flexible retrieval mechanism for an e-commerce chatbot.

The core components of the RAG system

The following are the components that make up the RAG system:

• Knowledge Base — The knowledge base for this application is an e-commerce dataset of Amazon products. 

• Large Language Model — Gemini 1.5 Flash by Google. This will be the AI responsible for the generation of text and reasoning in the application. 

• LangChain — LangChain is an LLM tooling and orchestration library that provides tools for connecting LLMs to other tools (databases, interpreters, etc.), building advanced prompt templates, and orchestrating LLM-based applications and utilities.

• Embedding Model — the  Google Text Embedding/004  embedding model converts texts into embeddings that can be indexed and searched in a vector database. 

• Semantic search with PostgreSQL pgvector — helps the retrieval system find contexts that are conceptually and semantically relevant. 

• Keyword Search Index with Elasticsearch — helps the retrieval system perform keyword-based search to find exact keyword matches. 

• ClusterControl —  In this demo, we use ClusterControl (CC) to set up the PostgreSQL pgvector database and Elasticsearch. ClusterControl is a database orchestration platform that lets anyone deploy and manage the world’s most advanced databases.

N.B. CC is not necessary, feel free to set up a standard PostgreSQL and Elasticsearch database using your preferred method. 

These components together provide the mechanism for retrieving relevant information to a user’s query about store items which is then integrated with the language model’s input to generate accurate and factually grounded outputs.

Demo walkthrough

The following demo was implemented in Jupyter Notebook, an interactive coding environment that lets you combine code visualizations, and explanatory text in a single document.

You can test the demo on Google Colab’s free Jupyter Notebook service and follow the demo by running each cell in the notebook. The cells in the notebook are to be run sequentially and go hand-in-hand with the walkthrough below.

If you like to follow along with the blog, below are the prerequisites.

Prerequisites: 

• Google AI Studio Developer API Key
• Python (3.8 and above) 
•  A Jupyter Notebook 
  

Install the necessary Python libraries: 

%pip install --upgrade --quiet langchain-elasticsearch elasticsearch
%pip install --upgrade --quiet langchain langchain-postgres langchain-google-genai langchain-community

Deploying PostgreSQL pgvector and Elasticsearch with ClusterControl

With ClusterControl, anyone can deploy high-availability databases in minutes in any environment. To deploy databases with ClusterControl, you self-host it in a Linux node  with the following specs:

•  x86_64 Arch
•  >2 GB RAM
•  >2 cores CPU
•  >40 GB disk space 
• SSH connectivity (so CC can connect to your target database nodes)

Download and install ClusterControl using one of the available methods (script, Docker, Helm, etc). After installing ClusterControl, click “Deploy a cluster” to get a deployment wizard with several database flavors to choose from. Follow the deployment wizard to deploy PostgreSQL with pgvector and Elasticsearch as highlighted in the image below.

Connecting to the PostgreSQL pgvector database

The Langchain Postgres wrapper is the easiest way to connect to a PostgreSQL pgvector database in the context of our application. Since Langchain provides a lot of tools that ease connecting all of our components to the LLMs, It is only natural to use Langchain’s Postgres PGVector class. 

from langchain_postgres.vectorstores import PGVector

To connect to your vector database you just set up, you only need your connection string and the collection name (also called the server group that hosts your database). These credentials can be found in your PostgreSQL database.

Your connection string should be in the following format:

postgresql://username:password@host:port/database_name

The LangChain PGVector class gives you complete control over your database and lets you index and query the database directly from the Python environment.

Connecting to the PostgreSQL pgvector database

LangChain also provides an Elasticsearch integration for connecting to and creating Elasticsearch indexes. To connect an Elasticsearch index, the following information is required:

• The IP address and port (defaults to 9200) of your Elasticsearch server.
• ElasticSearch username (defaults to ‘elastic’)
• ElasticSearch Password

We will use these credentials later on to connect to the database.

Step 1: Loading the dataset 

Before connecting to the databases, you need to prepare the data for indexing into both databases. In the e-commerce dataset, each row of the dataset describes a product with columns providing information such as the product’s name, the ‘product-id’, and descriptions of the product. The Pandas python library is used to convert the CSV format to a Pandas DataFrame object suitable for performing simple data analysis and having a better objective view of the dataset. 

The Pandas library comes bundled with any Jupyter Notebook distribution by default and doesn’t have to be installed.

import pandas as pd
data = pd.read_csv('e-commerce.csv')

We can view some rows of the dataset by running:

data.head(3)

The dataset has the following columns:

(['item_id', 'marketplace', 'country', 'main_image_id', 'domain_name',
       'bullet_point', 'item_keywords', 'material', 'brand', 'color',
       'item_name', 'model_name', 'model_number', 'product_type'],
      dtype='object')

Each row of the dataset contains information about the item-id, country, description e.t.c about a product in the store.

Before indexing, you need to create a LangChain document object from the dataset:

from langchain_community.document_loaders.csv_loader import CSVLoader
loader = CSVLoader('e-commerce.csv')
docs = loader.load()

The CSVLoader class is used to convert the CSV dataset to a document object which is the smallest unit of information LangChain is built on. Now the entire dataset has been transformed into a document object suitable for working with the rest of the langchain environment.

Step 2: Chunking

Next, the document is split into smaller chunks using the RecursiveCharacterTextSplitter library to split text along each column based on character count, allowing for overlap between chunks to preserve context. The length_function parameter specifies a function the text splitter uses for calculating the length of chunks, in this example we use Python’s default ‘len()’ function.

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
)
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)# Split documents
split_docs = text_splitter.split_documents(documents)

Step 3: Indexing the PGVector database

Before sending embeddings to the PGVector database, some considerations have to be made. Firstly, we had to choose the column of data to perform a semantic search against. Since the bullet_point column contains a brief description of each product, it makes more sense to use this column to generate embeddings that will be stored in the vector database. This implies that our semantic search retrieval finds products whose descriptions are most similar in meaning to a user’s query. 

For example, if a user asks about “ultra-fast cheap notebooks”, our semantic search system will find products whose descriptions are similar to that query. To do this, you need to create a new document object that stores the content of the bullet_point column (contains the description of a product) and the other columns such as item-name, country, etc. as metadata. The following code block does exactly that:

# code for indexing
documents = [
    Document(
        page_content=row['bullet_point'],
        metadata={
            'source': 'e-commerce.csv',
            'row': idx,
            'item_id': row['item_id'],
            'item_name': row['item_name'],
            'keywords': row['item_keywords'],
            'country': row['country'],
            'product_type': row['product_type'],
            'marketplace': row['marketplace'],
            'country': row['country'],
            'bullet_point': row['bullet_point'],
            'domain_name': row['domain_name'],
        }
    ) for idx, row in data.iterrows() if pd.notna(row['bullet_point'])
]

Next, we wrote a function for iterating through the docs and upserting them to the vector database in batches using the PGVector class mentioned earlier.

def insert_documents_in_batches(docs, batch_size=100):
    for i in tqdm(range(0, len(docs), batch_size), desc="Inserting batches"):
        batch = docs[i:i+batch_size]
        try:
            PGVector.from_documents(
                batch,
                embeddings,
                collection_name="E-commerce",
                connection=conn_string
            )
        except Exception as e:
            print(f"Error inserting batch {i//batch_size + 1}: {str(e)}")
            continue

The PGVector.from_documents() function takes embeddings as an argument. This function is the embedding model that creates the vector representation of documents stored in the vector database for semantic search. This application uses the Google embedding/001 model as specified earlier.

embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001",google_api_key=GOOGLE_API_KEY)

Now you can index the vector database by calling the insert_document_in_batches function on the documents we chunked earlier in split_docs.

insert_documents_in_batches(split_docs)

Step 4: Creating the ElasticSearch index 

Using the elasticsearch library provided by Elasticsearch and the LangChain Elasticsearch integrations, you can populate your search index with your data. First, import the necessary libraries:

from elasticsearch import Elasticsearch
from langchain_elasticsearch import ElasticsearchRetriever
from langchain_elasticsearch import ElasticsearchStore

The next step involves connecting to the Elasticsearch database index you created earlier with your credentials:

es_client = Elasticsearch(
    [f"http://es_host:es_port"],
    http_auth=(es_user, es_password)
)
es_client.info()

The es_host variable holds the IP address of the server the Elasticsearch instance is hosted on. The es_port variable holds the port Elasticsearch runs on. By default, Elasticsearch always runs on port 9200 on any machine.

Your credentials should look like this: 

es_host = "13.42.34.146" #could be "localhost"
es_port = "9200"
es_user = 'elastic'
es_password = "" #your elasticsearch cluster password

After successfully creating an Elasticsearch instance, you should have the following running at your localhost:9200:

As we did for the PostgreSQL pgvector database, the Elasticsearch index is created by iterating through the dataset (this time the Pandas dataset object called data) and upserting its content to the Elasticsearch Index.

for _, row in data.iterrows():
    doc = {
        "item_id": row['item_id'],
        "marketplace": row['marketplace'],
        "country": row['country'],
        "main_image_id": row['main_image_id'],
        "domain_name": row['domain_name'],
        "bullet_point": row['bullet_point']
    }
    es_client.index(index=ES_INDEX, body=doc)

You can confirm the process was successful by running:

es_client.indices.refresh(index=ES_INDEX)

The above code will return:

ObjectApiResponse({'_shards': {'total': 2, 'successful': 1, 'failed': 0}})

Now, you can use the Langchain ElastcsearchRetriever.from_es_params class to perform a keyword search on your index.

Define a function that specifies the schema the search index uses: 

def query_function(search_query: str):
    return {
        "query": {
            "multi_match": {
                "query": search_query,
                "fields": ["item_name", "item_id", "bullet_point"],
            }
        }
    }

The query_function searches through the item_name,item_id, and bullet_point columns of the dataset. This function is passed as an argument to the ElasticsearchRetriver.from_es_params class provided by LangChain. This class connects Langchain to your index.

keyword_retriever = ElasticsearchRetriever.from_es_params(
    url=f"https://{es_host}:{es_port}",
    username = es_user,
    password = es_password,
    index_name=ES_INDEX,
    body_func=query_function,
    # content_field={
    #     "item_name": "Name",
    #     "item_id": "ID",
    #     "bullet_point": "Description"
    # },
    document_mapper=custom_document_mapper,
)

The keyword_retriever will be used by your application for connecting and searching through the Elasticsearch index created by the es_client on your server.

You can test out a simple keyword search through the database by using the invoke() method of the keyword_retriever you created and searching for a product in the database.

query = "Micromax Yu Yureka Black"
results = keyword_retriever.invoke(query)
for doc in results:
    print(doc.page_content)
    print(doc.metadata)
    print("---")

Which returns all products in the dataset containing the input query “Micromax Yu Yureka Black”.

Step 5. Instantiating the LLM

As mentioned earlier, the Large Language Model used in this demo is the Google Gemini 1.5 Flash (the perfect balance between performance and efficiency among the models provided by Google).

llm = ChatGoogleGenerativeAI(
    model="gemini-1.5-flash",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # other params...
)

The final part of the application involves providing the Large Language model with information from the Hybrid Search Retrieval system to answer user queries during conversations.

The LLM must be prompted to assume the role of a Shopping Assistant. This is done by creating a system prompt which is just a message sent to the large language model describing its role in the conversation to customers. 

system_prompt = """
    You are an intelligent e-commerce shop assistant for a large online retailer. Your role is to provide helpful, accurate, and relevant information to customers about products in our inventory. Use the following guidelines:
    1. Always base your responses on the product information provided in the context. Do not invent or assume details that are not explicitly stated.
    2. If asked about a specific product, provide details such as the product name, ID, and key features from the bullet points.
    3. When comparing products or suggesting alternatives, only reference products mentioned in the given context.
    4. If the user asks about pricing, availability, or shipping, provide the information if it's available in the context. If it's not available, politely inform the user that you'd need to check the latest information.
    5. Be concise but informative. Organize information in a clear, easy-to-read format when appropriate.
    6. If the user's query cannot be fully answered with the information provided, acknowledge this and offer to help find more details or suggest contacting customer service.
    7. Maintain a friendly, professional tone. Be helpful and courteous, but remember that you are an AI assistant, not a human.
    8. If asked about personal opinions on products, politely explain that as an AI assistant, you don't have personal preferences, but you can provide factual information to help the customer make an informed decision.
    9. If the user's query is unclear or could have multiple interpretations, ask for clarification to ensure you provide the most relevant information.
    Remember, your primary goal is to assist customers in finding the right products and information, enhancing their shopping experience with our e-commerce platform.
    """

The system prompt is sent to the LLM as the first message in a format created by LangChain’s SystemMessage function:

messages = [
    SystemMessage(content=system_prompt),
    HumanMessage(content="Hi,"),
    AIMessage(content="hello")
]

Subsequent responses from the LLM are stored as AIMessage objects and Human messages or user queries are stored as HumanMessage.

The final part of the RAG application is a function that takes a user query, performs a hybrid search, and passes the relevant context together with the user’s query to the LLM as a single message. 

def hybrid_search(query: str, k: int = 5):
    # Perform keyword search
    db = PGVector(
    connection=conn_string,
    embeddings=embeddings,
    collection_name=collection_name
    )
    keyword_results = keyword_retriever.invoke(query)
    # Perform semantic search
    semantic_results = db.similarity_search(query)
    # Combine and rank results
    combined_results = []
    seen_ids = set()
    # Add keyword results first
    for doc in keyword_results:
        if doc.metadata['item_id'] not in seen_ids:
            combined_results.append({
                'item_name': doc.metadata['item_name'],
                'item_id': doc.metadata['item_id'],
                'description': doc.page_content.split('Description: ')[-1],
                'score': doc.metadata['score'],
                'source': 'keyword'
            })
            seen_ids.add(doc.metadata['item_id'])
    # Add semantic results
    for doc in semantic_results:
        if doc.metadata['item_id'] not in seen_ids:
            combined_results.append({
                'item_name': doc.metadata['item_name'],
                'item_id': doc.metadata['item_id'],
                'description': doc.page_content,
                'score': doc.metadata.get('score', 0),  # Semantic search might not have a score
                'source': 'semantic'
            })
            seen_ids.add(doc.metadata['item_id'])
    # Sort results by score in descending order
    combined_results.sort(key=lambda x: x['score'], reverse=True)
    # Return top k results
    return combined_results[:k]

The function above performs semantic and keyword searches on the Elasticsearch index and PostgreSQL pgvector database. 

The augment_prompt function below adds the retrieved information to the user query sent to the Large Language Model.

def augment_prompt(query):
    retrieved_context = hybrid_search(query)
    source_knowledge = "\n".join([f"Description: {doc['description']}\nItem Name: {doc['item_name']}\nItem ID: {doc['item_id']}\nScore: {doc['score']}\nSource: {doc['source']}" for doc in retrieved_context])
    
    augmented_prompt = f"""Query: {query}
    Retrieved Context:
    {source_knowledge}
    Based on the above context, please provide a detailed and accurate response to the query."""
    return augmented_prompt

When a user enters a query, the query is first passed to the augment_query function that adds relevant  context  before it is added to the rest of the conversation; 

user_query = "Find me a hard back case for a Lenovo K4 Note"
messages.append(HumanMessage(augment_prompt(user_query)))

User workflow

The components of the RAG system are now in place and here’s a simple workflow for how the application works : 

• A user enters a query about a product.
• The query is converted to an embedding vector used for semantic search against the vectors in the PostgreSQL pgvector database where the most semantically and contextually similar documents to the user query are returned.
• The user query is also used for keyword searches against the ElasticSearch Index to find keyword matches in cases where specific product names are mentioned.
• The returned documents from the Semantic Search and Keyword Search are fed to the large language model with the user’s query to provide factually grounded context for answering questions about a product.
  

An example scenario

In the following conversation, a user asks the assistant a question about the availability and price of a specific product.  The user’s query:

"Find me a hard back case for a Lenovo K4 Note"

In the example above if using just semantic search-based RAG, the LLM won’t provide the relevant context to correctly answer the user’s request. The hybrid search retrieval system provided the LLM with the right context to provide the best answer to the user’s query given the information in the database. As can be seen in the LLM’s output, the user requests an item that isn’t available in the store’s database. 

Going through the database,  you can see that there are only soft-back cases available and not the hard-back cases that the user requested.

Another example scenario 

With the Hybrid search system, users can also make descriptive queries if they don’t know exactly what they want since the semantic search component of the retrieval system will still provide the necessary context. The LLM can now provide accurate answers to questions like: 

 "are in-ear headphones available"

LLM’s response:

Scaling RAG systems with open source software

The implementation of our hybrid search RAG system marks a significant leap forward in the capabilities of an e-commerce chatbot, substantially improving the shopping experience for users. This advanced approach combines the strengths of both keyword-based and semantic search methodologies, resulting in a more versatile and user-friendly interface.

Open-source tools remain highly relevant in the AI landscape, particularly in the realm of customization. Unlike proprietary tools, open source tool enable organizations to innovate in their use cases contributing to the advancement of AI.

While our demonstration focuses on a simplified implementation, it’s important to note that as the system scales, the underlying infrastructure (models, databases, etc) needs scale too. This is where tools like ClusterControl become invaluable, streamlining the setup, maintenance and scaling of open-source databases like PostgreSQL  pgvector and Elasticsearch, ensuring optimal performance and reliability as RAG systems grows to handle larger datasets and increased user traffic. 

Try out ClusterControl with a free 30-day trial.

The post Beyond Semantics: Enhancing Retrieval Augmented Generation with Hybrid Search (pgvector + Elasticsearch) appeared first on Severalnines.

After a one-year hiatus, DevOpsDays returned to Houston with great success this year. Our Tech Evangelist, Divine Odazie, attended and presented a demo titled “Kubernetes at Scale: Going Multi-Cluster with Istio Service Mesh,” along with co-speaker Jubril Oyetunji.

“DevOpsDays Houston 2024 was my first-ever DevOpsDays conference, and I’m certain it won’t be my last. The event was packed with outstanding talks, fascinating people doing remarkable work across various industries, and even some incredible food!” says Divine.

In this blog, Divine provides a play-by-play of his experience – from mind-blowing talks to eureka moments – he’s got it all covered!

Key takeaways

DevOps, the Universe and Everything by Andrew Clay Shafer

This year’s DevOpsDays Houston kicked off with a keynote by none other than Andrew Clay Shafer, one of the founding members of DevOpsDays. 

Listening to Andrew dive into the realms of DevOps and software development was nothing short of incredible.

A standout point Andrew made was, “Focusing on the technical without accounting for the socio never goes very well.” 

He emphasized the importance of socio-technical alignment across the organization. Any disconnect, whether from top to bottom or across teams, can stifle innovation and efficiency.

Give his talk a watch. Below is a snapshot from Andrew’s slides, highlighting the challenges of driving change within an organization:

Cloud native empowering infrastructure scalability

Several talks delved into Kubernetes and its ability to enable scaling. 

In my session, I explored the multi-cluster strategy for deploying applications on Kubernetes and demonstrated how the Istio service mesh can facilitate service deployment across multiple Kubernetes clusters. 

Below is an architecture diagram from my slides, showcasing the final result of the demo: 

Another captivating talk on Kubernetes was by Danesh Manoharan on how PGS (Petroleum Geo-Services) Unlocked Supercomputing Power with GKE. In this talk, he discussed how PGS, as an HPC, was able to move to the Cloud with Kubernetes. 

Not only were they able to access more computing power, but they’ve also become more agile by not being tied to aging hardware. It’s also a more eco-friendly solution.

Shift-left Security to defend against growing threats

As always, security should never be an afterthought. Several insightful security talks highlighted the need for development teams to find common ground with IT operations and information security departments. 

This collaboration is essential to reliably build and deliver secure applications, whether cloud-native or not.

From my understanding and experience, the first step is to identify and analyze potential threats to your application and infrastructure. For ongoing operations (Day 2), it’s crucial to remain agile and continuously check for vulnerabilities using scanning tools.

To dive deep into defending against threats, check out the following talks from the conference:

• Security Lifecycle for Cloud Native Applications by Mark Hahn
• Guarding the Digital Realm: Defending Against Software Supply Chain Threats and AI-Induced Deception by Sean Casey
• Which Vault? Don’t tell me your secret(s)! by Michel Schildmeijer
• Security is an Awesome Product Feature in 5 minutes by Mark Hahn

Observability is more important than ever

Observability has always been a key part of building software, but it’s even more crucial in the modern cloud-native world. With applications splintered into microservices and scattered across various environments, keeping tabs on them and troubleshooting issues has become a high-stakes game of digital hide-and-seek.

The conference featured several important talks on observability, including one on “Network Observability.” Often overlooked and underappreciated, network observability is more important than ever. In this talk, Lean Adato unveiled the secrets hidden in our networks, suggesting ways to access this valuable information and highlighting the insights we miss if this data remains invisible, unused, or ignored.

Is observability worth it? Every new request coming into your infrastructure may not necessarily bring additional revenue to your company, but will most definitely cost money to observe as most vendors today charge by data ingested. 

In this talk on ”Measuring Observability ROI”, Ruchir Jha covered the best practices and presented a broader instrumentation philosophy based on open standards like OpenTelemetry that is targeted at ROI measurement.

DevOps in achieving data sovereignty

Data sovereignty, the ability to control and govern your own data, hinges on owning your data stack – the infrastructure, databases, and surrounding tools. DevOps practices are instrumental in achieving and maintaining this sovereignty.

At Severalnines, for over a decade we have enabled hundreds of companies to take control of their data through our products, i.e. ClusterControl and CCX. 

Check out our Sovereign DBaaS Decoded podcast where we interview industry leaders about implementing open-source databases, creating cloud vendor-neutral environments, and mitigating business risks, including data sovereignty and compliance requirements.

You can also join our “Sovereign DBaaS Community.” Whether you’re a seasoned database operations pro or a novice looking to connect, learn, and grow – join now! 

Follow us on LinkedIn and Twitter for more great content in the coming weeks. Stay tuned!

The post DevOpsDays Houston 2024: Our conference highlights appeared first on Severalnines.