Hazelcast Workspaces on VMs

◀️ Hazelcast WAN Example 🔗 Hazelcast-on-AWS-EC2 ▶️

Hazelcast/Jet Workspaces on VMs

PadoGrid provides a simple way to manage Hazelcast/Jet clusters on laptop, PadoGrid pods, and VMs. This article describes how to create and run a Hazelcast/Jet cluster on multiple VMs and/or PMs (Physical Machines). The instructions provided apply to any VMs and PMs including AWS, Azure, GCP, physical machines, VirtualBox, vSphere, PadoGrid pods (VMs), etc.

With PadoGrid, you can have a complete control over Hazelcast/Jet running on VMs from your local machine. PadoGrid preserves the state of VM configurations and deployments on your local machine so that you can conveniently activate or deactivate the VM environment at any time. After you are done running VMs, you can tear them down and reinstate the very same environment later with a single command. This saves you time and reduces VM costs when you are working in a cloud environment, for example.

Installation Steps

There are four (4) main steps to installing PadoGrid on VMs. Once these steps are performed, you can then synchronize your local machine with the remote VMs by simply executing the vm_sync command, which enables the local control of VMs.

Password-less SSH Login

The first step is to check to make sure you are able to login to the VMs that you want to cluster without the password. For AWS EC2, for example, this is already done for you so you can skip this section.

If you need to manually setup password-less SSH login, then follow the instructions here.

❗ Make sure you can ssh from any of the VMs including your local machine without the password.

Download Software

PadoGrid requires the following software products. If they are not installed on VMs, then download their tarball or zip distributions.

For our demo, we'll assume they are downloaded in the following directory. Note that they must be tarball or zip distributions.

/home/dpark/Downloads
├── jdk1.8.0_212.tar.gz
└── hazelcast-enterprise-4.0.1.tar.gz

Create VM Workspace

Create a VM workspace on your local machine, e.g., your laptop.

create_workspace -name ws-hz -vm

The above command interactively prompts for software installation paths information. The -vm option enables VMs and -name option names the workspace. The following shows my laptop environment. The environment variables that begin with VM_ are specific to the remote VMs. All other environment variables without the prefix VM_ pertain to your laptop environment. For example, VM_PADOGRID_HOME is the PadoGrid installation path in the VM hosts you specified.

Please answer the prompts that appear below. If you are not able to complete
the prompts at this time then use the '-quiet' option to bypass the prompts.
You can complete the requested values later in the generated 'setenv.sh' file
You can abort this command at any time by entering 'Ctrl-C'.

Enter Java home path.
[/home/dpark/Padogrid/products/jdk1.8.0_212]:

Enter the local product home directory path. Choose one
from the defaults listed below or enter another.
   /home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
   /home/dpark/Work/linux/hazelcast-jet-enterprise-3.2.2
   /home/dpark/Padogrid/products/pivotal-gemfire-9.10.0
[]:
/home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
Enter workspace name.
[ws-hz]:

Enter default cluster name.
[myhz]:

Enable VM? Enter 'true' or 'false' [true]:
Enter VM JDK home path.
[/home/dpark/Padogrid/products/jdk1.8.0_212]:
/home/dpark/Hazelcast/jdk1.8.0_212
Enter VM product home path.
[/home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1]:
/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
Enter VM padogrid home path.
[/home/dpark/Padogrid/products/padogrid_0.9.7]:
/home/dpark/Hazelcast/padogrid_0.9.7
Enter VM workspaces path.
[/home/dpark/Padogrid/workspaces/myrwe]:
/home/dpark/Hazelcast/workspaces/myrwe
Enter VM host names or IP addresses separated by comma.
[]:
ubuntu1,ubuntu2,ubuntu3,ubuntu4
Enter VM user name.
[]: dpark
Enter VM SSH private key path. If you don't have the key file (.pem) then
leave it blank for now. You can place the file in the workspace directory or
set the path in the workspace 'setenv.sh' file later.
[]:


You have entered the following.
                     JAVA_HOME: /home/dpark/Padogrid/products/jdk1.8.0_212
                  PRODUCT_HOME: /home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1
            PADOGRID_WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz
               Default Cluster: myhz
                    VM_ENABLED: true
                  VM_JAVA_HOME: /home/dpark/Hazelcast/jdk1.8.0_212
               VM_PRODUCT_HOME: /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
              VM_PADOGRID_HOME: /home/dpark/Hazelcast/padogrid_0.9.7
   VM_PADOGRID_WORKSPACES_HOME: /home/dpark/Hazelcast/workspaces/myrwe
         VM_PADOGRID_WORKSPACE: /home/dpark/Hazelcast/workspaces/myrwe/ws-hz
                      VM_HOSTS: ubuntu1,ubuntu2,ubuntu3,ubuntu4
                       VM_USER: dpark
           VM_PRIVATE_KEY_FILE:
Enter 'c' to continue, 'r' to re-enter, 'q' to quit: c
----------------------------------------------------------------
            WORKSPACE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz
      Cluster Created: myhz
         CLUSTER_TYPE: imdg
           Deployment: VM
          CLUSTER_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz
           CONFIG_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc
          CONFIG_FILE: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/hazelcast.xml
              LOG_DIR: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/log
    MEMBER_START_PORT: 5701
     DEBUG_START_PORT: 9101
       JMX_START_PORT: 12001
PROMETHEUS_START_PORT: 8091
         MC_HTTP_PORT: 8080
        MC_HTTPS_PORT: 8443
          MC_JMX_PORT: 9001
      MC_JMX_RMI_PORT: 9051

The default hazelcast.xml file has been placed in the CONFIG_DIR directory.
Modify it or replace it with the appropriate file.

One (1) member has been configured (added) to run in this cluster as follows:

      Member: myhz-padogrid-01
 Working Dir: /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/run/myhz-padogrid-01

1. Make sure to replace or update hazelcast.xml:
      /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/hazelcast.xml
2. Place your application jar files in one of the 'plugins' directories:
      /home/dpark/Padogrid/workspaces/myrwe/ws-hz/plugins/
      /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/plugins/
3. Configure cluster properties in:
      /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz/etc/cluster.properties
4. You must configure password-less ssh login to all VMs. On AWS, this
   is already configured but you will need to enter the correct private key
   file path in the cluster.properties file.
5. To change VM host names, list the VM host names in the cluster.properties file
   for management center and members.
      mc.host=host1
      vm.hosts=host1,host2,...
6. Deploy the cluster configuration changes you made to all the VM hosts listed in the
   previous step by running 'vm_copy' or 'vm_sync'.
      vm_copy /home/dpark/Padogrid/workspaces/myrwe/ws-hz/clusters/myhz
7. To start the cluster:
      start_cluster -cluster myhz
8. To verify the cluster state:
      show_cluster -cluster myhz
9. To view log file:
      show_log -cluster myhz
----------------------------------------------------------------

Workspace created:
   /home/dpark/Padogrid/workspaces/myrwe/ws-hz

The workspace directory also contains the following files.

   initenv.sh -  Workspace initialization file. Source this file in to
                 enter the workspace.

   setenv.sh -  You can set your workspace-wide environment variables in
                this file.

   .addonenv.sh - A hidden file containing the workspace environment variables.
                  You can overwrite them in setenv.sh as needed.

IMPORTANT:
----------
To use this workspace, execute the 'switch_workspace' command as follows:
   switch_workspace ws-hz

You have created a VM workspace. To add or remove VM hosts, set VM_HOSTS in
the workspace 'setenv.sh' file. Note that VM_HOSTS applies to the entire workspace
whereas the 'vm.hosts' property in the cluster 'etc/cluster.properties' file applies
only to the defining cluster. By default, 'vm.hosts' is initially set to VM_HOSTS.
   /home/dpark/Padogrid/workspaces/myrwe/ws-hz/setenv.sh

The above example shows that the installation paths in the VMs are different from the local installation paths. We choose to install all the software components in the /home/dpark/Hazelcast for the VMs.

If you specify the private key file, i.e., ecs.pem in our example, then it is automatically copied to the newly created workspace directory. It is also later deployed to the VMs when you execute the vm_sync command.

You can also run the create_workspace in the non-interactive mode by specifying the -quiet option. The following command produces the same workspace configurations as the above interactive example.

create_workspace -quiet \
-name ws-hz \
-cluster myhz \
-java /home/dpark/Padogrid/products/jdk1.8.0_212 \
-product /home/dpark/Padogrid/products/hazelcast-enterprise-4.0.1 \
-vm ubuntu1,ubuntu2,ubuntu3,ubuntu4 \
-vm-java /home/dpark/Hazelcast/jdk1.8.0_212 \
-vm-product /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1 \
-vm-padogrid /home/dpark/Hazelcast/padogrid_0.9.7 \
-vm-workspaces /home/dpark/Hazelcast/workspaces/myrwe \
-vm-user dpark

SSH Private Key

If a private key file is required and you have not specified it in the previous example, then you can place it in the workspace directory. PadoGrid automatically picks up the first .pem file found in the workspace directory if the VM_PRIVATE_KEY_FILE environment variable is not set in the $PADOGRID_WORKSPACE/setenv.sh file. The following shows the contents of the ws-hz workspace directory we created.

# If a private key file (.pem) is required then place it in in the
# ws-hz workspace directory.
switch_workspace ws-hz
tree -L 1 .
.
├── apps
├── bundles
├── clusters
├── docker
├── initenv.sh
├── k8s
├── lib
├── plugins
├── pods
└── setenv.sh

Test VM Workspace

Before you sync the local workspace with VMs - which may initially take some time to complete - if you need to install software products, we should first test the environment to make sure SSH works properly. This is done by executing vm_test.

vm_test

vm_test outputs the following:

------------------------------------------------------------------------------------------
Workspace: ws-hz

Environment:
                  VM_ENABLED=true
                    VM_HOSTS=ubuntu1,ubuntu2,ubuntu3,ubuntu4
                     VM_USER=dpark
         VM_PRIVATE_KEY_FILE=
                VM_JAVA_HOME=/home/dpark/Hazelcast/jdk1.8.0_212
           VM_HAZELCAST_HOME=/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
     VM_PADOGRID_HOME=/home/dpark/Hazelcast/padogrid_0.9.7
VM_PADOGRID_WORKSPACE=/home/dpark/Hazelcast/workspaces/myrwe/ws-hz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Network Test:
   ubuntu1 - OK
   ubuntu2 - OK
   ubuntu3 - OK
   ubuntu4 - OK
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
WARNING:
/home/dpark/Hazelcast/jdk1.8.0_212
   Java is not installed on the following VMs. The workspace will not be operational
   until you install Java on these VMs.
       ubuntu1 ubuntu2 ubuntu3 ubuntu4

   To install Java on the above VMs, download the correct version of Java and execute 'vm_install'.

   Example:
      vm_install -java jdk1.8.0_212.tar.gz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
WARNING:
/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
   Hazelcast is not installed on the following VMs. The workspace will not be operational
   until you install Hazelcast on these VMs.
       ubuntu1 ubuntu2 ubuntu3 ubuntu4

   To install Hazelcast on the above VMs, download the correct version of Hazelcast and
   execute 'vm_install'.

   Example:
      vm_install -product hazelcast-enterprise-4.0.1.tar.gz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
/home/dpark/Hazelcast/padogrid_0.9.7
   padogrid is not installed on the following VMs. It will be automatically installed
   when you execute 'vm_sync'.
       ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Summary:
   One or more VM issues found. Please correct them before executing 'vm_sync'.

Workspace Issues:
   None.
Network Issues:
   None.
Software Issues:
   Java missing from the following VMs. Install Java with 'vm_install -java'.
      ubuntu1 ubuntu2 ubuntu3 ubuntu4
   Hazelcast missing from the following VMs. Install Hazelcast with 'vm_install -product'.
      ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------

vm_test scans all the VMs and reports any issues it encounters. For example, the abovevm_test report shows that SSH sessions are working fine but Java and Hazelcast are not installed on all of the VMs. To install Java and Hazelcast on the VMs, we execute the vm_install command as follows.

vm_install -java /home/dpark/Downloads/jdk-8u212-linux-x64.tar.gz \
           -product /home/dpark/Downloads/hazelcast-enterprise-4.0.1.tar.gz

vm_install outputs the following:

Java installation
    Local Source: /home/dpark/Downloads/jdk-8u212-linux-x64.tar.gz
   Remote Target: /home/dpark/Hazelcast/jdk1.8.0_212
   Installing ubuntu1...
   Installing ubuntu2...
   Installing ubuntu3...
   Installing ubuntu4...
Java installation complete.

Hazelcast installation
    Local Source: /home/dpark/Downloads/hazelcast-enterprise-4.0.1.tar.gz
   Remote Target: /home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
   Installing ubuntu1...
   Installing ubuntu2...
   Installing ubuntu3...
   Installing ubuntu4...
Hazelcast installation complete.

Run 'vm_test' to check installation and configuration status.

If you run vm_test after installing Java and Hazelcast, you should see no issues in the VM workspace.

vm_test

Output:

------------------------------------------------------------------------------------------
Workspace: ws-hz

Environment:
                  VM_ENABLED=true
                    VM_HOSTS=ubuntu1,ubuntu2,ubuntu3,ubuntu4
                     VM_USER=dpark
         VM_PRIVATE_KEY_FILE=
                VM_JAVA_HOME=/home/dpark/Hazelcast/jdk1.8.0_212
           VM_HAZELCAST_HOME=/home/dpark/Hazelcast/hazelcast-enterprise-4.0.1
     VM_PADOGRID_HOME=/home/dpark/Hazelcast/padogrid_0.9.7
VM_PADOGRID_WORKSPACE=/home/dpark/Hazelcast/workspaces/myrwe/ws-hz
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Network Test:
   ubuntu1 - OK
   ubuntu2 - OK
   ubuntu3 - OK
   ubuntu4 - OK
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
/home/dpark/Hazelcast/padogrid_0.9.7
   padogrid is not installed on the following VMs. It will be automatically installed
   when you execute 'vm_sync'.
       ubuntu1 ubuntu2 ubuntu3 ubuntu4
------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
Summary:
   No issues found. To sync the VM workspace, execute 'vm_sync'.

Workspace Issues:
   None.
Network Issues:
   None.
Software Issues:
   None.
------------------------------------------------------------------------------------------

Hazelcast License Keys

For our example, we have installed Hazelcast Enterprise which requires a license key. The license keys are normally placed in the RWE .hazelcastenv.sh file ($PADOGRID_WORKSPACES_HOME/.hazelcastenv.sh), which is synchronized when you execute the vm_sync command (see next section). If you need to use license keys other than the ones set in this file then you can set them in the workspace's setenv.sh file, which overrides the environment variables set in the .hazelcastenv.sh file.

# Change directory to the current workspace, 'ws-hz', i.e., $PADOGRID_WORKSPACE
cd_workspace
vi setenv.sh

Add the license keys in setenv.sh to override .hazelcastenv.sh:

# Hazelcast IMDG license
IMDG_LICENSE_KEY=***
#Jet license
JET_LICENSE_KEY=***
# Management Center
MC_LICENSE_KEY=***

You can now proceed to sync the VM workspace.

Synchronize VM Workspace

In the previous section, we have locally created and tested a VM workspace. Before we can use it on the remote VMs, we need to synchronize it with the VMs. To do so, we execute the vm_sync command, which synchronizes the workspace you just created and automatically installs PadoGrid on all the VMs if it is not already installed.

vm_sync

vm_sync outputs the following:

Deploying padogrid_0.9.7 to ubuntu1...
Deploying padogrid_0.9.7 to ubuntu2...
Deploying padogrid_0.9.7 to ubuntu3...
Deploying padogrid_0.9.7 to ubuntu4...

Workspace sync: ws-hz
   Synchronizing ubuntu1...
   Synchronizing ubuntu2...
   Synchronizing ubuntu3...
   Synchronizing ubuntu4...
Workspace sync complete.

Configure Hazelcast Cluster

In our example, we have created the default cluster named, myhz. Let's switch into the myhz cluster.

switch_cluster myhz

The cluster directory has the following files.

myhz/
├── bin_sh
│   └── setenv.sh
├── etc
│   ├── cache.xml
│   ├── cluster.properties
│   ├── gemfire.properties
│   ├── hibernate.cfg-mysql.xml
│   ├── hibernate.cfg-postgresql.xml
│   ├── log4j2.properties
│   └── prometheus.yml
├── lib
├── log
├── plugins
└── run

`etc/cluster.properties`

In the myhz directory, you will find etc/cluster.properties which defines cluster-level properties. Let's edit this file to set the Management Center host (mc.host) to ubuntu4 and the cluster VMs (vm.hosts) to ubuntu1,ubuntu2,ubuntu3.

# Browse the cluster properties and change them as needed. 
# Pay attention to the following two (2) properties:
#   mc.host=
#   vm.privateKeyFile=
vi etc/cluster.properties

# Set Management Center host
mc.host=ubuntu4

# Set the private key file path. If this property is set then it overwrites the
# VM_PRIVATE_KEY_FILE  environment variable set in the workspace's setenv.sh file. 
#vm.privateKeyFile=~/Hazelcast/Workspaces/ecs.pem

# Include additional VM host names. Host names (or IP addresses) must be
# comma separated with no spaces. Spaces are not supported.
vm.hosts=ubuntu1,ubuntu2,ubuntu3

`etc/hazelcast.xml`

You can also update the hazelcast.xml file at this time as needed.

vi etc/hazelcast.xml

Sync Changes

Any changes you made can be easily deployed to all the VMs specified by the VM_HOSTS environment variable in the $PADOGRID_WORKSPACE/setenv.sh by running the vm_sync command, which copies the entire workspace directory to the VMs. Be aware that this can take some time to complete if you have many VMs and large binary files in the workspace.

After the VMs have been synchronized, if you make changes to only a few files, then instead of executing vm_sync again, you can execute vm_copy which copies only the specified file or directory to the VMs.

switch_cluster myhz

# To copy a single file
vm_copy etc/cluster.properties

# To copy the entire directory
vm_copy .

`vm_sync` vs `vm_copy`

The main difference between vm_sync and vm_copy is that vm_sync copies the entire workspace and vm_copy copies only the specified file or directory. Both commands allow you to apply the changes you made to all of the VMs. As you might have guessed, vm_sync will take longer to complete since it copies the entire workspace, but it is more convenient to use than vm_copy since you don't have to remember the files you modified. These commands will quickly become your favorite commands if you often reconfigure Hazelcast clusters or update application library files.

❗ Note that vm_copy by default only copies the files that are in the workspace directory hierarchy. If you try to copy a file that is outside of the workspace directory then it will fail and output an error message. If you need to copy non-workspace files, specify the -mirror option, which copies the absolute file path to the same absolute file path in the remote VMs.

Start Cluster

You are now ready to start the myhz cluster.

# Start cluster
start_cluster

# Start management center
start_mc

Monitor Cluster

To monitor the cluster:

show_cluster
show_mc

View Log

To view logs

# ubuntu1
show_log

# ubuntu2
show_log -num 2

Stop/Kill Cluster

To stop or kill the cluster:

stop_cluster
stop_mc
kill_cluster

Running from VMs

You can also run VM workspaces from any of the VMs. As with the local environment, you must first setup the PadoGrid environment on the VM in which you want to run workspaces by sourcing in the workspaces initenv.sh file.

# SSH into one of the VMs
ssh ubuntu1

# Source in intienv.sh
. ~/Hazelcast/workspaces/myrwe/initenv.sh

# Optionally add the above line in .bashrc (or .bash_profile) so that PadoGrid is
# automatically initialized when you login next time. Make sure to include the '-quiet' option.
echo ". ~/Hazelcast/workspaces/myrwe/initenv.sh -quiet" >> ~/.bash_rc

# Once you have initenv.sh sourced in you can then execute any of the PadoGrid
# commands as before. The following starts the `myhz` cluster.
switch_workspace ws-hz
swtich_cluster myhz
start_cluster

Teardown

If you want to remove the cluster from all the VMs, then you must first stop the cluster and execute the remove_cluster command. The workspace can be removed using the remove_workspace command.

# Stop cluster & management center
stop_cluster
stop_mc

# Remove cluster. Unlike other commands, this command requires the '-cluster' option.
remove_cluster -cluster myhz

# Simulate removing workspace from all VMs. Displays removal steps but does not
# actually remove the workspace.
remove_workspace -workspace ws-hz -simulate

# Remove workspace from all VMs. Runs in interactive mode.
remove_workspace -workspace ws-hz

References

◀️ Hazelcast WAN Example 🔗 Hazelcast-on-AWS-EC2 ▶️

PadoGrid Manual

Overview

Operations

Tools

Platforms

Clouds

PadoGrid on OCI Compute

Pado

Geode/GemFire

Hazelcast/Jet

ComputeDB/SnappyData

Coherence

Hadoop

Hadoop CLASSPATH

Kafka/Confluent

Mosquitto

Redis

Spark

Spark CLASSPATH

Explore

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hazelcast Workspaces on VMs

Hazelcast/Jet Workspaces on VMs

Installation Steps

Password-less SSH Login

Download Software

Create VM Workspace

SSH Private Key

Test VM Workspace

Hazelcast License Keys

Synchronize VM Workspace

Configure Hazelcast Cluster

`etc/cluster.properties`

`etc/hazelcast.xml`

Sync Changes

`vm_sync` vs `vm_copy`

Start Cluster

Monitor Cluster

View Log

Stop/Kill Cluster

Running from VMs

Teardown

References

Clone this wiki locally

Hazelcast Workspaces on VMs

Hazelcast/Jet Workspaces on VMs

Installation Steps

Password-less SSH Login

Download Software

Create VM Workspace

SSH Private Key

Test VM Workspace

Hazelcast License Keys

Synchronize VM Workspace

Configure Hazelcast Cluster

etc/cluster.properties

etc/hazelcast.xml

Sync Changes

vm_sync vs vm_copy

Start Cluster

Monitor Cluster

View Log

Stop/Kill Cluster

Running from VMs

Teardown

References

Clone this wiki locally

`etc/cluster.properties`

`etc/hazelcast.xml`

`vm_sync` vs `vm_copy`