◀️ VM-Enabled Pods 🔗 Multitenancy Best Practices ▶️

Since Padogrid v0.9.20

PadoGrid by design includes support for multi-tenant distributed workspaces, allowing a user or a group of users to own a workspace or a sandbox environment that is completely isolated from other workspaces.

We define the term workspace owner as one or more users that have full permissions (RWX) to a given workspace. Each workspace is independently managed and distributed by the workspace owner. This capability allows each workspace to branch out on their own to allocate system resources based on their needs. Some multi-tenant workspace examples are listed below.

• A developer owns a workspace that requires only local resources such as a small Hazelcast cluster with masked data ingested.
• A data scientist owns a workspace that spans several nodes with data analytics tools such as GemFire, Spark, and JupyterLab.
• An application team owns a workspace that spans several nodes integrating many endpoints such as data sources, middleware, application server, load-balancer, and microservices.
• A support team owns several workspaces for testing new releases, reproducing problems, capturing timelines of events, archiving system images, and etc.
• A production team owns several workspaces, each dedicated for a datacenter.

• A user owns a workspace. The user is sandboxed in the workspace.
• A group of users owns a workspace. All the users in the group are sandboxed in the workspace.
• A user owns one or more RWEs. The user is sandboxed in the RWE. The user can create and remove workspaces in the RWE.
• A group owns one or more RWEs. All the users belonging to the group are sandboxed in the RWE. The users can create and remove workspaces in the RWE.
• A user owns one or more workspaces in one or more RWEs.
• A group owns one or more workspaces in one or more RWEs.

• An RWE visible to all users.
• An RWE visible to only select users.
• An RWE visible to only a single user.
• An RWE visible to only a single group.
• A workspace visible to all users.
• A workspace visible to only select users.
• A workspace visible to only a single user.
• A workspace visible to only a single group.

Managing multi-tenant workspaces is made simple by having the padogrid system account that has access to all RWEs and workspaces. The padogrid account serves as an administrator for creating and removing workspaces, adding and removing users and user groups, allocating workspace resources, etc. The following example shows how to create the padogrid account.

# Create a system account for user 'padogrid'. Set home directory to /home/padogrid
sudo useradd -m -r -U -s /bin/bash -d /home/padogrid padogrid
sudo passwd padogrid

• Non-Red Hat - Add padogrid the sudo group

sudo usermod -aG sudo padogrid

• Red Hat - Add padogrid in the wheel group

sudo usermod -aG wheel padogrid
sudo visudo

Uncomment the following in /etc/sudoers to configure sudo to not ask for a password.

%wheel  ALL=(ALL)       NOPASSWD: ALL

2.1. Create /opt/padogrid directory where PadoGrid will be installed (Note that the PadoGrid installation directory of should be /opt/padogrid to be consistent with PadoGrid Docker containers.)

su - padogrid
sudo mkdir /opt/padogrid
sudo chown padogrid /opt/padogrid
sudo chgrp padogrid /opt/padogrid
chmod 750 /opt/padogrid

2.1. Install PadoGrid in /opt/padogrid, not /home/padogrid/Padogrid

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/padogrid/padogrid/develop/padogrid-deployment/src/main/resources/common/bin_sh/install_padogrid)"

Inputs:

❗️ Enter /opt/padogrid at the prompt.

Enter the PadoGrid environment base directory path. All of the products will be installed in this directory.  [/opt/padogrid/Padogrid]:
/opt/padogrid
...

2.2. If you are using PadoGrid v0.9.23 or less, then update .bashrc (assuming you have created the default myrwe RWE from the previous step). Starting v0.9.24, .bashrc is automatically updated by install_padogrid.

echo ". /opt/padogrid/workspaces/myrwe/initenv.sh -quiet" >> /home/padogrid/.bashrc
. /opt/padogrid/workspaces/myrwe/initenv.sh -quiet

2.3. Set permissions for PadoGrid base directories. The following security restrictions disallow PadoGrid users from downloading and installing products in the /opt/padogrid directory.

cd /opt/padogrid
# Allow only rX for groups
chmod -R u=rwX,g=rX,o-rwx downloads products snapshots workspaces
# Set permissions for the default RWE, `myrwe` to block all other users.
cd workspaces
chmod -R u=rwX,g-rwx,o-rwx myrwe
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myrwe

✏️ The default RWE, myrwe, should be used as a private RWE for padogrid. If you choose to make myrwe available to users, then set file permissions as described in Step 3.3.

❗️ You must follow the steps shown in the example below for creating all new RWEs.

3.1. Login as padogrid

su - padogrid
cd /opt/padogrid

3.2. Create an RWE in which workspaces to be created

create_rwe -rwe rwe-test

3.3. Set file permissions for RWE files. Users are not allowed to modify these files.

# Switch to the newly created RWE
switch_rwe rwe-test
# Set permissions for the RWE directory itself
chmod 750 .
# Set permissions for env files
chmod 640 .*env.sh* *env.sh*
# Set permissions for the default workspace. Allow only owner 'padogrid'.
chmod -R u=rwX,g-rwx,o-rwx myws
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myws

✏️ We set permissions for the default workspace, myws to block all other users so that it can be used as a private workspace for the padogrid user. You can rename it to a more suitable name as needed.

✏️ If you choose to make myws available to a group, then follow the example in Transferring Workspace Ownership.

3.4. Enable group permissions in RWE's setenv.sh

vi setenv.sh

At the bottom of setenv.sh set GROUP_PERMISSIONS_ENABLED to "true":

# Enable group permissions for workspace owners. If "true" then RWX permissions are
# given to the user's secondary group.
GROUP_PERMISSIONS_ENABLED="true"

With the security restrictions established for rwe-test in the previous section, PadoGrid effectively isolates each workspace from others by group. To access a workspace, a user must now belong to the group that owns the workspace.

❗️ You must follow the steps shown in the example below for creating all new workspaces.

4.1. Login as padogrid

su - padogrid

4.2. Create group

# Add 'finance' group
sudo groupadd finance

# Optionally, add 'padogrid' to the secondary group so that it has access to that
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report inaccurate workspace information
# if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG finance padogrid

4.3. Create workspace per primary or secondary group

# Create workspace 'ws-finance'.
create_workspace -name ws-finance -quiet
cd_rwe
sudo chgrp -R finance ws-finance
chmod -R u=rwX,g=rwX,o-rwx,ug+s ws-finance
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 ws-finance

To grant access to a workspace, users must be added in the group that owns the workspace.

❗️ You must follow the steps shown in the example below for adding users to a group.

5.1. Login as padogrid

su - padogrid

5.2 Create user

# Create the primary group to which the users will belong to
sudo groupadd staff

# Add user 'mike' to the primary group, 'staff', and the secondary group, 'padogrid'
sudo useradd -m -g staff -G padogrid -s /bin/bash mike

5.3. Add user to group

# Add 'mike' in the secondary groups 'finance' and 'padogrid'.
# Always assign a secondary group to a workspace. Otherwise, you will
# not be able to revoke workspace privileges from users.
sudo usermod -aG finance,padogrid mike

5.4. Update .bashrc of the user. This needs to be done as root.

# Login as root
su -

echo ". /opt/padogrid/workspaces/rwe-test/initenv.sh -quiet" >> /home/mike/.bashrc

Adding a new group is analogous to adding a group in the OS.

# Add 'staff' group. Used as the primary group in examples
sudo groupadd staff

# Add 'finance' group. Used as a secondary group in examples.
sudo groupadd finance

# Optionally, add 'padogrid' to the seconary group so that it has access to that 
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report inaccurate workspace
# information if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG finance padogrid

Removing a group is analogous to deleting a group in the OS.

sudo groupdel finance

✏️ Note that if there are existing RWEs and workspaces that belong to the removing group, then they will be assigned to a new numerical group ID.

✏️ For this command to take effect, the users belonged to the removed group must re-login.

Adding a new user account is analogous to adding a user in the OS.

# Add the user 'mitch' to the primary group, 'finance' and the secondary group, 'padogrid'
sudo useradd -m -g finance -G padogrid -s /bin/bash mitch

Each user account must be initialized with a default RWE. This is done in the user's .bashrc file as follows.

# Login as root
su -

# Login as root (using sudo)
sudo -i

# Append 'rwe-finance' as the default RWE for the user 'mitch' in .bashrc'
echo ". /opt/padogrid/workspaces/rwe-finance/initenv.sh -quiet" >> /home/mitch/.bashrc
exit

Granting workspace privileges for a user is analogous to adding a user to the workspace group.

# Add the user 'mitch' to the 'finance' and 'padogrid' groups
sudo usermod -aG finance,padogrid mitch

Revoking workspace privileges from a user is analogous to removing a user from the workspace group.

# Remove the user 'mitch' from the 'finance' group
sudo gpasswd -d mitch finance

To revoke privileges to all workspaces, remove the user from the padogrid group.

# Remove the user 'mitch' from the 'padogrid' group
sudo gpasswd -d mitch padogrid

Removing a user account is analogous to removing a user from the OS.

sudo userdel -f -r mitch

# Display all groups that the current user belongs to
groups

# Display all groups that 'mitch' belongs to
groups mitch
id mitch

# Display the primary group that 'mitch' belongs to
id -gn mitch

# Display all users belonging to the 'finance' group
getent group finance

# Display all groups
getent group

All PadoGrid administrative operations must be conducted by the padogrid user.

create_rwe -rwe rwe-test -quiet
switch_rwe rwe-test
# Set permissions for the RWE directory itself
chmod 750 .
# Set permissions for env files
chmod 640 .addonenv.sh .*env.sh initenv.sh setenv.sh
# Set permissions for the default workspace. Allow only owner 'padogrid'.
chmod -R u=rwX,g-rwx,o-rwx myws
setfacl -L -R -d -m u::rwX -m g::000 -m o::000 myws

Enable group permissions in RWE's setenv.sh

vi setenv.sh

At the bottom of setenv.sh set GROUP_PERMISSIONS_ENABLED to "true":

# Enable group permissions for workspace owners. If "true" then RWX permissions are
# given to the user's secondary group.
GROUP_PERMISSIONS_ENABLED="true"

To dedicate the entire RWE to a group, change the group ownership and set permissions. The following example changes the RWE group ownership to testers and sets permissions for creating workspaces in the RWE.

switch_rwe rwe-test
sudo chgrp testers .
# Set permissions for the group to allow creating workspaces in the RWE.
chmod 770 .

# Create workspace 'ws-finance'.
create_workspace -name ws-finance -quiet

# Change directory to 'ws-finance' and apply user and group permissions
cd_workspace ws-finance
sudo chgrp -R finance .
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s .
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 .

The following example transfers the ownership of an existing workspace.

# Change directory to 'ws-finance' and apply user and group permissions
cd_workspace ws-finance
sudo chgrp -R finance .
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s .
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 .

install_bundle -download -workspace bundle-hazelcast-4n5-cluster-session

# Change directory to 'bundle-hazelcast-4n5-cluster-session' and apply user and group permissions
cd_workspace bundle-hazelcast-4n5-cluster-session
sudo chgrp -R finance .
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s .
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 .

cd_rwe
getfacl -R ws-finance

switch_rwe
mv ws-finance ws-finance2

# Backup and remove workspace
remove_workspace -workspace ws-finance

# Remove without backup
cd_rwe
rm -rf ws-finance

switch_rwe rwe-test
cd ..
mv rwe-test rwe-test2

# Remove without backup
cd_rwe rwe-test
cd ..
rm -rf rwe-test

If you have access to the Internet then run install_padogrid; otherwise, download each product and inflate them in the /opt/padogrid/products directory. See Installing PadoGrid for instructions.

install_padogrid

update_product

Product removals may disrupt workspaces. Before you remove products, notify all users and provide enough time for them to migrate to new product releases. Once the products are removed, the workspaces that depend on the removed products will be no longer operational.

First, install a new release and update RWE (and workspaces). The following example upgrades hazelcast-5.1.1 to hazelcast-5.1.3.

install_padogrid -product hazelcast-oss
update_product -product hazelcast-oss

The update_product interactively guides you to update RWE and/or workspaces. Since you are removing the product, update RWE and all workspaces with the new release.

...
Hazelcast OSS
   [0] (none)
   [1] 5.1.3
   [2] 5.1.1
Enter a version to add [0]: 1
Selected: [1] 5.1.3

Apply selected version(s) to RWE?
Enter 'r' for RWE, 'n' for no RWE, 'q' to quit this command: r

Apply selected version(s) to the current workspace or all workspaces?
Enter 'w' for current workspace, 'a' for all workspaces,
      'n' for no workspaces,     'q' to quit this command: a
...
Enter 'c' to continue, 'r' to re-enter, 'q' to quit: c

If you have more than one RWEs, then run update_product on each RWE.

You are now ready to remove the product. All products are installed in the /opt/padogrid/products directory. Find the ones you want to remove from that directory and remove them by running rm.

# Remove hazelcast-5.1.1
cd ~/products
rm -rf hazelcast-5.1.1

Each user would take the following steps to access their workspace.

1. Login to the OS
2. Switch to the workspace
3. Start using the workspace. The workspace is completely isolated. The user can only view the workspaces owend by the groups he/she belongs to and cannot browse other workspaces. This applies to all RWEs.

✏️ The /opt/padogrid/downloads and /opt/padogrid/products directories are accessible by all users. The padogrid administrator can make product distributions and artifacts available in /opt/padogrid/downloads and install any products in /opt/padogrid/products.

Example:

Login via ssh:

ssh mike@padogrid-node

Or using the browser (JupyterLab):

• https://padogrid-node:8888/

Once logged in, from the shell execute PadoGrid commands:

# Display all RWEs and workspaces owned by 'mike'
padogrid

# Display all workspaces in the current RWE owned by 'mike'
show_rwe

# Switch to the 'ws-fiance' workspace in the current RWE owned by 'mike'
switch_workspace ws-finance

# Display current workspace (ws-finance) contents
show_workspace

Normally, workspaces run in a controlled environment accessible by only a select group of system administrators. Each administrator must strictly comply with the security governance established by the company across all the workspaces they are responsible for.

In an uncontrolled environment, the same security governance applies but the users are allowed to manage their own workspaces. This means workspaces would compete for system resources. To prevent this, each workspace should be installed on dedicated machines. This can be achieved by creating a VM-enabled workspace and auto-login users to one of the VMs. VMs can be any remote machines such as AWS EC2.

A workspace can be branched out to a separate mount point by simply dedicating a mount point to the workspace or creating a symbolic link to a mount point path. Similarly, the entire RWE can also be branched out to a separate mount point. By having a workspace on a separate mount point, it limits the use of the host OS resources.

Each workspace is independently configured such that you can assign any number of nodes. The workspace owner can enable VMs in the workspace by simply listing the VM addresses in the configuration file. These VMs could be AWS EC2 VMs or on-prem VMs, for example.

Docker containers can be launched on the host OS or remote machines using the create_docker and install_bundle commands.

If Kubernetes is preferred then the workspace owner creates the k8s components using the create_k8s and install_bundle commands.

The following example shows how to create multiple user accounts, groups, and workspaces. It 1) adds the users, mike, mary, matt, mia, and max in the secondary group finance, 2) creates the ws-finance workspace, and 3) updates each user's .bashrc file with a PadoGrid initialization command.

# Login as padogrid
su - padogrid

# Create primary group to which the users will belong to
PRIMARY_GROUP="staff"
sudo groupadd $PRIMARY_GROUP

# Create user/secondary group. Users must belong to a secondary group.
PADOGRID_USERS="mike mary matt mia max"
GROUP="finance"
sudo groupadd $GROUP
for USER in $PADOGRID_USERS; do \
   sudo useradd -m -g $PRIMARY_GROUP -s /bin/bash $USER; \
   sudo usermod -aG $GROUP,padogrid $USER; \
done

# Optionally, add 'padogrid' to the seconary group so that it has access to that
# group's workspaces. Note that you must re-login for this to take effect.
# (Without this, commands such as 'show_rwe' may report incurate workspace information
# if padogrid does not have read permissions to certain artifacts.)
sudo usermod -aG $GROUP padogrid

# Create workspace. Always assign a secondary group to a workspace. Otherwise, you
# will not be able to revoke workspace privileges from users.
WORKSPACE=ws-$GROUP
create_workspace -name $WORKSPACE -quiet
sudo chgrp -R $GROUP $WORKSPACE
# Set permissions for the workspace. Allow only owner and group.
chmod -R u=rwX,g=rwX,o-rwx,ug+s $WORKSPACE
setfacl -L -R -d -m u::rwX -m g::rwX -m o::000 $WORKSPACE

# Login as root
su -

# Append PadoGrid initialization command in each user's .bashrc
RWE="rwe-test"
PADOGRID_USERS="mike mary matt mia max"
for USER in $PADOGRID_USERS; do \
   echo ". /opt/padogrid/workspaces/$RWE/initenv.sh -quiet" >> /home/$USER/.bashrc; \
done
exit

We can remove all the accounts and the workspace we created in the previous example as follows.

# Login as padogrid
su - padogrid

# Remove user/group
PADOGRID_USERS="mike mary matt mia max"
GROUP="finance"
for USER in $PADOGRID_USERS; do \
   sudo userdel -f -r $USER; \
done
sudo groupdel $GROUP

# Remove workspace
RWE="rwe-test"
switch_rwe $RWE
GROUP="finance"
WORKSPACE=ws-$GROUP
sudo rm -rf $WORKSPACE

◀️ VM-Enabled Pods 🔗 Multitenancy Best Practices ▶️