Using the Connect gateway

This page tells you how to use the Connect gateway to connect to a registered cluster. Before reading this guide, you should be familiar with the concepts in our overview. The guide assumes that your project administrator has already set up the gateway, and given you the necessary roles and permissions.

Before you begin

  • Ensure that you have the following command line tools installed:

    If you are using Cloud Shell as your shell environment for interacting with Google Cloud, these tools are installed for you.

  • Ensure that you have initialized the gcloud CLI for use with your project.

Log in to your Google Cloud account

You can use your own Google Cloud account or a Google Cloud service account to interact with connected clusters via the gateway API.

Follow the instructions in Authorizing Google Cloud CLI tools to log in to your user account. The Connect gateway supports service account impersonation, so even if you're logged in to your own user account you can use a service account to interact with clusters, as you'll see in the following sections.

Select a registered cluster

If you don't know the name of the cluster you want to access, you can see all your current fleet's registered clusters by running the following command:

gcloud container fleet memberships list

This lists all your fleet's clusters, including their membership names and external IDs. Each cluster in a fleet has a unique membership name. For GKE clusters, the membership name generally matches the name you gave it when you created the cluster, unless the cluster's name was not unique within its project at registration.

Get the cluster's gateway kubeconfig

Use the following command to get the kubeconfig you need to interact with your specified cluster:

gcloud container fleet memberships get-credentials MEMBERSHIP_NAME

Replace MEMBERSHIP_NAME with your cluster's fleet membership name.

This command returns a special Connect gateway-specific kubeconfig that lets you connect to the cluster through the Connect gateway.

If you want to use a service account rather than your own Google Cloud account, use gcloud config to set auth/impersonate_service_account to the service account email address.

To fetch the cluster credential used to interact with the Connect gateway using a service account, run the following commands: Note the following:

  • Google Distributed Cloud (software only) clusters on bare metal and VMware: The membership name is the same as the cluster name.

  • GKE on AWS: Use gcloud container aws clusters get-credentials.

  • GKE on Azure: Use gcloud container azure clusters get-credentials.

If you want to use a service account rather than your own Google Cloud account, use gcloud config to set auth/impersonate_service_account to the service account email address. You can find out more about letting users impersonate a service account in Manage access to service accounts.

gcloud config set auth/impersonate_service_account SA_EMAIL_ADDRESS
gcloud container fleet memberships get-credentials MEMBERSHIP_NAME

Replace SA_EMAIL_ADDRESS with the email address of the service account. You can find out more about letting users impersonate a service account in Manage access to service accounts.

Run commands against the cluster

Once you have the necessary credentials, you can run commands using kubectl or a go-client as you normally would for any Kubernetes cluster. Your output should look something like the following:

# Get namespaces in the Cluster.
kubectl get namespaces
NAME              STATUS   AGE
default           Active   59d
gke-connect       Active   4d

Limitations

The following kubectl commands are not supported by the gateway using the gcloud container fleet memberships get-credentials command:

  • attach
  • cp
  • exec
  • port-forward

Preview support for the commands

The attach, cp, and exec commands (but not port-forward) are supported with the beta command gcloud beta container fleet memberships get-credentials. This command lets you use a preview feature of the Connect gateway.

Note the following requirements:

  • GKE on Google Cloud isn't supported in the preview.

  • Clusters must be at version 1.30 or later.

  • The kubectl client must be at version 1.29 or later. If you use version 1.29, you must set the following environment variable:

    export KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true

    kubectl versions 1.30 and later don't require the environment variable.

    To check your client version, look at the output of the kubectl version command. To install a newer version of kubectl see Install tools.

Users and service accounts with the roles/gkehub.gatewayAdmin IAM role and the cluster-admin ClusterRole have the necessary permissions to run the attach, cp, and exec commands. If users and service accounts have been granted a custom IAM role or custom RBAC role, you might need to grant additional permissions. See the following sections for additional information.

Grant an additional permission if needed

The gkehub.gateway.stream IAM permission is required to run the attach, cp, and exec commands through the Connect gateway. This permission is included in roles/gkehub.gatewayAdmin.

For users who aren't project owners, or for users or service accounts who haven't been granted roles/gkehub.gatewayAdmin in the project, you must either grant them roles/gkehub.gatewayAdmin or create a custom role that includes the other required roles and the gkehub.gateway.stream permission. For information on how to create a custom role, see Create and manage custom roles in the IAM documentation.

Create and apply additional RBAC policies if needed

Users and service accounts with the cluster-admin ClusterRole have the necessary permissions to run the attach, cp, and exec commands.

At a minimum, the following rules are needed to run the commands:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: stream-role
  namespace: NAMESPACE  # Specify the namespace
rules:
- apiGroups: ["*"]
  resources: ["pods/exec", "pods/attach"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: stream-rolebinding
  namespace: NAMESPACE  # Specify the namespace
roleRef:
   apiGroup: "rbac.authorization.k8s.io"
   kind: Role
   name: stream-role
subjects:
- kind: User
  name: EMAIL # Specify the user that should have stream access
  namespace: NAMESPACE  # Specify the namespace

kubectl exec/cp/attach Troubleshooting

The error returned from running the command is often not clear enough to debug the issue. In this case, rerun the command with a higher verbosity logging level, for example: kubectl exec -v 5 ....

Connect gateway beta track is not being used

If you get a 400 Bad Request error, regenerate the connect gateway kubeconfig file using the gcloud CLI beta track to see if that fixes the error:

gcloud beta container fleet memberships get-credentials my-membership`

If you still get the 400 Bad Request error, follow the steps in the next section.

kubectl environment flag is not set

If the kubectl client version that you are using is 1.29, the environment variable KUBECTL_REMOTE_COMMAND_WEBSOCKETS needs to be enabled. If this isn't enabled, you get a 400 Bad Request error. To see if the environment variable is enabled, run the following command:

 echo $KUBECTL_REMOTE_COMMAND_WEBSOCKETS`

If the variable isn't set, enable it by setting the following environment variable:

export KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true

kubectl version 1.30 and later doesn't require this flag as it is enabled by default.

kubectl version lower than 1.29 won't work with this feature.

Missing IAM permissions

If you get the error: error reading from error stream... message, this could indicate that you haven't been granted the required IAM permissions to run the command. This feature requires users to have the gkehub.gateway.stream IAM permission, which is included by default in the roles/gkehub.gatewayAdmin role. See the IAM permission section for instructions.

Error message generic::failed_precondition: error encountered within the cluster

If you get the error generic::failed_precondition: error encountered within the cluster, check the Connect Agent logs in the cluster to identify the underlying cause:

kubectl logs -n gke-connect -l app=gke-connect-agent --tail -1

The log to look for in the Connect Agent is failed to create the websocket connection....

Missing required RBAC permissions

If the error message in the Connect Agent logs contains 403 Forbidden, this indicates your are missing RBAC permissions. You have a set of RBAC permissions in the cluster to run these kubectl commands. See the RBAC policies section for setting up the required RBAC permissions.

Error message generic::resource_exhausted: Gateway's active_streams quota exhausted

There is a quota limit of 10 active streams per fleet host project. This is defined under the connectgateway.googleapis.com/active_streams quota. See View and manage quotas for instructions on managing your quotas.

Error message generic::failed_precondition: connection to Agent failed/terminated

If you encounter this error immediately when running the command, there is an issue with the cluster's connection to Google. See the general troubleshooting guide for more information.

If you encounter this error after about 5 to 10 minutes of the session being active, this is an expected limitation of the feature. The connection needs to be re-established.

Troubleshooting

If you're having problems connecting to a cluster through the gateway, you or your administrator can check for the following common issues.

  • The server doesn't have a resource type: You may see this error message when the command kubectl get ns fails. There are multiple potential reasons for this error. Run your kubectl commands in verbose mode to see more details, for example kubectl get ns -v 10.
  • Cannot find active connections for cluster(project: 12345, membership: my-cluster): You may see this error when Connect Agent loses connectivity or isn't installed properly (clusters outside Google Cloud only). To resolve this issue, you need to verify if gke-connect namespace exists on the cluster. If the gke-connect namespace exists on the cluster, see the Connect Troubleshooting page to fix the connectivity issues.
  • The requested URL was not found on this server: You may see this error when the kubeconfig contains an incorrect server address. Make sure the Google Cloud CLI version you're using is the latest version and retry to generate the gateway kubeconfig. Don't manually edit the kubeconfig file, which will cause unexpected errors.
  • The user identity does not have sufficient permissions to use the gateway API: You need the roles/gkehub.gatewayAdmin roles/gkehub.gatewayReader or roles/gkehub.gatewayEditor role to use the API. See Grant IAM roles to users in the gateway setup guide for more details.
  • The Connect agent is not authorized to send the user’s requests: The Connect agent needs to be allowed to forward requests on your behalf, which is specified using an impersonation policy on the cluster. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the gateway-impersonate role.
  • The user identity does not have sufficient RBAC permissions to perform the operation: You must have appropriate permissions on the cluster to run your chosen operations. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the appropriate ClusterRole.
  • The user identity does not have sufficient permissions to perform the operation when using Google Groups or third-party support: See Collecting GKE Identity Service logs for instructions on how to inspect logs related to identity information.
  • The Connect agent is unhealthy: See the Connect Troubleshooting page to make sure your cluster is connected.
  • executable gke-gcloud-auth-plugin not found or no Auth Provider found for name gcp: kubectl versions 1.26 and later may display this error due to changes to kubectl authentication starting with GKE v1.26. Install gke-gcloud-auth-plugin and re-run gcloud container fleet memberships get-credentials MEMBERSHIP_NAME with the latest version of Google Cloud CLI.
  • Connections to the gateway fail with older versions of the Google Cloud CLI: For GKE clusters, the Connect agent is no longer required for the gateway to function so it does not get installed by default during membership registration. Earlier versions of the Google Cloud CLI (399.0.0 and below) assume the existence of the Connect agent on the cluster. Attempting to use the gateway with these earlier versions may fail on clusters registered with a newer version of the Google Cloud CLI. To resolve this, either upgrade your Google Cloud CLI client to a newer version or re-run the membership registration command with the --install-connect-agent flag.

What's next?