Receive Pub/Sub events at a private HTTP endpoint in a private GKE cluster

Standard

This tutorial shows you how to create a private HTTP endpoint in a private Google Kubernetes Engine (GKE) cluster that receives Pub/Sub message events using Eventarc. To learn more about this event destination, see Route events to an internal HTTP endpoint in a VPC network.

Private GKE clusters are a type of Virtual Private Cloud (VPC)-native cluster where nodes only have internal IP addresses, which means that nodes and Pods are isolated from the internet by default. You can choose to have no client access, limited access, or unrestricted access to the control plane. You can't convert an existing, non-private cluster to a private cluster. For more information, see About private clusters.

You can run the following commands using the Google Cloud CLI in either your terminal or Cloud Shell.

Objectives

In this tutorial, you will:

Create a proxy-only subnet in the default VPC network, and create a VPC firewall rule.
Create a private GKE Autopilot cluster with no client access to the public endpoint.
Create a Compute Engine virtual machine (VM) instance in a specified subnet of the VPC network.
Establish an SSH connection to the VM instance and deploy an event receiver service on the VM instance.
Deploy a Gateway in your cluster and an HTTPRoute manifest to configure the routing of traffic in Kubernetes to application backends.
Create a network attachment that lets a producer VPC network initiate connections to a consumer VPC network.
Create an Eventarc trigger that routes Pub/Sub events to the event receiver on your VM instance.
Publish a message to a Pub/Sub topic to generate an event, and view the body of the event in the application Pod logs.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Make sure that billing is enabled for your Google Cloud project.

Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:

gcloud services enable compute.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com

Install the Google Cloud CLI.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Make sure that billing is enabled for your Google Cloud project.

Enable the Cloud Resource Manager, Compute Engine, Eventarc, GKE, and Pub/Sub APIs:

gcloud services enable compute.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com eventarc.googleapis.com pubsub.googleapis.com

Update Google Cloud CLI components:
```
gcloud components update
```
Sign in using your account:
```
gcloud auth login
```
If you are the project creator, you are granted the basic Owner role (roles/owner). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.

If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads). For more information, see the Roles and permissions page for your event destination.
Required permissions

To get the permissions that you need to complete this quickstart, ask your administrator to grant you the following IAM roles on your project:
- Compute Network Admin (roles/compute.networkAdmin)
- Compute Security Admin (roles/compute.securityAdmin)
- Eventarc Admin (roles/eventarc.admin)
- Kubernetes Engine Admin (roles/container.admin)
- Logs View Accessor (roles/logging.viewAccessor)
- Project IAM Admin (roles/resourcemanager.projectIamAdmin)
- Pub/Sub Publisher (roles/pubsub.publisher)
- Service Account Admin (roles/iam.serviceAccountAdmin)
- Service Account User (roles/iam.serviceAccountUser)
- Service Usage Admin (roles/serviceusage.serviceUsageAdmin)
For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.
Make note of the Compute Engine default service account as you will you attach it to an Eventarc trigger to represent the identity of the trigger for testing purposes. This service account is automatically created after enabling or using a Google Cloud service that uses Compute Engine, and with the following email format:
```
PROJECT_NUMBER[email protected]
```
Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:
```
gcloud projects describe PROJECT_ID --format='value(projectNumber)'
```
For production environments, we strongly recommend creating a new service account and granting it one or more IAM roles that contain the minimum permissions required and follow the principle of least privilege.

Note:
The iam.automaticIamGrantsForDefaultServiceAccounts organization policy constraint prevents the Editor role from being automatically granted to default service accounts. If you created your organization after May 3, 2024, this constraint is enforced by default.

We strongly recommend that you enforce this constraint to disable the automatic role grant. If you disable the automatic role grant, you must decide which roles to grant to the default service accounts, and then grant these roles yourself.

If the default service account already has the Editor role, we recommend that you replace the Editor role with less permissive roles. To safely modify the service account's roles, use Policy Simulator to see the impact of the change, and then grant and revoke the appropriate roles.
If you enabled the Cloud Pub/Sub service agent on or before April 8, 2021, to support authenticated Pub/Sub push requests, grant the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) to the service agent. Otherwise, this role is granted by default:
```
gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \
    --role=roles/iam.serviceAccountTokenCreator
```

Create a proxy-only subnet

Unless you create an organizational policy that prohibits it, new projects start with a default network (an auto mode VPC network) that has one subnetwork (subnet) in each region. Each VPC network consists of one or more IP address ranges called subnets. Subnets are regional resources, and have IP address ranges associated with them.

Use the gcloud compute networks subnets create command to create a proxy-only subnet in the default network.
```
gcloud compute networks subnets create proxy-only-subnet \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE \
    --region=us-central1 \
    --network=default \
    --range=10.10.10.0/24
```
Note that a subnet with purpose=REGIONAL_MANAGED_PROXY is reserved for Envoy-based load balancers and that the range must provide 64 or more IP addresses.

Create a firewall rule that matches the proxy-only subnet's range and that allows traffic on TCP port 8080.

gcloud compute firewall-rules create allow-proxy-connection \
    --allow tcp:8080 \
    --source-ranges 10.10.10.0/24 \
    --network=default

Create a private GKE cluster

Use the gcloud container clusters create-auto command to create a private GKE cluster in Autopilot mode that has private nodes, and that has no client access to the public endpoint.

The following example creates a private GKE cluster named private-cluster and also creates a subnet named my-subnet:

gcloud container clusters create-auto private-cluster \
    --create-subnetwork name=my-subnet \
    --enable-master-authorized-networks \
    --enable-private-nodes \
    --enable-private-endpoint \
    --region=us-central1

Note the following:

--enable-master-authorized-networks specifies that access to the public endpoint is restricted to IP address ranges that you authorize.
--enable-private-nodes indicates that the cluster's nodes don't have external IP addresses.
--enable-private-endpoint indicates that the cluster is managed using the internal IP address of the control plane API endpoint.

It might take several minutes for the creation of the cluster to complete. Once the cluster is created, the output should indicate that the status of the cluster is RUNNING.

Create a VM instance in a specified subnet

A Compute Engine VM instance is a virtual machine that is hosted on Google's infrastructure. The terms Compute Engine instance, VM instance, and VM are synonymous and are used interchangeably. VM instances include GKE clusters, App Engine flexible environment instances, and other Google Cloud products built on Compute Engine VMs.

Use the gcloud compute instances create command to create a Compute Engine VM instance in the subnet you created previously. Attach a service account and set the VM's access scope to cloud-platform.

gcloud compute instances create my-vm \
    --service-account=PROJECT_NUMBER[email protected] \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --zone=us-central1-a \
    --subnet=my-subnet

For more information, see Create and start a VM instance.

Deploy an event receiver on the VM

Using a prebuilt image, us-docker.pkg.dev/cloudrun/container/hello, deploy a service on your VM that listens on port 80, and that receives and logs events.

Establish an SSH connection to your VM instance by running the following command:
```
gcloud compute ssh my-vm --project=PROJECT_ID --zone=us-central1-a
```
After a connection to the SSH server is established, run the remaining commands on your VM instance.
If necessary, install kubectl and any required plugins.

From your VM instance, use the get-credentials command to enable kubectl to work with the cluster you created.

gcloud container clusters get-credentials private-cluster \
    --region=us-central1 \
    --internal-ip

Use a Kubernetes command, kubectl create deployment, to deploy an application to the cluster.
```
kubectl create deployment hello-app \
    --image=us-docker.pkg.dev/cloudrun/container/hello
```
This creates a Deployment named hello-app. The Deployment's Pod runs the hello container image.

After deploying the application, you can expose your application to traffic by creating a Kubernetes Service. Run the following kubectl expose command:

kubectl expose deployment hello-app \
    --type ClusterIP \
    --port 80 \
    --target-port 8080

You should see service/hello-app exposed in the output.

You can ignore any messages similar to the following:

E0418 14:15:33.970933    1129 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Configure Kubernetes traffic routing

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. For more information, see Deploying Gateways. An HTTPRoute manifest is deployed to create Routes and send traffic to application backends.

Deploy a Gateway in your cluster.
```
kubectl apply -f - <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: internal-http
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
  - name: http
    protocol: HTTP
    port: 80
EOF
```
Note the following:
- gatewayClassName: gke-l7-rilb specifies the GatewayClass that this Gateway is derived from. gke-l7-rilb corresponds to the internal Application Load Balancer.
- port: 80 specifies that the Gateway exposes only port 80 for listening for HTTP traffic.

Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.

kubectl describe gateways.gateway.networking.k8s.io internal-http

The output is similar to the following:

Name:         internal-http
Namespace:    default
...
API Version:  gateway.networking.k8s.io/v1beta1
Kind:         Gateway
...
Spec:
  Gateway Class Name:  gke-l7-rilb
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Addresses:
    Type:   IPAddress
    Value:  10.36.172.5
...
Events:
  Type    Reason  Age                From                   Message
  ----    ------  ----               ----                   -------
  Normal  ADD     80s                sc-gateway-controller  default/internal-http
  Normal  UPDATE  20s (x3 over 80s)  sc-gateway-controller  default/internal-http
  Normal  SYNC    20s                sc-gateway-controller  SYNC on default/internal-http was a success

Deploy an HTTPRoute manifest to route HTTP traffic to the hello-app service at port 80.

kubectl apply -f - <<EOF
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: hello-app-route
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  rules:
  - backendRefs:
    - name: hello-app
      port: 80
EOF

Create a network attachment

A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network through a Private Service Connect interface.

To publish events, Eventarc uses the network attachment to establish a connection to the internal HTTP endpoint hosted in a VPC network.

You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the HTTP destination service.

gcloud compute network-attachments create my-network-attachment \
    --region=us-central1 \
    --subnets=my-subnet\
    --connection-preference=ACCEPT_AUTOMATIC

For more information, see About network attachments.

Create an Eventarc trigger

Create an Eventarc trigger that creates a new Pub/Sub topic and routes events to the event receiver deployed on the VM when a message is published to the Pub/Sub topic.

Retrieve the Gateway address.

GATEWAY_ADDRESS=$(kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}")

Create a trigger.

gcloud eventarc triggers create my-trigger \
    --location=us-central1 \
    --destination-http-endpoint-uri="http://$GATEWAY_ADDRESS:80/" \
    --network-attachment="projects/PROJECT_ID/regions/us-central1/networkAttachments/my-network-attachment" \
    --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
    --service-account=PROJECT_NUMBER[email protected]

Replace PROJECT_NUMBER with your Google Cloud project number. You can find your project number on the Welcome page of the Google Cloud console or by running the following command:

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

For more information about configuring your trigger, see Route events to an internal HTTP endpoint in a VPC network.

Generate and view a Pub/Sub topic event

You can generate an event by publishing a message to a Pub/Sub topic.

Find and set the Pub/Sub topic as an environment variable.

export MY_TOPIC=$(gcloud eventarc triggers describe my-trigger \
    --location=us-central1 \
    --format='value(transport.pubsub.topic)')

Publish a message to the Pub/Sub topic to generate an event.
```
gcloud pubsub topics publish $MY_TOPIC --message "Hello World"
```
The Eventarc trigger routes the event to the internal HTTP endpoint in the private GKE cluster.

Check the application Pod logs and verify the event delivery.

POD_NAME=$(kubectl get pod --selector app=hello-app --output=name)
kubectl logs $POD_NAME

The body of the event should be similar to the following:

2024/04/18 20:31:43 Hello from Cloud Run! The container started successfully and is listening for HTTP requests on $PORT
{"severity":"INFO","eventType":"google.cloud.pubsub.topic.v1.messagePublished","message":"Received event of type google.cloud.pubsub.topic.v1.messagePublished.
Event data: Hello World","event":{"specversion":"1.0","id":"10935738681111260","source":"//pubsub.googleapis.com/projects/my-project/topics/eventarc-us-central1-my-trigger-224","type":"google.cloud.pubsub.topic.v1.messagePublished","datacontenttype":"application/json","time":"2024-04-18T20:40:03Z","data":
{"message":{"data":"SGVsbG8gV29ybGQ=","messageId":"10935738681111260","publishTime":"2024-04-18T20:40:03Z"}}}}

You have successfully deployed an event receiver service to an internal HTTP endpoint in a private GKE cluster, created an Eventarc trigger, generated an event from Pub/Sub, and confirmed that the event was routed as expected by the trigger to the target endpoint.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

Delete the Eventarc trigger:

  gcloud eventarc triggers delete my-trigger --location=us-central1

Exit the VM and then delete the VM instance:

  gcloud compute instances delete my-vm --zone=us-central1-a

Delete the network attachment:

  gcloud compute network-attachments delete my-network-attachment --region=us-central1

Delete the firewall rule:

  gcloud compute firewall-rules delete allow-proxy-connection

Delete the cluster:

  gcloud container clusters delete private-cluster --region=us-central1

Delete the subnet:

  gcloud compute networks subnets delete proxy-only-subnet --region=us-central1