Troubleshoot Eventarc for Google Kubernetes Engine (GKE)

This page shows you how to resolve issues that you might encounter when using Eventarc for GKE, including the public endpoints of private and public services running in a GKE cluster.

For other issues that you might also encounter, refer to troubleshooting for:

Trigger creation fails with resource creation permission errors

You receive error messages similar to one of the following: 

Error applying IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT':
Error setting IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT':
googleapi: Error 403: Permission iam.serviceAccounts.setIamPolicy is required to perform this
operation on service account SERVICE_ACCOUNT., forbidden

Error when reading or editing Container Cluster "cluster": googleapi:
Error 403: Required "container.clusters.get" permission(s) for "..."., forbidden

Error reading instance group manager returned as an instance group URL:
"googleapi: Error 403: Required 'compute.instanceGroupManagers.get' permission
for '...', forbidden"

This error occurs when you have not correctly set up the Eventarc service account that is required by Eventarc to manage resources in the Google Kubernetes Engine (GKE) cluster.

To resolve the issue, ensure that the Eventarc service account has been configured correctly with the appropriate permissions to create resources. For more information, follow the instructions for a specific provider, event type, and GKE destination.

Trigger creation fails because target cluster cannot be found or does not have Workload Identity Federation for GKE enabled

You receive error messages similar to one of the following:

ERROR: (gcloud.eventarc.triggers.create) NOT_FOUND: Not found: projects/PROJECT_ID/locations/
LOCATION/clusters/CLUSTER_NAME.
...

ERROR: (gcloud.eventarc.triggers.create) INVALID_ARGUMENT: workload identity is not enabled on
cluster CLUSTER_NAME in LOCATION
...

This error occurs when Eventarc cannot find the target GKE cluster or the cluster does not have Workload Identity Federation for GKE enabled.

To resolve the issue, ensure that the target cluster exists and has Workload Identity enabled.

Trigger creation fails because of invalid resource state

You receive an error message similar to the following for your project:

ERROR: (gcloud.eventarc.triggers.create) Invalid resource state for projects/PROJECT_ID/locations/LOCATION/triggers/TRIGGER_ID.
Ensure that you have enabled your GKE destination and assigned required permissions to the service account.
For more information, see https://cloud.google.com/eventarc/standard/docs/gke/troubleshooting.

This error occurs when:

  • Your GKE destination hasn't been enabled.
  • The service account has not been granted the required roles and permissions that enables Eventarc to manage events for GKE destinations.

To resolve this issue:

  1. Ensure that you have enabled GKE destinations: 
    gcloud eventarc gke-destinations init
    For more information, follow the instructions for a specific provider and event type in the "Enable GKE destinations" section.
  2. Ensure that the Eventarc service account has been configured with the appropriate permissions to create resources. For more information on how to grant the appropriate roles to the service account, follow the instructions for a specific provider and event type in the "Prepare to create a trigger" section.
  3. If the error persists, contact support.

Trigger is created successfully but target is not receiving events

Check that enough time has elapsed since creating the trigger. It can take up to two minutes before events are sent.

  1. If events are not being delivered at the target, events sent from Pub/Sub to the target might be getting dropped:

    To ensure that events aren't dropped, set up a Pub/Sub subscription retry policy or forward undelivered messages to a dead-letter topic (also known as a dead-letter queue).

    Before setting the dead-letter topic, retrieve the trigger's topic and subscription:

    gcloud eventarc triggers describe TRIGGER \
  --location=LOCATION

    Replace the following:

    • TRIGGER: the ID of the trigger or a fully qualified identifier.
    • LOCATION: the location of the Eventarc trigger.

  2. If the trigger still doesn't work, check the status of Eventarc's event forwarder component to see if it has been deployed properly:

    1. Identify the cluster namespaces by listing all pods that have event-forwarder in their name: 
      kubectl get pods --all-namespaces | grep event-forwarder
      The output is similar to the following: 
      event-forwarder-namespace-hash1   event-forwarder-hash1   1/1   Running   0   5d2h
event-forwarder-namespace-hash2   event-forwarder-hash2   1/1   Running   0   2d2h
      The first 30 characters of the namespaces should match the name of the Eventarc trigger you created, followed by a hash to disambiguate any triggers with the same initial 30 characters.
    2. Retrieve the deployment details: 
      kubectl get deployments -n EVENT_FORWARDER_NAMESPACE event-forwarder
    3. Retrieve the pod details: 
      kubectl describe pod -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
    4. Print the pod logs to investigate why or where the failure has occurred: 
      kubectl logs -f -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
      For example, in the following log, the service account has had the Pub/Sub Subscriber role revoked: 
      {"error":"generic::permission_denied: missing 'pubsub.subscriptions.consume' permission",
"message":"Permissions check failed","severity":"fatal","timestamp":"2021-07-07T14:57:49.038877483Z"}