Route Datastream events to Workflows

Console

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

2. Enable the Eventarc, Eventarc Publishing, Workflows, and Workflow Executions APIs.

   Enable the APIs

3. If applicable, enable the API related to the direct events. For example, for Datastream events, enable the Datastream API.

4. If you don't already have one, create a user-managed service account, then grant it the roles and permissions necessary so that Eventarc can manage events for a target workflow.

   1. In the Google Cloud console, go to the Service Accounts page.

      Go to Service Accounts

   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for event trigger.

   4. Click Create and continue.

   5. To provide appropriate access, in the Select a role list, select the required Identity and Access Management (IAM) roles to grant to your service account. For more information, see Roles and permissions for Workflows targets.

      For additional roles, click add Add another role and add each additional role.

   6. Click Continue.

   7. To finish creating the account, click Done.

gcloud

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Enable the Eventarc, Eventarc Publishing, Workflows, and Workflow Executions APIs:

   gcloud services enable eventarc.googleapis.com \
       eventarcpublishing.googleapis.com \
       workflows.googleapis.com \
       workflowexecutions.googleapis.com

3. If applicable, enable the API related to the direct events. For example, for Datastream events, enable datastream.googleapis.com.

4. If you don't already have one, create a user-managed service account, then grant it the roles and permissions necessary so that Eventarc can manage events for a target workflow.

   1. Create the service account:

      gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

      Replace SERVICE_ACCOUNT_NAME with the name of the service account. It must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you cannot change its name.

   2. Grant the required Identity and Access Management (IAM) roles or permissions. For more information, see Roles and permissions for Workflows targets.

Console

1. In the Google Cloud console, go to the Eventarc Triggers page.

   Go to Triggers

2. Click add_box Create trigger.
3. Type a Trigger name.

   This is the ID of the trigger and it must start with a letter. It can contain up to 63 lowercase letters, numbers, or hyphens.

4. For the Trigger type, select Google sources.
5. In the Event provider list, select Datastream.

   Note that the event provider name used in the associated Google Cloud documentation might not have a prefix of Cloud or Google Cloud. For example, on the console, Memorystore for Redis is referred to as Google Cloud Memorystore for Redis.

6. In the Event type list, from the Direct events, select an event type.
7. To specify the encoding of the event payload, in the Event data content type list, select application/json or application/protobuf.

   Note that an event payload formatted in JSON is larger than one formatted in Protobuf. This might impact reliability depending on your event destination and its limits on event size. For more information, see Known issues.

8. In the Region list, select the same region as the Google Cloud service that is generating events.

   For more information, see Eventarc locations.

9. If applicable to the event provider, click Add filter and specify the following:
   1. In the Attribute 1 field, depending on the direct event you chose, select a resource ID that can act as an event filter.
   2. Select an operator:
      • Equal
      • Path pattern

        For more information, see Understand path patterns.

   3. In the Attribute value 1 field, depending on the operator that you chose, type the exact value or apply a path pattern.
   4. If more attribute filters are applicable, click Add filter and specify the appropriate values.
10. Select the Service account that will invoke your service or workflow.

    Or, you can create a new service account.

    This specifies the Identity and Access Management (IAM) service account email associated with the trigger and to which you previously granted specific roles required by Eventarc.

11. In the Event destination list, select Workflows.
12. Select a workflow.
    

    This is the name of the workflow to pass events to. Events for a workflow execution are transformed and passed to the workflow as runtime arguments.

    For more information, see Create a trigger for Workflows.

13. Click Create.

After a trigger is created, the event source filters cannot be modified. Instead, create a new trigger and delete the old one. For more information, see Manage triggers.

gcloud

You can create a trigger by running the gcloud eventarc triggers create command along with required and optional flags.

  gcloud eventarc triggers create TRIGGER \
      --location=LOCATION \
      --destination-workflow=DESTINATION_WORKFLOW \
      --destination-workflow-location=DESTINATION_WORKFLOW_LOCATION \
      --event-filters="type=EVENT_FILTER_TYPE" \
      --event-filters="COLLECTION_ID=RESOURCE_ID" \
      --event-filters-path-pattern="COLLECTION_ID=PATH_PATTERN" \
      --event-data-content-type="EVENT_DATA_CONTENT_TYPE" \
      --service-account="SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com"

Replace the following:

• TRIGGER: the ID of the trigger or a fully qualified identifier.
• LOCATION: the location of the Eventarc trigger. Alternatively, you can set the eventarc/location property; for example, gcloud config set eventarc/location us-central1.

  To avoid any performance and data residency issues, the location must match the location of the Google Cloud service that is generating events. For more information, see Eventarc locations.

• DESTINATION_WORKFLOW: the ID of the deployed workflow that receives the events from the trigger. The workflow can be in any of the Workflows supported locations and does not need to be in the same location as the trigger. However, the workflow must be in the same project as the trigger.
• DESTINATION_WORKFLOW_LOCATION (optional): the location in which the destination workflow is deployed. If not specified, it is assumed that the workflow is in the same location as the trigger.
• EVENT_FILTER_TYPE: the identifier of the event. An event is generated when an API call for the method succeeds. For long-running operations, the event is only generated at the end of the operation, and only if the action is performed successfully. For a list of supported event types, see Google event types supported by Eventarc.

• COLLECTION_ID (optional): the resource component that can act as an event filter, and is one of the following:
  • connectionprofile
  • privateconnection
  • route
  • stream
• RESOURCE_ID: the identifier of the resource used as the filtering value for the associated collection. For more information, see Resource ID.
• PATH_PATTERN: the path pattern to apply when filtering for the resource
• EVENT_DATA_CONTENT_TYPE: (optional) the encoding of the event payload. This can be application/json or application/protobuf. The default encoding is application/json.

  Note that an event payload formatted in JSON is larger than one formatted in Protobuf. This might impact reliability depending on your event destination and its limits on event size. For more information, see Known issues.

• SERVICE_ACCOUNT_NAME: the name of the IAM service account you created to which you granted specific roles required by Workflows.
• PROJECT_ID: your Google Cloud project ID

Notes:

• The --event-filters="type=EVENT_FILTER_TYPE" flag is required. If no other event filter is set, events for all resources are matched.
• EVENT_FILTER_TYPE cannot be changed after creation. To change EVENT_FILTER_TYPE, create a new trigger and delete the old one.
• Each trigger can have multiple event filters, comma delimited in one --event-filters=[ATTRIBUTE=VALUE,...] flag, or you can repeat the flag to add more filters. Only events that match all the filters are sent to the destination. Wildcards and regular expressions are not supported; however, when using the --event-filters-path-pattern flag, you can define a resource path pattern.
• The --service-account flag is used to specify the Identity and Access Management (IAM) service account email associated with the trigger.

Example:

  gcloud eventarc triggers create helloworld-trigger \
      --location=us-central1 \
      --destination-workflow=my-workflow \
      --destination-workflow-location=us-central1 \
      --event-filters="type=google.cloud.datastream.connectionProfile.v1.updated" \
      --event-filters-path-pattern="connectionprofile=my-connectionprofile-*" \
      --service-account="${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

This command creates a trigger called helloworld-trigger for the event identified as google.cloud.datastream.connectionProfile.v1.updated and matches events for connectionprofile IDs starting with my-connectionprofile-.

Terraform

You can create a trigger for a workflow using Terraform. For details, see Trigger a workflow using Eventarc and Terraform.

Console

1. In the Google Cloud console, go to the Eventarc Triggers page.

   Go to Triggers

   This page lists your triggers in all locations, and includes details such as names, regions, event providers, destinations, and more.

2. To filter your triggers:

   1. Click filter_list Filter or the Filter triggers field.
3. In the Properties list, select an option to filter the triggers by.

You can select a single property or use the logical operator OR to add more properties.

gcloud

Run the following command to list your triggers:

gcloud eventarc triggers list --location=-

This command lists your triggers in all locations, and includes details such as names, types, destinations, and statuses.

What's next