Execute a Cloud Run job that processes event data saved in Cloud Storage

Console

1. In the Google Cloud console, go to the Workflows page:

   Go to Workflows

2. Click add Create.

3. Enter a name for the new workflow, such as message-payload-workflow.

4. Choose an appropriate region; for example, us-central1.

5. In the Service account field, select the service account you created earlier.

   The service account serves as the workflow's identity. You should have already granted the following roles to the service account:

   • Cloud Run Admin: to execute Cloud Run jobs
   • Logs Writer: to write log entries
   • Storage Object Creator: to create objects in Cloud Storage

6. Click Next.

7. In the workflow editor, enter the following definition for your workflow:

   main:
       params: [event]
       steps:
           - init:
               assign:
                   - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
                   - event_bucket: ${"message-payload-" + project_id}
                   - event_file: ${event.id + ".data.json"}
                   - job_name: message-payload-job
                   - job_location: us-central1
           - log_event:
               call: sys.log
               args:
                   data: ${event}
           - write_payload_to_gcs:
               call: http.post
               args:
                   url: ${"https://storage.googleapis.com/upload/storage/v1/b/" + event_bucket + "/o"}
                   auth:
                       type: OAuth2
                   query:
                       name: ${event_file}
                   body:
                       ${event.data}
           - run_job_to_process_payload:
               call: googleapis.run.v1.namespaces.jobs.run
               args:
                   name: ${"namespaces/" + project_id + "/jobs/" + job_name}
                   location: ${job_location}
                   body:
                       overrides:
                           containerOverrides:
                               env:
                                   - name: INPUT_BUCKET
                                     value: ${event_bucket}
                                   - name: INPUT_FILE
                                     value: ${event_file}
               result: job_execution
           - finish:
               return: ${job_execution}

8. Click Deploy.

gcloud

1. Create a source code file for your workflow:

   touch message-payload-workflow.yaml
   

2. Copy the following workflow definition to message-payload-workflow.yaml:

   main:
       params: [event]
       steps:
           - init:
               assign:
                   - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
                   - event_bucket: ${"message-payload-" + project_id}
                   - event_file: ${event.id + ".data.json"}
                   - job_name: message-payload-job
                   - job_location: us-central1
           - log_event:
               call: sys.log
               args:
                   data: ${event}
           - write_payload_to_gcs:
               call: http.post
               args:
                   url: ${"https://storage.googleapis.com/upload/storage/v1/b/" + event_bucket + "/o"}
                   auth:
                       type: OAuth2
                   query:
                       name: ${event_file}
                   body:
                       ${event.data}
           - run_job_to_process_payload:
               call: googleapis.run.v1.namespaces.jobs.run
               args:
                   name: ${"namespaces/" + project_id + "/jobs/" + job_name}
                   location: ${job_location}
                   body:
                       overrides:
                           containerOverrides:
                               env:
                                   - name: INPUT_BUCKET
                                     value: ${event_bucket}
                                   - name: INPUT_FILE
                                     value: ${event_file}
               result: job_execution
           - finish:
               return: ${job_execution}

3. Deploy the workflow by entering the following command:

   gcloud workflows deploy message-payload-workflow \
       --location=us-central1 \
       --source=message-payload-workflow.yaml \
       --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com
   

   Replace the following:

   • SERVICE_ACCOUNT_NAME: the name of the service account you created earlier
   • PROJECT_ID: the ID of your Google Cloud project

   The service account serves as the workflow's identity. You should have already granted the following roles to the service account:

   • roles/logging.logWriter: to write log entries
   • roles/run.admin: to execute Cloud Run jobs
   • roles/storage.objectCreator: to create objects in Cloud Storage

Console

1. In the Google Cloud console, go to the Topics page.

   Go to Topics

2. Click add Create topic.

3. In the Topic ID field, enter an ID for the topic, such as message-payload-topic.

4. Retain the option Add a default subscription.

5. Don't select the other options.

6. Click Create.

gcloud

To create a topic with the ID message-payload-topic, run the gcloud pubsub topics create command:

gcloud pubsub topics create message-payload-topic

Console

1. In the Google Cloud console, go to the Workflows page:

   Go to Workflows

2. Click the name of your workflow, such as message-payload-workflow.

3. On the Workflow details page, click edit Edit.

4. On the Edit workflow page, in the Triggers section, click Add new trigger > Eventarc.

   The Eventarc trigger pane opens.

5. In the Trigger name field, enter a name for the trigger, such as message-payload-trigger.

6. From the Event provider list, select Cloud Pub/Sub.

7. From the Event list, select google.cloud.pubsub.topic.v1.messagePublished.

8. From the Select a Cloud Pub/Sub topic list, select the Pub/Sub topic you previously created.

9. In the Service account field, select the service account you created earlier.

   The service account serves as the trigger's identity. You should have already granted the following roles to the service account:

   • Eventarc Event Receiver: to receive events
   • Workflows Invoker: to execute workflows

10. Click Save trigger.

    The Eventarc trigger now appears in the Triggers section on the Edit workflow page.

11. Click Next.

12. Click Deploy.

gcloud

Create an Eventarc trigger by running the following command:

gcloud eventarc triggers create message-payload-trigger \
    --location=us-central1 \
    --destination-workflow=message-payload-workflow \
    --destination-workflow-location=us-central1 \
    --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
    --transport-topic=projects/PROJECT_ID/topics/message-payload-topic \
    --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com

Replace the following:

• PROJECT_ID: the ID of your Google Cloud project
• SERVICE_ACCOUNT_NAME: the name of the service account you created earlier.

The service account serves as the trigger's identity. You should have already granted the following roles to the service account:

• roles/eventarc.eventReceiver: to receive events
• roles/workflows.invoker: to execute workflows