Create pipeline alerts

This page describes how to set up alerts for data pipelines in Cloud Data Fusion.

You can create alerts through the following services:

  • Alerts through Cloud Data Fusion: you set up these alerts when you design a batch pipeline in Cloud Data Fusion. After you deploy the pipeline, you can't add or edit an alert.

  • Log-based alerts through Cloud Monitoring: you set up these alerts in the Google Cloud console before and after you deploy the pipeline.

Before you begin

You need a Cloud Data Fusion instance. For more information, see Creating a Cloud Data Fusion instance.

Add an alert in Cloud Data Fusion

  1. Open your instance.

    1. In the Google Cloud console, go to the Cloud Data Fusion page.

    2. To open the instance in the Cloud Data Fusion Studio, click Instances, and then click View instance.

      Go to Instances

  2. Go to the Studio page where you're designing your pipeline.

  3. Click Configure > Pipeline alerts > Add.

  4. Select and configure the alert. For more information about configuring alerts, see Batch pipeline alerts.

  5. After you configure the alert, click Validate to check for errors, and then click Confirm.

Create an alert in Monitoring

  1. Enable Cloud Logging in your Cloud Data Fusion instance.

  2. Create a log-based alert.

    1. In the Google Cloud console, go to the Logs Explorer page.

      Go to the Logs Explorer

      This page lets you store, query, update, and delete data.

    2. Enter the following query:

      resource.type="cloud_dataproc_cluster"
logName="projects/PROJECT_ID/logs/datafusion-pipeline-logs"
resource.labels.project_id="PROJECT_ID"
severity=ERROR
jsonPayload.message=~"Pipeline '.*' failed."

      Replace PROJECT_ID with your project's ID.

    3. Click Create alert and configure the alert.

An alert is only generated when the pipeline runs on a Dataproc cluster and fails. For Dataproc cluster creation failures, the alert is not generated.

What's next