Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Pub/Sub, Google Cloud Functions, Google Cloud Dataflow

Global: Cloud Pubsub experiencing elevated latencies and failures.

Incident began at 2021-08-31 11:32 and ended at 2021-08-31 14:09 (all times are US/Pacific).

Date Time Description
1 Sep 2021 13:44 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 31 August 2021 11:32

Incident End: 31 August 2021 14:09

Duration: 2 hours, 37 minutes

Affected Services and Features:

  • Google Cloud Pubsub - AdminOperation CreateSubscription, Backlog Stats Lookup (how many messages are in the queue to be received and acknowledged, how old is the oldest unacknowledged message)
  • Google Cloud Functions - Deployments
  • Google Cloud Dataflow - Jobs and Watermarks

Regions/Zones: Global

Description: Google Cloud Pubsub experienced problems creating new subscriptions and viewing backlog statistics for a period of 2 hours and 37 minutes. The Pubsub issue also caused issues with Google Cloud Functions deployments and affected related Google Cloud Dataflow streaming jobs and watermarks. From preliminary analysis, the root cause of the issue was an unhealthy task in our Pubsub change notification system that created a feedback loop with increased errors on backlog stats which scaled up more tasks and put more load on our Pubsub change notification system. Google Cloud Dataflow Streaming jobs that interact with PubSub were stalled due to the dependency of those jobs on Backlog Stats which are required to calculate the watermark.

Customer Impact:

  • Google Cloud Pubsub - Users experienced latencies creating subscriptions and viewing their backlog status data such as how many messages they have in the queue to be received and acknowledged and how old is the oldest unacknowledged message)
  • Google Cloud Functions - Users experienced issues with deployments.
  • Google Cloud Dataflow - Streaming jobs that interact with PubSub stalled and experienced lagging watermarks.
31 Aug 2021 14:17 PDT

The issue with Cloud Pub/Sub, Cloud Functions, and Dataflow has been resolved for all affected projects as of Tuesday, 2021-08-31 14:09 US/Pacific, 13:02 US/Pacific, and 13:16 US/Pacific respectively.

We thank you for your patience while we worked on resolving the issue.
31 Aug 2021 14:01 PDT

Summary: Global: Cloud Pubsub experiencing elevated latencies and failures.

Description: The issue with Cloud Functions and Dataflow are now resolved and mitigation work is currently underway by our engineering team for Cloud Pub/Sub.

The mitigation is expected to complete by Tuesday, 2021-08-31 15:30 US/Pacific.

Cloud Pub/Sub:

  • Impact is ongoing
  • Customers may experience latencies creating subscriptions.
  • Customers are unavailable to lookup their backlog data.

Cloud Functions:

  • Impact resolved and customers should no longer experience errors with deployments

Dataflow:

  • Impact resolved and customers should no longer experience issues with jobs.

If you are still experiencing issues with Cloud Functions or Dataflow, please open a support case so we can work with you to resolve those issues.

We will provide more information by Tuesday, 2021-08-31 15:30 US/Pacific.

Diagnosis: Customers could experience elevated latencies and failures for Cloud Pub/Sub

Workaround: None at this time.
31 Aug 2021 13:27 PDT

Summary: Global: Cloud Pubsub, Cloud Functions and Cloud Dataflow experiencing elevated latencies and failures.

Description: The issue with Cloud Functions and Dataflow are now resolved and mitigation work is currently underway by our engineering team for Cloud Pub/Sub.

We do not have an ETA for Cloud Pub/Sub mitigation at this point.

Cloud Pub/Sub:

  • Impact is ongoing
  • Customers may experience latencies creating subscriptions.
  • Customers are unavailable to lookup their backlog data.

Cloud Functions:

  • Impact resolved and customers should no longer experience errors with deployments

Dataflow:

  • Impact resolved and customers should no longer experience issues with jobs.

If you are still experiencing issues with Cloud Functions or Dataflow, please open a support case so we can work with you to resolve those issues.

We will provide more information by Tuesday, 2021-08-31 14:30 US/Pacific.

Diagnosis: Customers could experience elevated latencies and failures for Cloud Pub/Sub

Workaround: None at this time.
31 Aug 2021 12:58 PDT

Summary: Global: Cloud Pubsub, Cloud Functions and Cloud Dataflow experiencing elevated latencies and failures.

Description: We are experiencing an issue with Cloud Pub/Sub, Cloud Functions and Cloud Dataflow beginning at Tuesday, 2021-08-31 11:30 US/Pacific.

Cloud Pub/Sub:

  • Customers may experience latencies creating subscriptions.
  • Customers are unavailable to lookup their backlog data.

Cloud Functions:

  • Users are experiencing errors with deployment

Dataflow:

  • Customers may experience lagging jobs/watermarks

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-08-31 13:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers could experience elevated latencies and failures for Cloud Pub/Sub, Cloud Functions and Dataflow jobs.

Workaround: None at this time.