Dataflow release notes

Dataflow is available in Queretaro, Mexico (northamerica-south1). Learn more about Google Cloud locations.

You can now use the Dataflow job builder UI to create and run Dataflow pipelines in the Google Cloud console, without writing any code. This feature is generally available (GA).

Iceberg read/write support is available through the new Managed I/O Java API. For more information, see Dataflow managed I/O.

You can now use Metrics Explorer to find individual DoFns that cause latencies in streaming jobs. These metrics are available in streaming pipelines that use Apache Beam 2.53.0 and later versions. The following new metrics are available:

• Average message processing time per DoFn (job/dofn_latency_average)
• Maximum message processing time per DoFn (job/dofn_latency_max)
• Minimum message processing time per DoFn (job/dofn_latency_min)
• Number of messages processed per DoFn (job/dofn_latency_num_messages)
• Oldest active message processing time per DoFn (job/oldest_active_message_age)
• Total message processing time per DoFn (job/dofn_latency_total)

For more information about Dataflow metrics, see Google Cloud metrics.

The following Dataflow templates now support user-defined functions (UDFs) written in Python:

• Cloud Storage Text to BigQuery
• Cloud Storage Text to BigQuery (Stream)
• Pub/Sub to BigQuery
• Pub/Sub Proto to BigQuery

The following Dataflow templates are generally available (GA):

• Google Cloud to Neo4j
• Pub/Sub to Datadog

The Dataflow right fitting feature is now supported by non-Prime batch pipelines.

You can now use worker utilization hints to tune horizontal autoscaling for streaming pipelines.

Added new autoscaling metrics:

• Autoscaling rationale chart: explains the factors driving autoscaling decisions
• Worker CPU utilization chart: shows current user worker CPU utilization and customer autoscaling hint value
• Timer backlog per stage: shows an estimate of time needed to materialize the output for windows whose timer has expired
• Parallel processing: the number of keys available for parallel processing

You can now use committed use discounts (CUDs) with Dataflow streaming jobs. Committed use discounts provide discounted prices in exchange for your commitment to continuously use a certain amount of Dataflow compute resources for a year or longer.

Dataflow now supports at-least-once streaming mode. You can use this mode to achieve lower latency and reduced costs for workloads that can tolerate duplicate records. This feature is generally available (GA). For more information, see Set the pipeline streaming mode.

You can now use Gemma models in your Apache Beam inference pipelines. For more information, see Use Gemma open models with Dataflow.

You can now use a turnkey transform to enrich streaming data in your Dataflow pipeline. When you enrich data, you augment the raw data from one source by adding related data from a second source. For more information, see Enrich streaming data.

Dataflow Streaming Engine now supports resource-based billing. When you enable resource-based billing with Streaming Engine, you're billed for the total resources consumed by your job.

Dataflow is available in Johannesburg, South Africa (africa-south1).

Dataflow now supports data sampling for pipeline exceptions. With this feature, you can see samples of the data being processed when an unhandled exception occurs. Use exception sampling to help troubleshoot pipeline errors. For more information, see Use exception sampling.

You can now run a job graph validation check to verify whether a replacement job is valid before you launch the new job. For more information, see Validate a replacement job.

You can now archive completed Dataflow jobs. When you archive a Dataflow job, the job is moved from the Dataflow Jobs page in the console to the Archived jobs page. For more information, see Archive Dataflow jobs.

The Dataflow web-based monitoring interface now includes a dashboard that monitors your Dataflow jobs at the project level. For more information, see Dataflow project monitoring dashboard.

The Cloud Spanner to Vertex AI Vector Search template is generally available (GA).

Dataflow jobs now scale to 4,000 worker VMs.

The Cloud Spanner to BigQuery template for batch pipelines is available in preview.

Dataflow is now available in Dammam, Saudi Arabia (me-central2).

Dataflow now supports the Tau T2A Arm machine series as a worker machine type. This feature is generally available (GA). For more information, see Use Arm VMs on Dataflow.

The following Dataflow templates are generally available (GA):

• BigQuery to Bigtable
• Pub/Sub to Splunk

Dataflow is available in Berlin (europe-west10).

You can now update streaming job options without stopping your job. For more information, see In-flight job option update.

Dataflow cost monitoring is generally available (GA).

The following Dataflow templates are generally available (GA):

• MySQL to BigQuery
• PostgreSQL to BigQuery
• SQL Server to BigQuery

Dynamic thread scaling is generally available (GA). Dynamic thread scaling is a part of Dataflow's suite of vertical scaling features.

When you run multiple SDK processes on a shared Dataflow GPU, you can improve GPU efficiency and utilization by enabling the NVIDIA Multi-Process Service (MPS).

You can now view streaming stragglers in the Google Cloud console. For more information, see Troubleshoot stragglers in streaming jobs.

Dataflow now supports Confidential VMs for Dataflow worker VMs. For more information, see Dataflow service options.

Data sampling is now generally available (GA). Data sampling lets you observe the data at each step of a pipeline. For more information, see Use data sampling to observe pipeline data.

Dataflow ML now supports the Automatic Model Refresh feature, which lets you update your machine learning model without stopping your Apache Beam pipeline.

You can now manage Dataflow jobs by using Eventarc. For more information, see Use Eventarc to manage Dataflow jobs.

Dataflow cost monitoring is now available in preview.

Dataflow is now available in Doha (me-central1).

Vertical Autoscaling now supports batch jobs.

Dataflow is now available in Turin (europe-west12).

Starting in version 2023-01-03-00_RC00, the Google-provided Dataflow templates support ES6 syntax for JavaScript user-defined functions (UDFs). This change is backwards-compatible. ES5 syntax and existing user-defined functions are still supported.

When you run Google-provided templates using the latest version, your jobs are upgraded automatically on restart. If you want to keep running an earlier version of a template, when you run the template, specify version 2022-12-15-00_RC00 or earlier.

Dataflow now supports regional placement for workers.

Dataflow is now available in Tel Aviv (me-west1).

Dataflow Prime is now in General Availability.

You can use the Apache Beam SDK for Go to create batch and streaming Dataflow pipelines. This feature is now in General Availability.

Dataflow is now available in Dallas, Texas (us-south1).

Dataflow is now available in Columbus (us-east5).

Dataflow now supports Flex Template images from private registries. To learn more, see Use an image from a private registry.

Dataflow is now available in Madrid (europe-southwest1).

Dataflow is now available in Paris (europe-west9).

Dataflow is now available in Milan (europe-west8).

Dataflow now supports Runner v2 in GA for all languages.

Dataflow now supports the following Google-provided templates in GA:

• Pub/Sub Subscription to BigQuery
• Pub/Sub Topic to BigQuery
• Pub/Sub Avro to BigQuery
• Pub/Sub Proto to BigQuery
• Pub/Sub to Pub/Sub
• Pub/Sub to Cloud Storage Avro
• Pub/Sub to Cloud Storage Text
• Cloud Storage Text to BigQuery (Stream)
• Cloud Storage Text to Pub/Sub (Stream)
• Change Data Capture to BigQuery (Stream)
• Apache Kafka to BigQuery
• BigQuery export to Parquet (via Storage API)
• Firestore to Cloud Storage Text
• Cloud Spanner to Cloud Storage Text
• Cloud Storage Text to BigQuery
• Cloud Storage Text to Firestore
• Cloud Storage Text to Pub/Sub (Batch)
• Apache Cassandra to Bigtable
• Datastream to Cloud Spanner
• File Format Conversion
• Bulk Compress Cloud Storage Files
• Bulk Decompress Cloud Storage Files
• Firestore Bulk Delete
• Streaming Data Generator to Pub/Sub, BigQuery, and Cloud Storage

You can now use the Apache Beam SDK for Go to create batch Dataflow pipelines. This feature is in Preview.

Profiling Dataflow pipelines with Cloud Profiler is generally available (GA). Use Dataflow integration with Cloud Profiler to monitor pipeline performance.

Dataflow now fully supports using Identity and Access Management (IAM) custom roles. You can create a custom IAM role and assign it to a user-managed service account used in Dataflow instead of assigning the Dataflow Worker role.

Dataflow is now available in Santiago (southamerica-west1).

Dataflow now supports Shielded VM workers.

Dataflow Prime is now available in Preview.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Toronto (northamerica-northeast2).

Dataflow now supports storing Flex Template images in Artifact Registry.

Dataflow now supports custom containers in GA.

GPU support on Dataflow is now in General Availability.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Delhi (asia-south2).

Dataflow snapshots are now available in GA.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Melbourne (australia-southeast2).

In addition to scalar functions, Dataflow SQL now supports aggregate user-defined functions (UDFs) for Java. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

Dataflow SQL now supports user-defined functions (UDFs) written using Java. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

You can now enable logging of human-readable hot keys. For more information, see the hot key entry in Pipeline options.

Dataflow Shuffle is now the default mode for all batch pipelines.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in europe-central2 (Warsaw).

Dataflow SQL now supports user-defined functions (UDFs) written using SQL. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

Execution details are now available in Preview.

Dataflow now supports Dataflow Shuffle, Streaming Engine, FlexRS, and the following regional endpoints in GA:

• asia-east2 Hong Kong
• asia-northeast2 - Japan (Osaka)
• asia-northeast3 - Seoul
• asia-southeast2 - Jakarta
• europe-north1 - Finland
• us-west3 - Salt Lake City
• us-west4 - Las Vegas

Dataflow snapshots are now available in Preview.

GPU support on Dataflow is currently available in Preview. To enroll in this Preview offering, contact Support or Sales.

Dataflow now supports custom containers as a Preview offering.

Dataflow now supports Interactive Notebooks in GA.

Dataflow now supports Dataflow Shuffle, Streaming Engine, FlexRS, and the following regional endpoints in GA:

• us-west2 (Los Angeles)
• southamerica-east1 (São Paulo)
• europe-west6 (Zurich)
• asia-south1 (Mumbai)

Dataflow now supports Flex Templates in GA.

You can now use a network tags parameter to add network tags to all worker VMs that execute a particular Dataflow job.

Dataflow now supports Dataflow Shuffle, Streaming Engine, FlexRS, and the following regional endpoints in GA:

• northamerica-northeast1 (Montréal)
• asia-southeast1 (Singapore)
• australia-southeast1 (Sydney)

Dataflow is now able to use workers in zones in the asia-southeast2 region (Jakarta).

Dataflow is now able to use workers in zones in the us-west4 region (Las Vegas).

Cloud Dataflow SQL is now generally available. You can now run parameterized queries from the Dataflow SQL UI.

Dataflow now provides beta support for Flex Templates.

Dataflow now provides beta support for Interactive Notebooks.

Dataflow now supports Dataflow Shuffle, Streaming Engine, FlexRS, and the following regional endpoints in GA:

• us-east4 (Northern Virginia)
• europe-west2 (London)
• europe-west3 (Frankfurt)

Cloud Dataflow SQL is now available in beta. You can now do the following in Cloud Dataflow SQL:

• Write data to two destinations, including Cloud Pub/Sub
• Specify how to load data into a BigQuery table
• Set pipeline options in the Cloud Dataflow SQL UI

Using Cloud Dataflow with Cloud Key Management Service to create a customer-managed encryption key (CMEK) is generally available.

Cloud Dataflow is now able to use workers in zones in the us-west3 region (Salt Lake City).

The Cloud Dataflow monitoring UI now has enhanced observability features to help with troubleshooting batch and streaming pipelines.

Cloud Dataflow is now able to use workers in zones in the asia-northeast3 region (Seoul).

Flexible Resource Scheduling (FlexRS) in Cloud Dataflow is generally available. The service is available in five additional regions:

• us-east1 (South Carolina)
• us-west1 (Oregon)
• asia-east1 (Taiwan)
• asia-northeast1 (Tokyo)
• europe-west4 (Netherlands)

You can now do the following in Cloud Dataflow SQL:

• Use Cloud Storage filesets as a data source
• Assign schemas to data sources in the Cloud Dataflow SQL UI
• Preview the content of Cloud Pub/Sub messages from the Cloud Dataflow SQL UI

Cloud Dataflow Shuffle and Streaming Engine are now available in us-east1 (South Carolina).

You can now see audit logs of Cloud KMS key operations and protect Cloud Dataflow Shuffle state using a customer-managed encryption key.

Python streaming for Apache Beam SDK 2.16 or higher is generally available. You can now do the following in Python:

• Update and Drain streaming pipelines.
• Enable streaming autoscaling.
• Use Streaming Engine.

Python 3 support for Apache Beam SDK 2.16.0 or higher is now generally available. This feature provides support for using Python 3.5, 3.6, and 3.7. You can run run any existing Python 2.7 batch and streaming pipelines that use DirectRunner or DataflowRunner. However, you might need to make changes to ensure that your pipeline code is compatible with Python 3. Keyword-only arguments (a syntactic construct introduced in Python 3) are not yet supported by Apache Beam SDK. For the current status and summary of recent Python 3-specific improvements, follow updates on the Apache Beam issue tracker.

Cloud Dataflow Shuffle and Streaming Engine are now available in two additional regions:

• us-west1 (Oregon)
• asia-east1 (Taiwan)

Automatic hot key detection is now enabled in batch pipelines for Apache Beam SDK 2.15.0 or higher.

Integration with Cloud Dataflow VPC Service Controls is generally available.

Using Cloud Dataflow with Cloud Key Management Service is now available in beta. Customer-managed encryption keys (CMEK) allow for encryption of your pipeline state. This feature is limited to Persistent Disks attached to Cloud Dataflow workers and used for Persistent Disk-based shuffle and streaming state storage.

Python 3 support for Apache Beam SDK 2.14.0 or higher is now in beta. This feature provides support for using Python 3.5, 3.6, and 3.7. You can run any existing Python 2.7 batch and streaming pipelines that use DirectRunner or DataflowRunner. However, you might need to make changes to ensure that your pipeline code is compatible with Python 3. Some syntactic constructs introduced in Python 3 are not yet fully supported by the Apache Beam SDK. For details and current status, follow updates on the Apache Beam issue tracker.

Cloud Dataflow SQL is now publicly available in alpha. Cloud Dataflow SQL lets you use SQL queries to develop and run Cloud Dataflow jobs from the BigQuery web UI.

Cloud Dataflow is now able to use workers in zones in the asia-northeast2 region (Osaka, Japan).

Cloud Dataflow Streaming Engine is generally available. The service is available in two additional regions:

• asia-northeast1 (Tokyo)
• europe-west4 (Netherlands)

Note that Streaming Engine requires the Apache Beam SDK for Java, versions 2.10.0 or higher.

Cloud Dataflow Shuffle is now available in two additional regions:

• asia-northeast1 (Tokyo)
• europe-west4 (Netherlands)

Cloud Dataflow provides beta support for Flexible Resource Scheduling (FlexRS) in the us-central1 and europe-west1 regions.

Streaming autoscaling is generally available for pipelines that use Streaming Engine.

Cloud Dataflow provides beta support for Flexible Resource Scheduling (FlexRS) in the us-central1 and europe-west1 regions.

Cloud Dataflow provides beta support for VPC Service Controls.

Cloud Dataflow is now able to use workers in zones in the europe-west6 region (Zürich, Switzerland).

You can now view system latency and data freshness metrics for your pipeline in the Cloud Dataflow monitoring interface.

Cloud Dataflow is now able to use workers in zones in the asia-east2 region (Hong Kong).

Cloud Dataflow now has a Public IP parameter that allows you to turn off public IP addresses for your worker nodes.

Cloud Dataflow Shuffle is now generally available.

Cloud Dataflow is now able to use workers in zones in the us-west2 region (Los Angeles).

Streaming Engine is now publicly available in beta. Streaming Engine moves streaming pipeline execution out of the worker VMs and into the Cloud Dataflow service backend.

You can now specify a user-managed controller service account when you run your pipeline job.

Cloud Dataflow is now able to use workers in zones in the europe-north1 region (Finland).

You can now view side input metrics for your pipeline from the Cloud Dataflow monitoring interface.

Cloud Dataflow now supports the following regional endpoints in GA: us-central1, us-east1, europe-west1, asia-east1, and asia-northeast1.

Cloud Dataflow is now able to use workers in zones in the northamerica-northeast1 region (Montréal).

Cloud Dataflow is now able to use workers in zones in the europe-west4 region (Netherlands).

Cloud Dataflow is now able to use workers in zones in the asia-south1 region (Mumbai).

Cloud Dataflow Shuffle is now available in the europe-west1 region.

Cloud Dataflow Shuffle is now available for pipelines using the Apache Beam SDK for Python version 2.1 or later.

Cloud Dataflow provides beta support for additional Google-provided templates. To get started with templates, follow the quickstart.

• Google Cloud Pub/Sub to Google BigQuery
• Google Cloud Storage Text to Google Cloud Pub/Sub
• Google Cloud Pub/Sub to Google Cloud Storage Text
• Google Cloud Datastore to Google Cloud Storage Text

Cloud Dataflow provides beta support for regional endpoints us-central1 and europe-west1.

Cloud Dataflow is now able to use workers in zones in the southamerica-east1 region (São Paulo).

Cloud Dataflow is now able to use workers in zones in the europe-west3 region (Frankfurt).

You can now access the Stackdriver error report for your pipeline directly from the Dataflow monitoring interface.

Cloud Dataflow is now able to use workers in zones in the australia-southeast1 region (Sydney).

Cloud Dataflow is now able to use workers in zones in the europe-west2 region (London).

Per-step worker logs are now accessible directly in the Cloud Dataflow UI. Consult the documentation for more information.

Autoscaling for streaming pipelines is now publicly available in beta for use with select sources and sinks. See the autoscaling documentation for more details.

Announced beta support for the 0.4.0 release of the Cloud Dataflow SDK for Python. Get started and run your pipeline remotely on the service.

The Cloud Dataflow Service is now generally available.

The Cloud Dataflow Service is now in beta.

Command line interface now available for Cloud Dataflow in gcloud alpha.

Improved mechanisms for elastic scaling of compute resources. Batch pipelines can now grow and shrink the worker pool size at different stages of execution.

Dynamic work rebalancing