Dataproc release notes

Dataproc Serverless for Spark: Added common AI/ML Python packages by default to Dataproc Serverless for Spark 1.2 and 2.2 runtimes.

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 3.0.3 version in the latest 1.2 and 2.2 runtimes.

Dataproc Clusters created with image versions 2.0.57+, 2.1.5+, or 2.2+: Secondary workers' control plane operations are made by the Dataproc Service Agent service account (service-<project-number>@dataproc-accounts.iam.gserviceaccount.com). They will no longer use the Google APIs Service Agent service account (<project-number>@cloudservices.gserviceaccount.com).

Dataproc Serverless for Spark: In runtime versions 1.2 and 2.2, minimized the dynamic memory footprint of the Spark application by setting XX:MaxHeapFreeRatio to 30% and XX:MinHeapFreeRatio to 10%.

Dataproc Serverless for Spark: Added the google-cloud-dlp Python package by default to the Dataproc Serverless for Spark runtimes.

Dataproc on Compute Engine: The latest 2.2 image versions now support Hudi 0.15.0.

Dataproc on Compute Engine: The latest 2.2 image versions support Hudi Trino integration natively. If both components are selected when you create a Dataproc cluster, Trino will be configured to support Hudi automatically.

Dataproc on Compute Engine: Apache Spark upgraded to version 3.5.1 in image version 2.2 starting with image version 2.2.30.

New Dataproc Serverless for Spark runtime versions:

• 1.1.76
• 1.2.20
• 2.0.84
• 2.2.20

New Dataproc Serverless for Spark runtime versions:

• 1.1.75
• 1.2.19
• 2.0.83
• 2.2.19

New Dataproc Serverless for Spark runtime versions:

• 1.1.74
• 1.2.18
• 2.0.82
• 2.2.18

Dataproc Serverless for Spark: Upgraded Spark BigQuery connector to version 0.36.4 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.73
• 1.2.17
• 2.0.81
• 2.2.17

Replaced Spark external packages with connector folder on Dataproc 2.2 clusters.

Hyperdisks for Dataproc clusters are now created with default throughput and IOPS. When this behavior becomes configurable, it will be announced in a future release note.

Added support for N4 and C4 machine types for Dataproc image versions 2.1 and above. The following default configurations are now applied to clusters created with N4 or C4 machine types:

• bootdisktype = "hyperdisk-balanced"
• nictype = "gvnic"

New Dataproc Serverless for Spark runtime versions:

• 1.1.72
• 1.2.16
• 2.0.80
• 2.2.16

Note: Dataproc Serverless for Spark runtime versions 1.1.71, 1.2.15, 2.0.79, and 2.2.15 were not released.

New Dataproc on Compute Engine subminor image versions:

• 2.0.111-debian10, 2.0.112-rocky8, 2.0.112-ubuntu18
• 2.1.59-debian11, 2.1.60-rocky8, 2.1.60-ubuntu20, 2.1.60-ubuntu20-arm
• 2.2.25-debian12, 2.2.26-rocky9, 2.2.26-ubuntu22

Dataproc on Compute Engine: Apache Hadoop upgraded to version 3.2.4 in image version 2.0 starting with image version 2.0.109.

The Dataproc Component Gateway is now activated by default when you create a Dataproc on Compute Engine cluster using the Google Cloud console.

New Dataproc on Compute Engine subminor image versions:

• 2.0.107-debian10, 2.0.107-rocky8, 2.0.107-ubuntu18
• 2.1.55-debian11, 2.1.55-rocky8, 2.1.55-ubuntu20, 2.1.55-ubuntu20-arm
• 2.2.21-debian12, 2.2.21-rocky9, 2.2.21-ubuntu22

Dataproc Serverless for Spark: Upgraded Spark BigQuery connector to version 0.36.3 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

Support configuration to prevent HiveMetaStore metrics expensive database queries. To prevent expensive queries during HiveMetaStore startup, set Hive property metastore.initial.metadata.count.enabled to false.

Dataproc on Compute Engine: When creating a cluster with the latest Dataproc on Compute Engine image versions, the secondary worker boot disk type now defaults to the primary worker boot disk type, which is pd-standard if the primary worker boot disk type is not specified.

Dataproc on Compute Engine: Update restartable job error messages to include job IDs.

Dataproc Serverless for Spark: Automatically apply goog-dataproc-session-id, goog-dataproc-session-uuid and goog-dataproc-location labels for a session resource.

Dataproc Serverless for Spark: Removed Spark data lineage support for runtime version 1.2.

Dataproc Serverless for Spark: Enabled Spark checkpoint (spark.checkpoint.compress) and RDD (spark.rdd.compress) compression in the latest 1.2 and 2.2 runtime versions.

Blocklisted the following Dataproc on Compute Engine subminor image versions:

• 2.0.103-debian10, 2.0.103-rocky8, 2.0.103-ubuntu18
• 2.1.51-debian11, 2.1.51-rocky8, 2.1.51-ubuntu20, 2.1.51-ubuntu20-arm
• 2.2.17-debian12, 2.2.17-rocky9, 2.2.17-ubuntu22

Upgraded Spark BigQuery connector to version 0.36.2 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

New Dataproc on Compute Engine subminor image versions:

• 2.0.101-debian10, 2.0.101-rocky8, 2.0.101-ubuntu18

• 2.1.49-debian11, 2.1.49-rocky8, 2.1.49-ubuntu20, 2.1.49-ubuntu20-arm

• 2.2.15-debian12, 2.2.15-rocky9, 2.2.15-ubuntu22

Dataproc on Compute Engine:

• Backported patches for HIVE-14557, HIVE-19326, HIVE-20514, HIVE-21100, HIVE-22165, HIVE-22416, HIVE-24435.
• Hive: Improved ORC split generation.

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS to version 24.04.0 in 1.2 and 2.2 Dataproc Serverless for Spark runtimes.

Set the soft delete policy of newly created Dataproc staging and temp Cloud Storage buckets to 0 days.

Updated the default autoscaling V2 cool-down time from 2m to 1m to reduce scaling latency.

Added bigframes Python package by default in the Dataproc Serverless for Spark runtime versions 1.2 and 2.2

Dataproc Serverless for Spark:

• Upgraded Spark to version 3.5.1 in the latest 1.2 and 2.2 runtimes.
• Upgraded Conda to version 24.1 in the latest 1.2 and 2.2 runtimes.
• Upgraded Spark BigQuery connector to version 0.36.1 in the latest 1.2 and 2.2 runtimes.

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS plugin to version 24.2.0 in the latest runtimes.
• Upgraded Spark to version 3.3.4 in the latest 1.1 and 2.0 runtimes.
• Backported SPARK-44198 in the latest 1.2 and 2.2 runtimes.

Added the bigframes (BigQuery DataFrames) Python package in the Dataproc Serverless for Spark 2.1 runtime.

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.20 version in the latest 1.1, 2.0, and 2.1 runtimes.

Dataproc on Compute Engine: Mounted Java cacerts into containers by default when the Docker-on-YARN feature is enabled.

Dataproc Serverless for Spark: Extended Spark metrics collected for a batch now include executor:resultSize, executor:shuffleBytesWritten, and executor:shuffleTotalBytesRead.

Dataproc on Compute Engine:

• Upgraded Zookeeper to 3.8.3 for Dataproc 2.2.
• Upgraded ORC for Hive to 1.15.13 for Dataproc 2.1.
• Upgraded ORC for Spark to 1.7.10 for Dataproc 2.1.
• Extended expiry for the internal Knox Gateway certificate from one year to five years from cluster creation for Dataproc images 2.0, 2.1, and 2.2.

Dataproc on Compute Engine: Backported patch for HADOOP-18652.

Dataproc Serverless for Spark: Backported patch for HADOOP-18652.

Dataproc on Compute Engine: Bucket ttl validation now also runs for buckets created by Dataproc.

Dataproc on Compute Engine: Added a warning during cluster creation if the cluster Cloud Storage staging bucket is using the legacy fine-grained/ACL IAM configuration instead of the recommended Uniform bucket-level access controls.

Dataproc Serverless for Spark: When dynamic allocation is enabled, the initial executor number is determined by max of spark.dynamicAllocation.initialExecutors and spark.executor.instances.

Dataproc on Compute Engine: Backported patches for HIVE-21214, HIVE-23154, HIVE-23354 and HIVE-23614.

Backport HIVE-19568: Active/Passive HiveServer2 HA: Disallow direct connection to passive instance.

Backport HIVE-27715: Remove ThreadPoolExecutorWithOomHook.

Dataproc on Compute Engine: The default yarn.nm.liveness-monitor.expiry-interval-ms Hadoop YARN setting has been changed in the latest image versions from 15000 (15 seconds) to 120000 (2 minutes).

Dataproc on Compute Engine: Upgraded Cloud Storage connector version to 2.2.19 in the latest 2.0 and 2.1 images.

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.19 version in the latest 1.1, 2.0, and 2.1 runtimes.

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS to version 23.12.1

• Upgraded the following components to the following versions in the latest 2.2 runtime:

  • Spark BigQuery connector version 0.35.0
  • Cloud Storage connector version 3.0.0
  • Conda version 23.11

Dataproc on Compute Engine: Changed the Hive Server2 and MetaStore maximum default JVM heap size to 32GiB. Previously, the limit was set to 1/4 of total node memory, which could be too large on large-memory machines.

Dataproc on Compute Engine: Backported the patch for YARN-10975 in the latest 2.0 images.

Added the google-cloud-secret-manager Python package in the latest Dataproc Serverless for Spark runtimes.

Monitoring-agent-defaults metrics are not available in Dataproc on Compute Engine image version 2.2 clusters unless the Ops Agent is installed. Other metrics for Dataproc provided components will continue to work.

Blocklisted the following Dataproc on Compute Engine Images due to issue with increase in startup time:

• 2.0.86-debian10, 2.0.86-rocky8, 2.0.86-ubuntu18
• 2.1.34-debian11, 2.1.34-rocky8, 2.1.34-ubuntu20, 2.1.34-ubuntu20-arm

Updated the Zookeeper component version from 3.8.0 to 3.8.3 in the latest Dataproc on Compute Engine 2.1 image version.

The Cloud Storage connector has been upgraded to version 2.2.18 in all Dataproc Serverless for Spark runtimes.

Upgraded the Cloud Storage connector version to 2.2.18 in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.

In the Flink component in the latest Dataproc on Compute Engine 2.1 image version, added the following java-storage client properties:

• gs.retry.max.attempts property to set the max number of retry attempts
• gs.retry.total.timeout property to set the total retry timeout

Increased the TTL for Dataproc on Compute Engine custom images from 60 days to 365 days.

Dataproc on Compute Engine: Properties dataproc:agent.ha.enabled and dataproc:componentgateway.ha.enabled now default to true to provide high availability for the Dataproc Agent and Component Gateway.

Upgraded Hadoop version from 3.3.3 to 3.3.6 in the latest Dataproc on Compute Engine 2.1 image version.

Upgraded the Cloud Storage connector version to 2.2.17 in the latest Dataproc Serverless for Spark runtimes.

Added the gs.http.connect-timeout and gs.http.read-timeout properties in Flink to set the connection timeout and read timeout for java-storage client in the latest Dataproc on Compute Engine 2.1 image version.

Added the gs.filesink.entropy.enabled property in Flink to enable entropy injection in filesink Cloud Storage path in the latest Dataproc on Compute Engine 2.1 image version.

Upgraded the Cloud Storage connector version to 2.2.17 in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.

Upgraded Hive version from 3.1.2 to 3.1.3 in the latest Dataproc on Compute Engine 2.0 image version.

In the latest Dataproc on Compute Engine 2.0 and 2.1 image versions, unset the CLOUDSDK_PYTHON variable to allow the gcloud command-line tool to use its bundled Python interpreter.

Scala has been upgraded to version 2.12.18 and Apache Tez has been upgraded to version 0.10.2 in Dataproc on Compute Engine 2.1 images.

The dataproc.diagnostics.enabled property is now avaiable to enable running diagnostics on Dataproc Serverless for Spark. The existing spark.dataproc.diagnostics.enabled property will be deprecated for use with newer runtimes.

The Apache Spark version has been upgraded from 3.3.0 to 3.3.2 in Dataproc on Compute Engine 2.1 images.

New Dataproc Serverless for Spark runtime versions:

• 1.1.29
• 2.0.37
• 2.1.16

Backported the patches for HIVE-20618 in the new Dataproc on Compute Engine 2.0 and 2.1 images.

The following previously released sub-minor versions of Dataproc on Compute Engine images unintentionally reverted several dependency library versions. This caused a risk of backward-incompatibility for some workloads.

These sub-minor versions have been rolled back, and can only be used when updating existing clusters that already use them:

• 2.0.71-debian10, 2.0.71-rocky8, 2.0.71-ubuntu18
• 2.1.19-debian11, 2.1.19-rocky8, 2.1.19-ubuntu20, 2.1.19-ubuntu20-arm

Upgraded Hudi to 0.12.3 and added the BigQuery Sync tool as part of the Hudi optional component.

Downgraded Cloud Storage connector version to 2.2.15 in all Dataproc on Compute Engine image versions to prevent potential performance regression.

Downgraded Cloud Storage connector to 2.2.15 version in all Dataproc Serverless for Spark runtimes to prevent potential performance regression.

New Dataproc on Compute Engine image versions:

• 2.0.71-debian10, 2.0.72-rocky8, 2.0.72-ubuntu18
• 2.1.19-debian11, 2.1.20-rocky8, 2.1.20-ubuntu20, 2.1.20-ubuntu20-arm

Note: The above image versions were rolled back. See the August 6, 2023 release note

Added recommendation details in Autoscaler Stackdriver logs for the CANCEL and DO_NOT_CANCEL recommendations.

Upgraded the Cloud Storage connector version to 2.2.16 in Dataproc Serverless for Spark runtimes.

Upgraded the Cloud Storage connector version to 2.2.16 for Dataproc on Compute Engine 2.0 and 2.1 images.

Backported ZEPPELIN-5755 to Zeppelin 0.10 in 2.1 images for Spark 3.3 support.

Upgrade Cloud Storage connector to 2.2.14 version in Dataproc Serverless for Spark runtimes.

Upgrade Cloud Storage connector version to 2.2.14 for 2.0 and 2.1 images

Backport HIVE-22891, HIVE-21660, HIVE-21915 to 2.0 images.

Backport HIVE-22891, HIVE-21660, HIVE-25520, HIVE-25521 to 2.1 images.

Upgraded the Cloud Storage connector to 2.2.13 version in Dataproc on Compute Engine 2.0 and 2.1 image versions.

Unauthorized callers attempting to get, delete, or terminate non-existent Sessions will now receive a 403 response code instead of a 404 response code. This does not impact authorized callers.

Upgraded the Cloud Storage connector to 2.2.13 version in Dataproc Serverless for Spark runtimes.

Upgraded Conda to 23.3 in Dataproc Serverless for Spark runtime 2.1.

Upgrade Spark to 3.4.0 and its dependencies in Dataproc Serverless for Spark 2.1 runtime:

• Jetty to 9.4.51.v20230217
• ORC to 1.8.3
• Parquet to 1.13.0
• Protobuf to 3.22.3

hive principal will be used for Hive catalog queries via presto in kerberos cluster.

Default batch TTL is set to 4 hours for Dataproc Serverless for Spark runtime version 2.1.

Add Autoscaler recommendation reasoning details in Cloud Logging.

Dataproc on GKE SLM force delete timeout exception converted to DataprocIoException.

Upgrade Hadoop to 3.3.5 in Dataproc Serverless for Spark runtimes

Dataproc on GKE now disallows update operations.

Dataproc on GKE diagnose operation now verifies that the master agent is running.

Upgrade Python to 3.11 and Conda to 23.1 in Dataproc Serverless for Spark runtime 2.1

New Dataproc Serverless for Spark runtime versions:

• 1.1.7
• 2.0.15
• 2.1.0-RC3

Upgraded Spark BigQuery connector version to 0.28.1 in 1.1 and 2.1 Dataproc Serverless for Spark runtimes.

Added stronger validations to disallow upper-case characters in template IDs per Resource Names guidance, which allows Workflow template creation to fail fast instead of failing at workflow template instantiation.

Added decision metric field in Stackdriver autoscaler logs.

Release Dataproc Serverless for Spark runtime 2.1 preview:

• Spark 3.4.0-rc1
• BigQuery Spark Connector 0.28.0
• Cloud Storage Connector 2.2.11
• Conda 22.11
• Java 17
• Python 3.10
• R 4.2
• Scala 2.13

Upgrade Spark to 3.3.2 and its dependencies in 1.1 and 2.0 Dataproc Serverless for Spark runtimes:

• Jackson to 2.13.5
• Jetty to 9.4.50.v20221201
• ORC to 1.8.2
• Protobuf to 3.21.12
• RoaringBitmap to 0.9.39

Generate a warning when initialization actions are used in a cluster created with a driver node group.

The default Dataproc Serverless for Spark runtime version has changed to 2.0.

Upgrade Cloud Storage connector to 2.2.11 version in Dataproc Serverless for Spark runtimes.

Dataproc Serverless for Spark runtime version 1.0 changed to non-LTS because of the release of backward-compatible Dataproc Serverless for Spark runtime version 1.1 LTS.

Upgraded Parquet to 1.12.2 for 2.1 images.

The value of hive.server2.builtin.udf.blacklist is now set by default to reflect,reflect2 in hive-site.xml to prevent arbitrary code execution.

Backported Spark patch in Dataproc Serverless for Spark runtime 1.0 and 2.0:

• SPARK-40481: Ignore stage fetch failure caused by decommissioned executor.

Upgrade Cloud Storage connector version to 2.1.9 for 1.5 images.

Upgrade Cloud Storage connector version to 2.2.9 for 2.1 images.

Dataproc Serverless for Spark runtime 1.0:

• Upgrade to Spark to 3.2.3
• Upgrade Cloud Storage connector to 2.2.9
• Upgrade Spark dependencies:
  • Jetty to 9.4.49.v20220914
  • ORC to 1.7.7
  • Protobuf to 3.19.6
  • RoaringBitmap to 0.9.35
  • Scala to 2.12.17

Dataproc Serverless for Spark runtime 2.0:

• Upgrade Cloud Storage connector to 2.2.9
• Upgrade Spark dependencies:
  • Protobuf to 3.21.9
  • RoaringBitmap to 0.9.35

Use jemalloc as a default OS memory allocator in Dataproc Serverless for Spark runtime.

Backport Spark patches in Dataproc Serverless for Spark runtime 1.0 and 2.0:

• SPARK-39324: Log ExecutorDecommission as INFO level in TaskSchedulerImpl
• SPARK-40168: Handle SparkException during shuffle block migration
• SPARK-40269: Randomize the orders of peer in BlockManagerDecommissioner
• SPARK-40778: Make HeartbeatReceiver as an IsolatedRpcEndpoint

Added support for Dataproc to attach to a gRPC Dataproc Metastore in any region.

Secure Boot, Virtual trusted platform module (vTPM), and Integrity monitoring Shielded VM features are enabled by default for Dataproc on Compute Engine clusters that use 2.1 preview images.

Nodemanagers in DECOMMISSIONING, NEW, and SHUTDOWN state are now included in the /cluster/yarn/nodemanagers metric.

Dataproc Serverless for Spark now shows the subminor runtime version used in the runtimeConfig.version field,

Removed the Auto Zone placement check for supported machine types.

Disabled auto deletion of files under /tmp in the latest Rocky images. Previous Rocky images have files in the /tmp folder deleted every 10 days due to default OS system setting in /usr/lib/tmpfiles.d/tmp.conf.

Changed Hive TokenStoreDelegationTokenSecretManager in the latest 1.5 and 2.0 images so that it updates the base class's current key ID after generating a new master key. This is important for users of DBTokenStore, which generates key IDs based on a monotonically increasing sequence from the database. Prior to this fix, there was a race condition during master key rollover that could cause it to attempt updating the prior master key using an incorrect ID value. This would fail and then quickly retry, sometimes multiple times, causing too many rows in the database.

Set yarn:spark.yarn.shuffle.stop_on_failure to true by default in the latest 1.5 and 2.0 images. This change causes YARN node manager startup to fail if the Spark external shuffle service startup fails. On VM boot, Dataproc will continuously restart the YARN node manager until it is able to start. This change reduces Spark executor errors, such as: org.apache.spark.SparkException: Unable to register with external shuffle server due to : Failed to connect to <worker host>:7337, particularly when starting a stopped cluster. See Spark external shuffle service documentation for details.

Dataproc Serverless for Spark now uses runtime version 1.0.19 and 2.0.0-RC4, which also upgrades both runtimes to Cloud Storage Connector to 2.2.8.

Dataproc Serverless for Spark: Reduced the latency between batch workload completion and when a batch is marked SUCCEEDED.

Dataproc Serverless for Spark: Increased initial and maximum Spark executor limits to 500 and 2,000, respectively.

Dataproc Serverless for Spark: Sets a maximum limit of 500 workers per scale up or scale down operation.

Dataproc on Compute Engine: Stop all master and worker VMs when starting a cluster fails due to stockout or insufficient quota.

Dataproc Serverless for Spark now uses runtime version 1.0.18 and 2.0.0-RC3.

Dataproc Serverless for Spark now uses runtime version 1.0.17 and 2.0.0-RC2.

Dataproc Serverless for Spark now uses runtime version 1.0.16, which upgrades the following components to the following versions:

• Spark 3.2.2
• Avro 1.11.1
• Hadoop 3.3.4
• Jetty 9.4.48.v20220622
• ORC 1.7.5
• RoaringBitmap 0.9.31
• Scala 2.12.16

New sub-minor versions of Dataproc images:

1.5.73-debian10, 1.5.73-rocky8, 1.5.73-ubuntu18

2.0.47-debian10, 2.0.47-rocky8, 2.0.47-ubuntu18

Enabled Spark authentication and encryption for Kerberos clusters in 1.5 and 2.0 images.

Dataproc Serverless for Spark now uses runtime version 1.0.15, which upgrades the following Spark dependencies to the following versions:

• Jackson 2.13.3
• Jetty 9.4.46.v20220331
• ORC 1.7.4
• Parquet 1.12.3
• Protobuf 3.19.4
• RoaringBitmap 0.9.28

Dataproc on Compute Engine images now have master VM memory protection enabled by default. Jobs may be terminated to prevent the master VM running out of memory.

New sub-minor versions of Dataproc images:

1.5.72-debian10, 1.5.72-rocky8, 1.5.72-ubuntu18

2.0.46-debian10, 2.0.46-rocky8, 2.0.46-ubuntu18

Upgraded Hadoop to version 3.2.3 in 2.0 images.

Upgraded Hadoop to version 2.10.2 version 2.10.2 in 1.5 images.

Default MySQL instance root password changed to a random value in 1.5 and 2.0 images. New password is now stored in MySQL configuration file accessible only by the OS level root user.

New sub-minor versions of Dataproc images:

1.5.71-debian10, 1.5.71-rocky8, 1.5.71-ubuntu18

2.0.45-debian10, 2.0.45-rocky8, 2.0.45-ubuntu18

New sub-minor versions of Dataproc images:

1.5.70-debian10, 1.5.70-rocky8, 1.5.70-ubuntu18

2.0.44-debian10, 2.0.44-rocky8, 2.0.44-ubuntu18

New sub-minor versions of Dataproc images:

1.5.69-debian10, 1.5.69-rocky8, 1.5.69-ubuntu18

2.0.43-debian10, 2.0.43-rocky8, 2.0.43-ubuntu18

Backported the patch for HBASE-23287 to HBase 1.5.0 in 1.5 image

Dataproc Serverless for Spark now uses runtime version 1.0.13.

New sub-minor versions of Dataproc images:

1.5.68-debian10, 1.5.68-rocky8, 1.5.68-ubuntu18

2.0.42-debian10, 2.0.42-rocky8, 2.0.42-ubuntu18

Dataproc on GKE Spark 3.1 images upgraded to Spark version 3.1.3.

Upgrade Cloud Storage connector version 2.1.8 for 1.5 images only.

New sub-minor versions of Dataproc images:

1.5.67-debian10, 1.5.67-ubuntu18, 1.5.67-rocky8

2.0.41-debian10, 2.0.41-ubuntu18, 2.0.41-rocky8

Dataproc on GKE error messages now provide additional information.

New sub-minor versions of Dataproc images:

1.5.66-debian10, 1.5.66-ubuntu18, 1.5.66-rocky8

2.0.40-debian10, 2.0.40-ubuntu18, 2.0.40-rocky8

Upgraded Spark to 3.1.3 in Dataproc image version 2.0.

New sub-minor versions of Dataproc images:

1.5.65-debian10, 1.5.65-ubuntu18, 1.5.65-rocky8

2.0.39-debian10, 2.0.39-ubuntu18, 2.0.39-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.12.

New sub-minor versions of Dataproc images:

1.5.64-debian10, 1.5.64-ubuntu18, 1.5.64-rocky8

2.0.38-debian10, 2.0.38-ubuntu18, 2.0.38-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.11.

If you request to cancel a job in one of the following states, Dataproc will return the job, but not initiate cancellation, since it is already in progress: CANCEL_PENDING, CANCEL_STARTED, or CANCELLED.

When submitting a Dataproc job or workflow that selects a cluster that matches the specified labels, Dataproc will avoid choosing clusters that are in a state that disallows running jobs. Specifically, Dataproc will only choose among clusters in one of the following states: RUNNING, UPDATING, CREATING, or ERROR_DUE_TO_UPDATE.

Added Dataproc Serverless support for updating the Cloud Storage connector using the dataproc.gcsConnector.version and dataproc.gcsConnector.uri properties.

New sub-minor versions of Dataproc images:

1.5.63-debian10, 1.5.63-ubuntu18, 1.5.63-rocky8

2.0.37-debian10, 2.0.37-ubuntu18, 2.0.37-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.10.

Cloud Storage connector version upgraded to 2.2.6 in image version 2.0.

New sub-minor versions of Dataproc images:

1.5.62-debian10, 1.5.62-ubuntu18, and 1.5.62-rocky8

2.0.36-debian10, 2.0.36-ubuntu18, and 2.0.36-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.9.

New sub-minor versions of Dataproc images:

1.5.61-debian10, 1.5.61-ubuntu18, and 1.5.61-rocky8

2.0.35-debian10, 2.0.35-ubuntu18, and 2.0.35-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.8.

New sub-minor versions of Dataproc images:

1.5.60-debian10, 1.5.60-ubuntu18, and 1.5.60-rocky8

2.0.34-debian10, 2.0.34-ubuntu18, and 2.0.34-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.7.

New sub-minor versions of Dataproc images:

1.5.59-debian10, 1.5.59-ubuntu18, and 1.5.59-rocky8

2.0.33-debian10, 2.0.33-ubuntu18, and 2.0.33-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.6.

New sub-minor versions of Dataproc images:

1.5.58-debian10, 1.5.58-ubuntu18, and 1.5.58-rocky8

2.0.32-debian10, 2.0.32-ubuntu18, and 2.0.32-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.5.

Dataproc Serverless for Spark now uses runtime version 1.0.4, which updates GCS connector to 2.2.5 version.

New sub-minor versions of Dataproc images:

1.5.57-debian10, 1.5.57-ubuntu18, and 1.5.57-rocky8

2.0.31-debian10, 2.0.31-ubuntu18, and 2.0.31-rocky8

Upgraded Cloud Storage connector version to 2.2.5 in image version 2.0.

Upgraded Cloud Storage connector version to 2.1.7 in image version 1.5.

Dataproc Serverless for Spark now uses runtime version 1.0.3.

New sub-minor versions of Dataproc images:

1.4.80-debian10 and 1.4.80-ubuntu18

1.5.56-debian10, 1.5.56-ubuntu18, and 1.5.56-centos8

2.0.30-debian10, 2.0.30-ubuntu18, and 2.0.30-centos8

Configured Zeppelin Spark interpreter to run in YARN client mode by default for image version 2.0.

Dataproc Serverless for Spark now uses runtime version 1.0.2, which updates Spark to 3.2.1 version.

Dataproc Serverless for Spark now uses runtime version 1.0.1, which includes improved error messaging for network connectivity issues.

Dataproc extracts the warehouse directory from the Dataproc Metastore service for the cluster-local warehouse directory.

New sub-minor versions of Dataproc images:

1.4.79-debian10 and 1.4.79-ubuntu18

1.5.55-debian10, 1.5.55-ubuntu18, and 1.5.55-centos8

2.0.29-debian10, 2.0.29-ubuntu18, and 2.0.29-centos8

Dataproc images 1.4.79, 1.5.55, and 2.0.29, listed above, were updated with log4j version 2.17.1. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.

The Cloud Storage connector jar is installed on the Solr server (even if dataproc:solr.gcs.path property is not set). Applies to image versions 1.4, 1.5, and 2.0.

Migrated to Eclipse Temurin JDK in image versions 1.4, 1.5, and 2.0.

New sub-minor versions of Dataproc images:

1.4.78-debian10, and 1.4.78-ubuntu18

1.5.54-centos8, 1.5.54-debian10, and 1.5.54-ubuntu18

2.0.28-centos8, 2.0.28-debian10, and 2.0.28-ubuntu18

Dataproc images 1.4.78, 1.5.54, and 2.0.28, listed above, were updated with log4j version 2.17.0. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.

Upgraded Cloud Storage connector version to 2.2.4 in image version 2.0.

Dataproc has released 1.3.95-debian10/-ubuntu18 images with a one-time patch that addresses the Apache Log4j 2 CVE-2021-44228 and CVE-2021-45046 vulnerabilities, but note that that all 1.3 images remain unsupported, and Dataproc will not provide upgrades to 1.3 images .

Dataproc has released the following sub-minor image versions to address an Apache Log4j 2 vulnerability (also see Create a cluster and Recreate and update a cluster for more information). Note: These images supersede the 1.5 and 2.0 images listed in the December 16, 2021 release note:

1.5.53-centos8, 1.5.53-debian10, 1.5.53-ubuntu18,

2.0.27-centos8, 2.0.27-debian10, 2.0.27-ubuntu18

Removed the Geode interpreter from Zeppelin notebook, which is affected by https://nvd.nist.gov/vuln/detail/CVE-2021-45046.

Dataproc has released the following [sub-minor image versions](https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions:

1.4.77-debian10, 1.4.77-ubuntu18,

1.5.52-centos8, 1.5.52-debian10, 1.5.52-ubuntu18,

2.0.26-centos8, 2.0.26-debian10, 2.0.26-ubuntu18

Upgraded log4j version to 2.16.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-44228.

New sub-minor versions of Dataproc images:

1.4.76-debian10, 1.4.76-ubuntu18,

1.5.51-centos8, 1.5.51-debian10, 1.5.51-ubuntu18,

2.0.25-centos8, 2.0.25-debian10, 2.0.25-ubuntu18

Upgraded log4j version to 2.15.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-45046.

New sub-minor versions of Dataproc images:

1.4.75-debian10, 1.4.75-ubuntu18,

1.5.50-centos8, 1.5.50-debian10, 1.5.50-ubuntu18,

2.0.24-centos8, 2.0.24-debian10, 2.0.24-ubuntu18

Removed Apache Iceberg and Delta Lake libraries in 2.0 images because they are not compatible with Spark 3.1.

Upgraded Cloud Storage connector to version 2.2.3 on 2.0 Images.

The dataproc:dataproc.cluster-ttl.consider-yarn-activity cluster property is now set to true by default for image versions 1.4.64+, 1.5.39+, and 2.0.13+. With this change, with clusters created with these image versions, Dataproc Cluster Scheduled Deletion by default will consider YARN activity, in addition to Dataproc Jobs API activity, when determining cluster idle time . This change does not affect clusters with images with lower version numbers: cluster idle time for those clusters will continue to be computed based on Dataproc Jobs API activity only. When using image versions 1.4.64+, 1.5.39+, and 2.0.13+, you can opt out of this changed behavior by setting this property to false when you create the cluster.

New sub-minor versions of Dataproc images:

1.4.73-debian10, 1.4.73-ubuntu18,

1.5.48-centos8, 1.5.48-debian10, 1.5.48-ubuntu18,

2.0.22-centos8, 2.0.22-debian10, 2.0.22-ubuntu18

New sub-minor versions of Dataproc images: 1.4.71-debian10, 1.4.71-ubuntu18, 1.5.46-centos8, 1.5.46-debian10, 1.5.46-ubuntu18, 2.0.20-centos8, 2.0.20-debian10, 2.0.20-ubuntu18

Added support for enabling/disabling Ubuntu Snap daemon with cluster property dataproc:dataproc.snap.enabled. The default value is "true". If set to "false", pre-installed Snap packages in the image won't be affected, but auto refresh will be disabled. Applies to all Ubuntu images.

The following previously released sub-minor versions of Dataproc images included a bug where the dataproc user account was broken. This prevented some Dataproc services from functioning properly, which resulted in features being unavailable. In particular, this prevented Jupyter from running in clusters with Personal Cluster Authentication enabled.

These sub-minor versions have been rolled back, and can only be used when updating existing clusters that already use them:

• 1.4.66-debian10, 1.4.66-ubuntu18
• 1.4.67-debian10, 1.4.67-ubuntu18
• 1.5.41-centos8, 1.5.41-debian10, 1.5.41-ubuntu18
• 1.5.42-centos8, 1.5.42-debian10, 1.5.42-ubuntu18
• 2.0.15-centos8, 2.0.15-debian10, 2.0.15-ubuntu18
• 2.0.16-centos8, 2.0.16-debian10, 2.0.16-ubuntu18

Added additional messages to the error messages for networking and IAM errors when creating a new cluster.

New sub-minor versions of Dataproc images: 1.4.70-debian10, 1.4.70-ubuntu18, 1.5.45-centos8, 1.5.45-debian10, 1.5.45-ubuntu18, 2.0.19-centos8, 2.0.19-debian10, 2.0.19-ubuntu18

Backported SPARK-34295: Added a new spark.yarn.kerberos.renewal.excludeHadoopFileSystemsconfiguration option.

Image 2.0:

OOZIE-3599: Upgraded Jetty version to 9.4.

New sub-minor versions of Dataproc images: 1.4.69-debian10, 1.4.69-ubuntu18, 1.5.44-centos8, 1.5.44-debian10, 1.5.44-ubuntu18, 2.0.18-centos8, 2.0.18-debian10, and 2.0.18-ubuntu18.

Configured YARN ResourceManager to use port 8554 and Druid to use port 17071 for JMX Remote RMI port.

Backported the following Trino (PrestoSQL) BigQuery connector patches in image 2.0:

• Make BigQuery views cache ttl configurable
• Move DestinationTableBuilder to BigQueryClient
• Fix wrong result due to column position mismatch in BigQuery
• Escape single quote in BigQuery string condition
• Fix information_schema query failures in BigQuery connector
• Add support for CREATE and DROP SCHEMA in BigQuery
• Fasten listing tables in BigQuery connector
• Implement case insensitive name matching for BigQuery
• Add BigQuerySqlExecutor and refactor tests to use it
• Add view_definition system table for BigQuery view
• Prefer ImmutableMap to Map
• Add test for aggregating BigQuery view more than once
• Upgrade Google Could library bom to 16.3.0
• Fix TestBigQueryIntegrationSmokeTest.testShowCreateTable
• Enable views in BigQuery tests
• Fix incorrect result when aggregating count BigQuery view
• Remove dependency on presto-tests from Druid, BigQuery
• Refactor unused var in presto-bigquery
• Fix projection pushdown in BigQuery connector
• Add test for yearly partitioned table in BigQuery
• Upgrade BigQuery library to 11.0.0
• Add BigQuery smoke test for HOUR-ly partitioned table
• Redact bigquery.credentials-key config
• Update BigQuery dependencies to support HOURLY partitioning of tables

Dataproc issues a warning message if the staging or test bucket name contains an underscore.

New sub-minor versions of Dataproc images: 1.4.68-debian10, 1.4.68-ubuntu18, 1.5.43-centos8, 1.5.43-debian10, 1.5.43-ubuntu18, 2.0.17-centos8, 2.0.17-debian10, and 2.0.17-ubuntu18.

Upgrade Flink to version 1.12.5 in image 2.0.

New sub-minor versions of Dataproc images: 1.4.67-debian10, 1.4.67-ubuntu18, 1.5.42-centos8, 1.5.42-debian10, 1.5.42-ubuntu18, 2.0.16-centos8, 2.0.16-debian10, and 2.0.16-ubuntu18.

New sub-minor versions of Dataproc images: 1.4.66-debian10, 1.4.66-ubuntu18, 1.5.41-centos8, 1.5.41-debian10, 1.5.41-ubuntu18, 2.0.15-centos8, 2.0.15-debian10, and 2.0.15-ubuntu18.

In image 2.0, set the mapreduce.fileoutputcommitter.algorithm.version=2 property in Spark. This makes Spark commit algorithm version consistent with prior Dataproc image versions.

New sub-minor versions of Dataproc images: 1.3.94-debian10, 1.3.94-ubuntu18, 1.4.65-debian10, 1.4.65-ubuntu18, 1.5.40-centos8, 1.5.40-debian10, 1.5.40-ubuntu18, 2.0.14-centos8, 2.0.14-debian10, and 2.0.14-ubuntu18.

The following component versions were updated in image 2.0:

• Upgraded Scala version to 2.12.14.
• Upgrades Spark version to 3.1.2.
• Upgraded Zeppelin version to 0.9.1-SNAPSHOT.

New sub-minor versions of Dataproc images: 1.3.93-debian10, 1.3.93-ubuntu18, 1.4.64-debian10, 1.4.64-ubuntu18, 1.5.39-centos8, 1.5.39-debian10, 1.5.39-ubuntu18, 2.0.13-centos8, 2.0.13-debian10, and 2.0.13-ubuntu18.

Upgraded Cloud Storage connector to version 2.2.2 on 2.0 images.

For 2.0+ image clusters, the dataproc:dataproc.master.custom.init.actions.mode cluster property can be set to RUN_AFTER_SERVICES to run initialization actions on the master after HDFS and any services that depend on HDFS are initialized. Examples of HDFS-dependent services include: HBase, Hive Server2, Ranger, Solr, and the Spark and MapReduce history servers. Default: RUN_BEFORE_SERVICES.

Custom image limitation: New images announced in the Dataproc release notes are not available for use as the base for custom images until one week from their announcement date.

New sub-minor versions of Dataproc images: 1.3.92-debian10, 1.3.92-ubuntu18, 1.4.63-debian10, 1.4.63-ubuntu18, 1.5.38-centos8, 1.5.38-debian10, 1.5.38-ubuntu18, 2.0.12-centos8, 2.0.12-debian10, and 2.0.12-ubuntu18.

Upgraded Spark version to 2.4.8 in the following images:

• Image 1.4
• Image 1.5

Upgrade HBase to version 2.2.7 in image 2.0.

Minimum boot disk sizes for Dataproc images:

• Image 2.0: 30GB
• Image 1.5: 20GB
• Image 1.4: 15GB
• Image 1.3: 15GB

The following previously released sub-minor versions of Dataproc images have been rolled back and can only be used when updating existing clusters that already use them:

• 1.3.91-debian10, 1.3.91-ubuntu18
• 1.4.62-debian10, 1.4.62-ubuntu18
• 1.5.37-centos8, 1.5.37-debian10, 1.5.37-ubuntu18
• 2.0.11-centos8, 2.0.11-debian10, and 2.0.11-ubuntu18.

Added support for Dataproc Metastore in three new recently turned up regions: europe-west3, us-west1, and us-east1.

Introduced a new ERROR_DUE_TO_UPDATE state, which indicates a cluster has encountered an irrecoverable error while scaling. Clusters in this state cannot be scaled, but can accept jobs.

Dataproc Component Gateway URLs for any two new clusters that have the same project ID, region, and name will be identical unless Dataproc Personal Cluster Authentication is enabled.

Custom image limitation: Currently, the following Dataproc image versions are the latest images that can be used as the base for custom images:

• 1.3.89-debian10, 1.3.89-ubuntu18
• 1.4.60-debian10, 1.4.60-ubuntu18
• 1.5.35-debian10, 1.5.35-ubuntu18, 1.5.35-centos8
• 2.0.9-debian10, 2.0.9-ubuntu18, 2.0.11-centos8

New sub-minor versions of Dataproc images: 1.3.91-debian10, 1.3.91-ubuntu18, 1.4.62-debian10, 1.4.62-ubuntu18, 1.5.37-centos8, 1.5.37-debian10, 1.5.37-ubuntu18, 2.0.11-centos8, 2.0.11-debian10, and 2.0.11-ubuntu18.

• Rollback Notice: See the June 29, 2021 release note rollback notice.

Image 1.3 - 2.0

• All jobs now share a single JobthreadPool.

• The number of Job threads in the Agent is configurable with the dataproc:agent.process.threads.job.min and dataproc:agent.process.threads.job.max cluster properties, defaulting to 10 and 100, respectively. The previous behavior was to always use 10 Job threads.

Image 2.0

• Added snappy-jar dependency to Hadoop.
• Upgraded versions of Python packages: nbdime 2.1 -> 3.0, pyarrow 2.0 -> 3.0, spyder 4.2 -> 5.0, spyder-kernels 1.10 -> 2.0, regex 2020.11 -> 2021.4.

Added validation for clusters created with Dataproc Metastore services to determine compatibility between the Dataproc image's Hive version and the DPMS service's hive version

New sub-minor versions of Dataproc images: 1.3.89-debian10, 1.3.89-ubuntu18, 1.4.60-debian10, 1.4.60-ubuntu18, 1.5.35-centos8, 1.5.35-debian10, 1.5.35-ubuntu18, 2.0.9-centos8, 2.0.9-debian10, and 2.0.9-ubuntu18.

Image 1.4

• Updated Conda to version 4.6.

Image 1.5

• CentOS only: adoptopenjdk is set as the default Java environment.

Image 1.5 and 2.0

• Updated Oozie version to 5.2.1
• The Jupyter optional component now uses the "GCS" subdirectory as the initial working directory when you open the JupyterLab UI.

Added the ability to stop and start high-availability clusters.

Image 2.0:

• New sub-minor versions of Dataproc images: 2.0.8-centos8, 2.0.8-debian10, and 2.0.8-ubuntu18.

• Added HBase Spark Connector.

New sub-minor versions of Dataproc images: 1.3.88-debian10, 1.3.88-ubuntu18, 1.4.59-debian10, 1.4.59-ubuntu18, 1.5.34-centos8, 1.5.34-debian10, 1.5.34-ubuntu18, 2.0.7-centos8, 2.0.7-debian10, and 2.0.7-ubuntu18.

Image 2.0:

• Updated Iceberg to version 0.11.0.
• Updated Flink to version 1.12.2.

New sub-minor versions of Dataproc images: 1.3.87-debian10, 1.3.87-ubuntu18, 1.4.58-debian10, 1.4.58-ubuntu18, 1.5.33-centos8, 1.5.33-debian10, 1.5.33-ubuntu18, 2.0.6-centos8, 2.0.6-debian10, and 2.0.6-ubuntu18.

Image 2.0: Upgraded Spark to version 3.1.1

New sub-minor versions of Dataproc images: 1.3.86-debian10, 1.3.86-ubuntu18, 1.4.57-debian10, 1.4.57-ubuntu18, 1.5.32-centos8, 1.5.32-debian10, 1.5.32-ubuntu18, 2.0.5-debian10, and 2.0.5-ubuntu18

Image 2.0:

• Upgraded Delta Lake to 0.8.0 version.

New sub-minor versions of Dataproc images: 1.3.85-debian10, 1.3.85-ubuntu18, 1.4.56-debian10, 1.4.56-ubuntu18, 1.5.31-centos8, 1.5.31-debian10, 1.5.31-ubuntu18, 2.0.4-debian10, and 2.0.4-ubuntu18

Image 2.0: Upgraded Spark to 3.1.1 RC2 version

Allow stopping clusters that have autoscaling enabled, and allow enabling autoscaling on clusters that are STOPPED, STOPPING, or STARTING. If you stop a cluster that has autoscaling enabled, the Dataproc autoscaler will stop scaling the cluster. It will resume scaling the cluster once it has been started again. If you enable autoscaling on a stopped cluster, the autoscaling policy will only take effect once the cluster has been started (see Starting and stopping clusters).

Deactivated mysql and hive-metastore components for clusters created with a Dataproc Metastore service on an image that has the DISABLE_COMPONENT_HIVE_METASTORE and DISABLE_COMPONENT_MYSQL capabilities.

New sub-minor versions of Dataproc images: 1.3.84-debian10, 1.3.84-ubuntu18, 1.4.55-debian10, 1.4.55-ubuntu18, 1.5.30-centos8, 1.5.30-debian10, 1.5.30-ubuntu18, 2.0.3-debian10, and 2.0.3-ubuntu18

New sub-minor versions of Dataproc images: 2.0.2-debian10, and 2.0.2-ubuntu18.

Image 2.0:

• Upgraded Spark built-in Hive to version 2.3.8.
• Upgraded Druid to version 0.20.1
• HIVE-24436: Fixed Avro NULL_DEFAULT_VALUE compatibility issue.
• SQOOP-3485: Fixed Avro NULL_DEFAULT_VALUE compatibility issue.
• SQOOP-3447: Removed usage of org.codehaus.jackson and org.json packages.

New sub-minor versions of Dataproc images: 1.3.83-debian10, 1.3.83-ubuntu18, 1.4.54-debian10, 1.4.54-ubuntu18, 1.5.29-centos8, 1.5.29-debian10, 1.5.29-ubuntu18, 2.0.1-debian10, and 2.0.1-ubuntu18.

Image 2.0:

• Upgraded Hadoop to version 3.2.2

• Upgraded Scala to version 2.12.13

• Upgraded Flink to version 1.12.1

• Upgraded Spark to version 3.1.1 RC1

New sub-minor versions of Dataproc images: 1.3.82-debian10, 1.3.82-ubuntu18, 1.4.53-debian10, 1.4.53-ubuntu18, 1.5.28-centos8, 1.5.28-debian10, 1.5.28-ubuntu18, 2.0.0-debian10, and 2.0.0-ubuntu18.

New sub-minor versions of Dataproc images: 1.3.81-debian10, 1.3.81-ubuntu18, 1.4.52-debian10, 1.4.52-ubuntu18, 1.5.27-centos8, 1.5.27-debian10, 1.5.27-ubuntu18, 2.0.0-RC23-debian10, and 2.0.0-RC23-ubuntu18.

Image 2.0 preview:

• Upgraded Spark to version 3.1.0 RC1.

• Upgraded Zeppelin to version 0.9.0.

• Upgraded Cloud Storage Connector to version 2.2.0.

• Upgraded JupyterLab to version 3.0.

New sub-minor versions of Dataproc images: 1.3.80-debian10, 1.3.80-ubuntu18, 1.4.51-debian10, 1.4.51-ubuntu18, 1.5.26-centos8, 1.5.26-debian10, 1.5.26-ubuntu18, 2.0.0-RC22-debian10, and 2.0.0-RC22-ubuntu18.

Image 2.0 preview:

• Upgraded Delta Hive connector to version 0.2.0.
• Upgraded Flink to version 1.12.0.
• Updated Iceberg to version 0.10.0.

New sub-minor versions of Dataproc images: 1.3.79-debian10, 1.3.79-ubuntu18, 1.4.50-debian10, 1.4.50-ubuntu18, 1.5.25-centos8, 1.5.25-debian10, 1.5.25-ubuntu18, 2.0.0-RC21-debian10, and 2.0.0-RC21-ubuntu18.

Image 2.0 preview:

Changed the default value of Spark SQL property spark.sql.autoBroadcastJoinThreshold to 0.75% of executor memory.

Fixed SPARK-32436: Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

New sub-minor versions of Dataproc images: 1.3.78-debian10, 1.3.78-ubuntu18, 1.4.49-debian10, 1.4.49-ubuntu18, 1.5.24-debian10, 1.5.24-ubuntu18, 2.0.0-RC20-debian10, and 2.0.0-RC20-ubuntu18.

Image 1.5:

• Upgraded Hadoop to version 2.10.1.

New sub-minor versions of Dataproc images: 1.3.77-debian10, 1.3.77-ubuntu18, 1.4.48-debian10, 1.4.48-ubuntu18, 1.5.23-debian10, 1.5.23-ubuntu18, 2.0.0-RC19-debian10, and 2.0.0-RC19-ubuntu18.

Image 2.0 preview

• Upgraded Hue to version 4.8.0

New sub-minor versions of Dataproc images: 1.3.76-debian10, 1.3.76-ubuntu18, 1.4.47-debian10, 1.4.47-ubuntu18, 1.5.22-debian10, 1.5.22-ubuntu18, 2.0.0-RC18-debian10, and 2.0.0-RC18-ubuntu18.

Image 2.0 preview

• Upgraded Miniconda to version 4.9.0
• Upgraded Druid to version 0.20.0
• Upgraded Upgrade FasterXML Jackson to version 2.10.5
• Upgraded Apache ORC to version 1.5.12
• Upgraded Apache Parquet to version 1.11.1

New sub-minor versions of Dataproc images: 1.3.75-debian10, 1.3.75-ubuntu18, 1.4.46-debian10, 1.4.46-ubuntu18, 1.5.21-debian10, 1.5.21-ubuntu18, 2.0.0-RC17-debian10, and 2.0.0-RC17-ubuntu18.

Decreased the minimum allowed value of Dataproc Scheduled Deletion LifecycleConfig.idleDeleteTtl (Dataproc API) and --max-idle flag (gcloud command-line tool) from 10 minutes to 5 minutes.

New sub-minor versions of Dataproc images: 1.3.74-debian10, 1.3.74-ubuntu18, 1.4.45-debian10, 1.4.45-ubuntu18, 1.5.20-debian10, 1.5.20-ubuntu18, 2.0.0-RC16-debian10, and 2.0.0-RC16-ubuntu18.

2.0 preview image versions:

• Upgraded Apache Arrow version used by Spark to Apache Arrow 1.0.1.

New sub-minor versions of Dataproc images: 1.3.72-debian10, 1.3.72-ubuntu18, 1.4.43-debian10, 1.4.43-ubuntu18, 1.5.18-debian10, 1.5.18-ubuntu18, 2.0.0-RC14-debian10, and 2.0.0-RC14-ubuntu18.

New sub-minor versions of Dataproc images: 1.3.71-debian10, 1.3.71-ubuntu18, 1.4.42-debian10, 1.4.42-ubuntu18, 1.5.17-debian10, 1.5.17-ubuntu18, 2.0.0-RC13-debian10, and 2.0.0-RC13-ubuntu18.

Image 1.4

• Upgraded Spark to version 2.4.7.

Image 1.5

• Upgraded Spark to version 2.4.7.
• Installed google-cloud-bigquery-storage package by default in the Anaconda component.
• Changed default value of zeppelin.notebook.storage in zeppelin-site.xml to "org.apache.zeppelin.notebook.repo.GCSNotebookRepo".

Image 2.0

• Updated HBase to version 2.2.6.
• Installed google-cloud-bigquery-storage in default conda environment.
• Changed default value of zeppelin.notebook.storage in zeppelin-site.xml to "org.apache.zeppelin.notebook.repo.GCSNotebookRepo".

Creating clusters and instantiating workflow requests that succeed even when the requester did not have ActAs permission on the service account now generate a warning field in the audit log request.

New sub-minor versions of Dataproc images: 1.3.70-debian10, 1.3.70-ubuntu18, 1.4.41-debian10, 1.4.41-ubuntu18, 1.5.16-debian10, 1.5.16-ubuntu18, 2.0.0-RC12-debian10, and 2.0.0-RC12-ubuntu18.

All supported images

Upgraded Conscrypt to the 2.5.1 version.

Image 1.5

• Upgraded Delta Lake to the 0.6.1 version.

Image 1.5 and Image 2.0 Preview

• Upgraded Cloud Storage connector to the 2.1.5 version.

Image 2.0 preview

• The Anaconda optional component is no longer available or required when using the Jupyter optional component (Miniconda is installed and activated by default).

• Updated R to the 4.0.0 version.

• Configured YARN aggregated logs to use the IFile format.

• Upgrade Flink to the 1.11.2 version.

New sub-minor versions of Dataproc images: 1.3.69-debian10, 1.3.69-ubuntu18, 1.4.40-debian10, 1.4.40-ubuntu18, 1.5.15-debian10, 1.5.15-ubuntu18, 2.0.0-RC11-debian10, and 2.0.0-RC11-ubuntu18.

All image versions

• Upgrade Conscrypt to version 2.5.0

Image 2.0 preview

• Enable outbound private IPv6 Google APIs access from cluster nodes by default.
• Upgrade Miniconda to version 4.8.3
• Upgrade Python to version 3.8
• Upgrade Delta Lake to version 0.7.0
• Upgrade Iceberg to version 0.9.1
• Upgrade Spark to version 3.0.1
• Backport HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting.

Added the PrivateIpv6GoogleAccess API field to allow configuring IPv6 access to Dataproc cluster.

New sub-minor versions of Dataproc images: 1.3.68-debian10, 1.3.68-ubuntu18, 1.4.39-debian10, 1.4.39-ubuntu18, 1.5.14-debian10, 1.5.14-ubuntu18, 2.0.0-RC10-debian10, and 2.0.0-RC10-ubuntu18.

1.3-1.5 Images:

HIVE-18323: Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet

1.5 and 2.0 preview images:

Upgraded the jupyter-core and jupyter-client packages in the 1.5 and 2.0 images to be compatible with the installed notebook package version.

2.0 preview image:

• HIVE-21085: Materialized views registry starts non-external Tez session.
• HIVE-22647: Enable session pool by default.
• HDFS-14759: HDFS cat logs an info message.
• Upgraded Scala to 2.12.12 version.
• Upgraded Druid to 0.19.0 version.
• Upgraded Flink to 1.11.1 version.
• Upgraded Hue to 4.7.1 version
• Upgraded Presto SQL to 340 version.

Switched 1.3 and 1.3-debian image version aliases to point to 1.3 Debian 10 images.

When Enhanced Flexibility Mode is enabled, increased app master, task, and Spark stage retries to 10 to improve resiliency of applications to downscaling and preemption of preemptible VMs.

Support more than 8 local SSDs on VMs. Compute Engine supports 16 and 24 SSDs for larger machine types.

Changed secondary workers default boot disk size to 1000 GB in clusters created with 2.0 preview images.

Improved node memory utilization in clusters created with 2.0 preview images.

New sub-minor versions of Dataproc images: 1.3.67-debian10, 1.3.67-ubuntu18, 1.4.38-debian10, 1.4.38-ubuntu18, 1.5.13-debian10, 1.5.13-ubuntu18, 2.0.0-RC9-debian10, and 2.0.0-RC9-ubuntu18.

Image 1.3 and 1.4: upgraded the Cloud Storage connector to version to 1.9.18.

Changed 1.4 and 1.4-debian image version aliases to point to 1.4-debian10. The version name 1.4-debian9 will continue to be available, but it won't get updates in future releases.

Dataproc quotas are now regional: each region now has its own quota, which can be separately adjusted. All existing quota overrides have been migrated; customer action is not required.

Enabled Spark SQL parquet metadata cache (removed spark.sql.parquet.cacheMetadata=false from Spark default configuration).

New sub minor versions of Dataproc images: 1.3.66-debian10, 1.3.66-ubuntu18, 1.4.37-debian10, 1.4.37-ubuntu18, 1.5.12-debian10, 1.5.12-ubuntu18, 2.0.0-RC8-debian10, and 2.0.0-RC8-ubuntu18.

Image 1.4:

• Fixed a bug in Spark EFM HCFS shuffle where failures after partial commits are not recoverable.
• Upgraded Spark to 2.4.6 version.

Image 1.5:

• Fixed a bug in Spark EFM HCFS shuffle where failures after partial commits are not recoverable.
• Upgraded Spark to 2.4.6 version.
• Upgraded Zeppelin to 0.9.0-preview2 version.
• Included all plugins in Zeppelin installation by default.

Image 2.0 preview:

• Upgraded Zeppelin to 0.9.0-preview2 version.
• Included all plugins in Zeppelin installation by default.
• Upgraded HBase to 2.2.5 version.

Dataproc users are required to have service account ActAs permission to deploy Dataproc resources, for example, to create clusters and submit jobs. See Managing service account impersonation for more information.

Opt-in for existing Dataproc customers: This change does not automatically apply to current Dataproc customers without ActAs permission. To opt in, see Securing Dataproc, Dataflow, and Cloud Data Fusion.

New sub-minor versions of Dataproc images: 1.3.65-debian10, 1.3.65-ubuntu18, 1.4.36-debian10, 1.4.36-ubuntu18, 1.5.11-debian10, 1.5.11-ubuntu18, 2.0.0-RC7-debian10, and 2.0.0-RC7-ubuntu18.

1.3+ images (includes Preview image):

• HADOOP-16984: Added support to read history files only from the done directory.

• MAPREDUCE-7279: Display the Resource Manager name on the HistoryServer web page.

• SPARK-32135: Show the Spark driver name on the Spark history web page.

• SPARK-32097: Allow reading Spark history log files via the Spark history server from multiple directories.

Images 1.3 - 1.5:

• HIVE-20600: Fixed Hive Metastore connection leak.

Images 1.5 - 2.0 preview:

• Upgraded the Cloud Storage connector to version 2.1.4 (see the GitHub change notes).

Upgraded the jupyter-gcs-contents-manager package to the latest version. This upgrade includes a bug fix to a 404 (NOT FOUND) error message that was issued in response to an attempt to create a file in the virtual top-level directory instead of the expected 403 (PERMISSION DENIED) error message.

New sub-minor versions of Dataproc images: 1.3.64-debian10, 1.3.64-ubuntu18, 1.4.35-debian10, 1.4.35-ubuntu18, 1.5.10-debian10, 1.5.10-ubuntu18, 2.0.0-RC6-debian10, and 2.0.0-RC6-ubuntu18.

New sub-minor versions of Dataproc images: 1.3.63-debian10, 1.3.63-ubuntu18, 1.4.34-debian10, 1.4.34-ubuntu18, 1.5.9-debian10, 1.5.9-ubuntu18, 2.0.0-RC5-debian10, and 2.0.0-RC5-ubuntu18.

Image 2.0 preview:

• Updated Pig 0.18.0-SNAPHOT to 0b2066a revision to fix Hive 3.1 compatibility (PIG-4764).

New sub-minor versions of Dataproc images: 1.2.102-debian9, 1.3.62-debian9, 1.4.33-debian9, 1.3.62-debian10, 1.4.33-debian10, 1.5.8-debian10, 1.3.62-ubuntu18, 1.4.33-ubuntu18, 1.5.8-ubuntu18, 2.0.0-RC4-debian10, 2.0.0-RC4-ubuntu18

New subminor image versions: 1.2.100-debian9, 1.3.60-debian9, 1.4.31-debian9, 1.3.60-debian10, 1.4.31-debian10, 1.5.6-debian10, 1.3.60-ubuntu18, 1.4.31-ubuntu18, 1.5.6-ubuntu18, preview 2.0.0-RC2-debian10, and preview 2.0.0-RC2-ubuntu18.

• Image 2.0 preview:

  • SPARK-22404: set spark.yarn.unmanagedAM.enabled property to true on clusters where Kerberos is not enabled to run Spark Application Master in driver (not managed in YARN) to improve job execution time.

  • Updated R version to 3.6

  • Updated Spark to 3.0.0 version

• Image 1.5

  • Updated R version to 3.6

• New subminor image versions: 1.2.99-debian9, 1.3.59-debian9, 1.4.30-debian9, 1.3.59-debian10, 1.4.30-debian10, 1.5.5-debian10, 1.3.59-ubuntu18, 1.4.30-ubuntu18, and 1.5.5-ubuntu18.

• New preview image 2.0.0-RC1-debian10, 2.0.0-RC1-ubuntu18, with the following components:

  • Anaconda 2019.10
  • Atlas 2.0.0
  • Druid 0.18.1
  • Flink 1.10.1
  • Hadoop 3.2.1
  • HBase 2.2.4
  • Hive 3.1.2 (with LLAP support)
  • Hue 4.7.0
  • JupyterLab 2.1.0
  • Kafka 2.3.1
  • Miniconda3 4.8.3
  • Pig 0.18.0
  • Presto SQL 333
  • Oozie 5.2.0
  • R 3.6.0
  • Ranger 2.0.0
  • Solr 8.1.1
  • Spark 3.0.0
  • Sqoop 1.5.0
  • Zeppelin 0.9.0

• Image 1.3+

  • Patched HIVE-23496 Adding a flag to disable materialized views cache warm up.

Druid's Historical's and Broker's JVM and runtime properties are now calculated using server resources. Previously, only the Historical's and MiddleManager's MaxHeapSize property was calculated using server resources. This change modifies how new values for MaxHeapSize and MaxDirectMemorySize properties are calculated for Broker and Historical processes. Also, new runtime properties druid.processing.numThreads and druid.processing.numMergeBuffers are calculated using server resources.

If the project-level staging bucket is manually deleted, it will be recreated when a cluster is created.

Dataproc Job container logging now supports Dataproc Kerberized clusters.

New sub-minor versions of Dataproc images: 1.2.98-debian9, 1.3.58-debian9, 1.4.29-debian9, 1.3.58-debian10, 1.4.29-debian10, 1.5.4-debian10, 1.3.58-ubuntu18, 1.4.29-ubuntu18, 1.5.4-ubuntu18.

Image 1.3, 1.4, and 1.5

• Restrict Jupyter, Zeppelin, and Knox to only accept connections from localhost when Component Gateway is enabled. This restriction reduces the risk of remote code execution over unsecured notebook server APIs. To override this change, when you create the cluster, set the Jupyter, Zeppelin, and Knox cluster properties, respectively, as follows: dataproc:jupyter.listen.all.interfaces=true, zeppelin:zeppelin.server.addr=0.0.0.0, and knox:gateway.host=0.0.0.0.

• Upgrade Hive to version 2.3.7.

Image 1.4 and 1.5

SPARK-29367: Add ARROW_PRE_0_15_IPC_FORMAT=1 in yarn-env.sh to fix the Pandas UDF issue with pyarrow 0.15.

Image 1.5

• Upgrade Druid to version 0.17.1.

• Upgrade Cloud Storage Connector to version 2.1.3.

• Upgrade Delta Lake to version 0.6.0.

Hide the "Quit" button from Jupyter notebook (c.NotebookApp.quit_button = False) when using the Jupyter optional component. The Jupyter environment is shut down when the cluster is deleted.

Set the hive.localize.resource.num.wait.attempts property to 25 to improve reliability of Hive queries.

New sub-minor image versions: 1.2.96-debian9, 1.3.56-debian9, 1.4.27-debian9, 1.3.56-debian10, 1.4.27-debian10, 1.5.2-debian10, 1.3.56-ubuntu18, 1.4.27-ubuntu18, 1.5.2-ubuntu18

Image 1.5

Cloud Storage connector upgraded to version 2.1.2 (for more information, review the change notes in the GitHub repository)

Dataproc is now available in the us-west4 region (Las Vegas).

New sub-minor versions of Dataproc images: 1.2.95-debian9, 1.3.55-debian9, 1.4.26-debian9, 1.3.55-debian10, 1.4.26-debian10, 1.5.1-debian10, 1.3.55-ubuntu18, 1.4.26-ubuntu18, 1.5.1-ubuntu18.

Image 1.5

Updated Presto to version 331.

Created cloud-sql-proxy log file for the Cloud SQL Proxy initialization action and for Dataproc clusters with Ranger that use Cloud SQL Proxy.

Image 1.3 and 1.4

Debian 10 images will become default images for 1.3 and 1.4 image tracks and Debian 9 images will not be released for these tracks anymore after June 30, 2020.

New sub-minor versions of Dataproc images: 1.2.94-debian9, 1.3.54-debian9, 1.4.25-debian9, 1.5.0-debian10, 1.3.54-ubuntu18, 1.4.25-ubuntu18, and 1.5.0-ubuntu18

Image 1.5
Upgraded the Cloud Storage connector to version 2.1.1.

Images 1.2 and 1.4

Dataproc 1.4 will be the default image version after April 31, 2020.

Dataproc 1.2 will have no further releases after June 30, 2020.

Added the following flag to the gcloud dataproc clusters update command:

• --num-secondary-workers

New sub-minor versions of Dataproc images: 1.2.93-debian9, 1.3.53-debian9, 1.4.24-debian9, 1.5.0-RC9-debian10, 1.3.53-ubuntu18, 1.4.24-ubuntu18, and 1.5.0-RC9-ubuntu18

Image 1.2, 1.3, 1.4

Upgraded HBase to 1.3.6

Image 1.4, 1.5 preview

Added ARROW_PRE_0_15_IPC_FORMAT=1 for spark-env for pyarrow upgrade from 0.13 to 0.15.

Image 1.5 preview

• Preinstalled additional Python packages and Jupyter[Lab] extensions to align Jupyter notebook environment with AI Platforms Notebooks when Jupyter optional component is enabled.

• Upgraded Druid version to 0.17.0

Normalized custom image URLs to a full URL, for example: https://www.googleapis.com/compute/v1/projects/foo-project/global/images/foo-image. The cluster creation API will continue to accept relative names, such as projects/foo-project/global/images/foo-image or foo-image (see Dataproc API doesn't return imageUri in a consistent format).

Cluster list methods now return results in lexical order.

Added the following flags to gcloud dataproc clusters create and gcloud dataproc workflow-templates set-managed-cluster commands:

• --num-secondary-workers
• --num-secondary-worker-local-ssds
• --secondary-worker-boot-disk-size
• --secondary-worker-boot-disk-type
• --secondary-worker-accelerator

New sub-minor versions of Dataproc images: 1.2.92-debian9, 1.3.52-debian9, 1.4.23-debian9, 1.5.0-RC8-debian10, 1.3.52-ubuntu18, 1.4.23-ubuntu18, 1.5.0-RC8-ubuntu18

1.5 preview image:

Upgraded the Cloud Storage connector to version 2.0.1.

In addition to the staging bucket, Dataproc now creates a temporary bucket for storing feature-related data with a 90-day retention period per project per region. The bucket name will be of the following form: dataproc-temp-<REGION>-<PROJECT_NUMBER>-<RANDOM_STRING>.

Changed the staging bucket naming scheme for new buckets from dataproc-<UUID>-<REGION> to dataproc-staging-<REGION>-<PROJECT_NUMBER>-<RANDOM_STRING>.

New sub-minor versions of Dataproc images: 1.2.91-debian9, 1.3.51-debian9, 1.4.22-debian9, 1.5.0-RC7-debian10, 1.3.51-ubuntu18, 1.4.22-ubuntu18, 1.5.0-RC7-ubuntu18

Add OS-family image version aliases: 1.0-debian, 1.1-debian, 1.2-debian, 1.3-debian, 1.3-ubuntu, 1.4-debian, 1.4-ubuntu, preview-debian, preview-ubuntu. Users can now reference these aliases: Example:

gcloud dataproc clusters create $CLUSTER_NAME --image-version=1.4-debian

Increase Dataproc agent buffer size for streaming driver output to Cloud Storage from 60KB to 8MB. This significantly increases streaming throughput: from ~200KBps to ~7MBps. However, it also increases the memory footprint for each job in the Dataproc agent from ~100KB to ~10MB.

Added support for parameterizing numInstances field in Workflow Templates.

Images 1.3, 1.4, & 1.5:

Backport HIVE-16958.

Image 1.2, 1.3, & 1.4:

Upgrade Oozie to 4.3.1 and backport OOZIE-3112.

Image 1.3:

• Added BigQuery magics for Python2 and PySpark kernels to Dataproc Jupyter notebooks. You can now use the BigQuery Python client library and Pandas in a Jupyter notebook to visualize data in the BigQuery table.The BigQuery client library provides a %%bigquery cell magic, which runs a SQL query and returns the results as a Pandas DataFrame (see Visualizing BigQuery data in a Jupyter notebook https://cloud.google.com/bigquery/docs/visualize-jupyter ). Example:

  %%bigquer SELECT * FROM bigquery-public-data.samples.natality LIMIT 1

• Upgrade Zeppelin to 0.8.1 version

Image 1.4 & 1.5:

Added BigQuery magics for Python3 and PySpark kernels to Dataproc Jupyter notebooks. You can now use the BigQuery Python client library and Pandas in a Jupyter notebook to visualize data in the BigQuery table.The BigQuery client library provides a %%bigquery cell magic, which runs a SQL query and returns the results as a Pandas DataFrame (see Visualizing BigQuery data in a Jupyter notebook https://cloud.google.com/bigquery/docs/visualize-jupyter ). Example:

%%bigquer SELECT * FROM `bigquery-public-data.samples.natality` LIMIT 1

Images 1.4 & 1.5 :

Upgrade Spark to 2.4.5 version

New sub-minor versions of Dataproc images: 1.2.90-debian9, 1.3.50-debian9, 1.4.21-debian9, 1.5.0-RC6-debian10, 1.3.50-ubuntu18, 1.4.21-ubuntu18, 1.5.0-RC6-ubuntu18.

1.5 preview image:

Add BigQuery connector to Presto SQL

New sub-minor versions of Dataproc images: 1.2.89-debian9, 1.3.49-debian9, 1.4.20-debian9,1.5.0-RC5-debian10, 1.3.49-ubuntu18, 1.4.20-ubuntu18, 1.5.0-RC5-ubuntu18.

1.2 – 1.4 image updates: * Upgraded Apache Solr to 6.6.6 version

1.5 preview image updates:

• Upgraded Apache Druid to 0.16.1 version
• Upgraded Presto SQL to 326 version
• Upgraded Apache Oozie to 5.2.0 version
• Added Apache Atlas 1.2.0
• Set spark.ui.port property to 0 to run Spark UI on random port - removes limit on max number of concurrent Spark jobs (16 by default). Previous behavior can be restored by setting spark.ui.port=4040 property
• Set yarn.nodemanager.address property to 0.0.0.0:8026 to run YARN Node Manager on the same port - if a Node Manager process restarts (e.g. when a VM is preempted), it shows up as the same node in YARN. Otherwise, you would see both a LOST node for the old port, as well as a new RUNNING node with the new port
• Set hive.fetch.task.conversion property to none to disable execution of the SELECT * WHERE conditions LIMIT x queries in the job driver
• Set mapreduce.jobhistory.always-scan-user-dir property to true to enable better support of history files stored on Cloud Storage.
• Set mapreduce.jobhistory.recovery.store.class property to org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService to reduce HDFS usage
• Configured default Spark fair scheduling pool in the /etc/spark/conf/fairscheduler.xml configuration file
• Added R kernel to Jupyter Optional Component

New sub-minor versions of Dataproc images: 1.2.88-debian9, 1.3.48-debian9, 1.4.19-debian9, 1.5.0-RC4-debian10, 1.3.48-ubuntu18, 1.4.19-ubuntu18, 1.5.0-RC4-ubuntu18.

Component Gateway UIs now include a banner showing the project and cluster name being accessed and a button to sign out of Component Gateway for the cluster.

New sub-minor versions of Cloud Dataproc images:: 1.2.87-debian9, 1.3.47-debian9, 1.4.18-debian9, 1.5.0-RC3-debian10, 1.3.47-ubuntu18, 1.4.18-ubuntu18, 1.5.0-RC3-ubuntu18

New images now install the latest Linux kernel version available in Debian backports repository through linux-image-amd64 meta package.

Disabled noisy Jersey logging in the YARN Timeline Server.

1.5 preview image updates

• Upgraded Debian OS to Debian 10 (buster)
• Upgraded Apache Atlas to version 1.2.0
• Upgraded Apache Solr to version 6.6.6
• Upgraded Apache Knox to version 1.3.0
• Backported SPARK-28921: Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)
• Enabled HBase optional component in Dataproc beta.
• Installed Delta Lake 0.4.0 and Apache Iceberg 0.7.0 jars
• Updated Jupyter Python 3 kernel to add SparkMonitor 0.11 support. To create SparkContext, user can import SparkContext and create SparkContext using an auto-created conf object using sc=SparkContext.getOrCreate(conf=conf). To create SparkSession, user can import SparkSession and use SparkSession.builder.appName("Any Name").getOrCreate()

New sub-minor versions of Cloud Dataproc images: 1.2.86-debian9, 1.3.46-debian9, 1.4.17-debian9, 1.5.0-RC2-debian9, 1.3.46-ubuntu18, 1.4.17-ubuntu18, 1.5.0-RC2-ubuntu18

Added a warning when clusters are created with component gateway and kerberos as they are not currently supported together

1.3 image update - Upgraded Hive to version 2.3.6

1.4 image update - Fixed a bug in the Jupyter component that prevented creating text files using the Jupyter and JupyterLab UIs

1.5 preview image update:

• Updated Anaconda to version 2019.10
• Updated Miniconda to version 4.7.12.1
• Upgrade Hue to version 4.5.0
• Upgraded Scala to version 2.12.10
• Upgrade Hadoop to version 2.10.0
• Unset java.net.preferIPv4Stack=true property in Hadoop

• New Dataproc image versions: 1.2.85-debian9, 1.3.45-debian9, 1.4.16-debian9, 1.3.45-ubuntu18, 1.4.16-ubuntu18, and Preview.

  • Note that Dataproc 1.1 will have no further releases.

• Update Conscrypt library to 1.4.2

• Failed Jobs error messages now include a link to the logs in the Cloud Console rather than to the Google Cloud Storage directory containing them.

• 1.3 image

  • Upgrade Spark to 2.3.4.

• 1.4 image

  • Upgrade Hive to 2.3.6.
  • Apply SPARK-28921 to prevent Spark jobs from failing on newer versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

• Preview image

  • Hadoop 2.10
  • Spark 2.4.4
  • Hive 2.3.6
  • Pig 0.17.0
  • Tez 0.9.2
  • Cloud Storage Connector 2.0.0
  • Python 3.7
  • Scala 2.12.0
  • Anaconda3 5.3.0
  • Druid 0.16
  • Hive Webhcat 2.3.2
  • Jupyter 4.5.0
  • Kerberos 1.15.1
  • Presto 324 (prestosql)
  • Zeppelin 0.9.0-SNAPSHOT
  • Zookeeper 3.4.13

• New sub-minor versions of Cloud Dataproc images: 1.1.121-debian9, 1.2.84-debian9, 1.3.44-debian9, 1.4.15-debian9, 1.3.44-ubuntu18, 1.4.15-ubuntu18.

• Image 1.4

  • Upgraded Spark to 2.4.4.

• New sub-minor versions of Cloud Dataproc images: 1.1.120-debian9, 1.2.83-debian9, 1.3.43-debian9, 1.4.14-debian9, 1.3.43-ubuntu18, 1.4.14-ubuntu18.
• Druid Optional Component can now be configured via druid-broker, druid-overlord, druid-coordinator, druid-historical, and druid-middlemanager property prefixes.
• Image 1.3
  • Applied patch for YARN-6629 to Hadoop.
• Image 1.4
  • Applied patch for YARN-6629 to Hadoop.

• New sub-minor versions of Cloud Dataproc images: 1.1.119-debian9, 1.2.82-debian9, 1.3.42-debian9, 1.4.13-debian9, 1.3.42-ubuntu18, 1.4.13-ubuntu18.
• Image 1.4
  • Upgraded Zeppelin to 0.8.1 with fix for compatibility issue with Spark 2.4.

• New sub-minor versions of Cloud Dataproc images: 1.1.118-debian9, 1.2.81-debian9, 1.3.41-debian9, 1.4.12-debian9, 1.3.41-ubuntu18, 1.4.12-ubuntu18.
• Image 1.3
  • Applied patch for MAPREDUCE-7101 to Hadoop.
  • Upgraded Hive to 2.3.5.
• Image 1.4
  • Applied patch for MAPREDUCE-7101 to Hadoop.
  • Upgraded Hive to 2.3.5.
  • Added Sqoop version 1.4.6.

• New sub-minor versions of Cloud Dataproc images: 1.1.117-debian9, 1.2.80-debian9, 1.3.40-debian9, 1.4.11-debian9, 1.3.40-ubuntu18, 1.4.11-ubuntu18.

• New sub-minor versions of Cloud Dataproc images: 1.1.116-debian9, 1.2.79-debian9, 1.3.39-debian9, 1.4.10-debian9, 1.3.39-ubuntu18, 1.4.10-ubuntu18.

• New major and sub-minor versions of Cloud Dataproc images: 1.1.115-debian9, 1.2.78-debian9, 1.3.38-debian9, 1.4.9-debian9, 1.3.38-ubuntu18, 1.4.9-ubuntu18
• Image 1.1:
  • Upgraded Hadoop to 2.7.7

• New major and sub-minor versions of Cloud Dataproc images: 1.1.114-debian9, 1.2.77-debian9, 1.3.37-debian9, 1.4.8-debian9, 1.3.37-ubuntu18, 1.4.8-ubuntu18
• Added a warning when using the alpha version of Kerberos configuration. Kerberos is now configured on a Dataproc cluster via the KerberosConfig message instead of Dataproc properties and optional components. Create cluster requests that use the Component+Properties mechanism in the clusters API to configure Kerberos will now generate a warning to use KerberosConfig instead.
• Clusters created with small PD Standard disks will no longer trigger a warning during creation if they also have local SSDs specified on the disk configuration.
• Audit requests and responses for the Cloud Dataproc Autoscaling Policies API. Previously, API methods were audited, but not the actual request and response bodies.
• Image 1.3 and 1.4:
  • Cloud Storage connector upgraded to 1.9.17
• Image 1.3:
  • Upgraded Tez to 0.9.2
• Image 1.4:
  • Upgraded Spark to 2.4.3
  • Upgraded Flink to 1.6.4
  • Upgraded Zeppelin to 0.8.1

• New major and sub-minor versions of Cloud Dataproc images: 1.1.113-debian9, 1.2.76-debian9, 1.3.36-debian9, 1.4.7-debian9, 1.3.36-ubuntu18, 1.4.7-ubuntu18
• Failed autoscaling operations now appear with error messages in logs. Previously, log included only the failing operation ID, which user was expected to use to lookup the error message.
• Image 1.4:
  • Upgraded Tez to 0.9.2.
• Image 1.3:
  • The zstandard codec is now supported in Hadoop.

• New major and sub-minor versions of Cloud Dataproc images: 1.1.111-debian9, 1.2.74-debian9, 1.3.34-debian9, 1.4.5-debian9, 1.3.34-ubuntu18, 1.4.5-ubuntu18
• Image 1.1:
  • Parallelized Hive.copyFiles method in Spark's fork of Hive.
  • Resolved scheme-less Spark event log directory relative to default filesystem.
• Image 1.4:
  • Support zstandard in Hadoop

• New major and sub-minor versions of Cloud Dataproc images: 1.0.119-debian9, 1.1.110-debian9, 1.2.73-debian9, 1.3.33-debian9, 1.4.4-debian9, 1.3.33-ubuntu18, 1.4.4-ubuntu18
• This is the final release of Dataproc 1.0 image.
• Image 1.3 and 1.4:
  • Allow remapping Jupyter and Zeppelin Optional Component ports via dataproc:{jupyter,zeppelin}.port properties
• Image 1.4:
  • Upgrade Spark to 2.4.2

• New major and sub-minor versions of Cloud Dataproc images: 1.0.118-debian9, 1.1.109-debian9, 1.2.72-debian9, 1.3.32-debian9, 1.4.3-debian9, 1.3.32-ubuntu18, 1.4.3-ubuntu18
• Image 1.3:
  • Upgraded Spark to 2.3.3.

• New major and sub-minor versions of Cloud Dataproc images: 1.0.117-debian9, 1.1.108-debian9, 1.2.71-debian9, 1.3.31-debian9, 1.4.2-debian9, 1.3.31-ubuntu18, 1.4.2-ubuntu18
• Image 1.2:
  • Parallelized the Hive.copyFiles method in Spark's fork of Hive.
• Image 1.4:
  • Upgraded Spark to 2.4.1.
  • Included S3 and Azure Storage connectors in Spark 2.4 libraries.
• Changed compute root API URL from www.googleapis.com to https://compute.googleapis.com due to a change in compute client.
• Allowed updating clusters after the expiration of its custom image. Previously, after the expiration of a cluster's custom image, the cluster could not be resized manually or through autoscaling. Now, clusters created with valid custom images can be resized indefinitely.
• Disallowed using the dataproc:am.primary_only cluster property with autoscaling since this property, which restricts the scheduling of app masters to primary workers, uses YARN Node Labels. Non-exclusive node labels are not compatible with autoscaling since they cause YARN to incorrectly report cluster utilization metrics (including pending memory). See YARN-9088 Note: Hadoop MapReduce and Spark jobs in client mode, which is the Cloud Dataproc default, are resilient to app master failures (scheduling app masters on primary workers is not necessary in client mode).

• New sub-minor versions of Cloud Dataproc images - 1.0.116-debian9, 1.1.107-debian9, 1.2.70-debian9, 1.3.30-debian9, 1.4.1-debian9, 1.3.30-ubuntu18, 1.4.1-ubuntu18
• Removed limit on concurrent number of running workflow templates per project.
• Put Zookeeper Server in secure (Kerberos) mode if Kerberos is enabled on HA clusters.
• The dataproc:jobs.file-backed-output.enable service property now configures Cloud Dataproc jobs to pipe their output to temporary files in the /var/log/google-dataproc-job directory.
• Added cluster UUID in the description of Compute Engine instance groups created by Cloud Dataproc to be used as secondary identifier to classify whether a Google Compute Engine instance group manager is created by Cloud Dataproc.
• Image 1.3 only:
  • Parallelized Hive.copyFiles method in the Spark fork of Hive.

• New major and sub-minor versions of Cloud Dataproc images: 1.0.115-debian9, 1.1.106-debian9, 1.2.69-debian9, 1.3.29-debian9, 1.4.0-debian9, 1.3.29-ubuntu18, 1.4.0-ubuntu18
• Image 1.4:
  • Installed PyArrow for Spark UDF to Pandas conversion (see PySpark Usage Guide for Pandas with Apache Arrow).
• Image 1.3 and 1.4:
  • Cloud Storage connector upgraded to version 1.9.16.
• Modified the Kerberos optional component parameter requirement: when cross realm trust is enabled, the admin server of the remote realm is optional.

• New sub-minor versions of Cloud Dataproc images - 1.0.114-deb9, 1.1.105-deb9, 1.2.68-deb9, 1.3.28-deb9, 1.4.0-RC12-deb9
• Image 1.3 and 1.4 preview
  • Cloud Storage connector upgraded to version 1.9.16.
• Image 1.4 preview
  • Added Spylon (Scala) kernel for Jupyter.

• New sub-minor versions of Cloud Dataproc images - 1.0.113-deb9, 1.1.104-deb9, 1.2.67-deb9, 1.3.27-deb9, 1.4.0-RC11-deb9
• Image 1.0-1.2
  • YARN-6868: Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml.
• Image 1.0-1.4 preview
  • MAPREDUCE-7185: Parallelize part files move in FileOutputCommitter.
• Image 1.1-1.4 preview
  • Parallelized files move and deletion in Spark Driver during SQL queries execution:
    • HIVE-20395: Parallelize files move in replaceFiles method.
    • Parallelize Hive.moveFile method.
    • Parallelize FileUtils.trashFilesUnderDir method.
  • Upgraded Zookeeper to 3.4.13 version.
  • Upgraded Hadoop LZO to 0.4.20 version.
• Image 1.3
  • Added Solr 6.6.0
  • Upgraded Kafka to 1.1.1 version.
  • Upgraded the Cloud Storage connector to 1.9.15 version. Review the change notes in the GitHub repository for more information.
• Image 1.4 preview
  • Upgraded Solr to 7.6.0 version.
  • Use 1TB sized PD-Standard for masters and primary workers by default to improve I/O performance.
  • Moved Cloud Storage connector to /usr/local/share/google/dataproc/lib directory.
  • Upgraded the Cloud Storage connector to 1.9.15 version. Review the change notes in the GitHub repository for more information.

• Due to the issue announced on February 22, 2019, Cloud Dataproc image versions announced on February 14, 2019 (1.0.111-deb9, 1.1.102-deb9, 1.2.65-deb9, 1.3.25-deb9, 1.4.0-RC9-deb9) and the changes and fixes associated with those image versions are currently not available. The latest images versions available for cluster creation, and the changes and fixes associated with those image versions, are those announced on February 11, 2019 (1.0.110-deb9, 1.1.101-deb9, 1.2.64-deb9, 1.3.24-deb9, 1.4.0-RC8-deb9).

• New sub-minor versions of Cloud Dataproc images - 1.0.111-deb9, 1.1.102-deb9, 1.2.65-deb9, 1.3.25-deb9, 1.4.0-RC9-deb9
• The yarn-site.xml yarn.resourcemanager.webapp.methods-allowed property now defaults to "GET,HEAD". This change restricts the HTTP methods that can be called on the YARN Resource Manager web UI (default port 8088) and REST APIs to GET and HEAD only, and disables job submission and modifications via the YARN REST API. You can override the default values and enable specific HTTP methods on port 8088 by setting the yarn.resourcemanager.webapp.methods-allowed property to one or more comma-separated HTTP method names when you create a cluster. An ALL value will allow all HTTP methods on the port.
  Example: gcloud dataproc clusters create --properties='yarn:yarn.resourcemanager.webapp.methods allowed=GET,POST,DELETE'
  Recommendation: If you set this property to allow non-default HTTP methods, make sure to configure firewall rules and other security settings to restrict access to port 8088 (see Cluster Web Interfaces→Avoid Security Vulnerabilities).
• Image 1.2 only:
  • Upgraded Spark to 2.2.3
  • HIVE-15208: Query string is now HTML encoded for Web UI
• Image 1.3 only:
  • Upgraded Flink to 1.5.6
  • The Cloud Storage connector has been upgraded to 1.9.13. Review the change notes in the GitHub repository for more information.
• Image 1.4 preview only:
  • The cloud Storage connector has been upgraded to 1.9.13. Review the change notes in the GitHub repository for more information.
  • Used Python 3.6 instead of 3.7 as the default user Python version
    • Downgraded default Miniconda3 to 4.5.4
    • Downgraded Anaconda3 optional component to 5.2.0
  • Added Solr 6.6.0
  • HIVE-18786: NPE in Hive windowing functions
  • Upgraded Zookeeper to 3.4.13
  • Upgraded Hadoop LZO to 0.4.20

• New sub-minor versions of Cloud Dataproc images - 1.0.110-deb9, 1.1.101-deb9, 1.2.64-deb9, 1.3.24-deb9, 1.4.0-RC8-deb9.
• YARN applications that are submitted via the Cloud Dataproc Jobs API now include the Job UUID in the application tags with the format "dataprocuuid${UUID}".
• Removed upper limit on total number of threads in job drivers.
• Image 1.3 only:
  • Added a new yarn-site.xml property (yarn.resourcemanager.webapp.methods-allowed) which is set with a comma- separated list of the names of HTTP methods that the YARN Resource Manager will allow to be called in its web UI and REST API (default port 8088). The current default value is "ALL", which allows all HTTP methods (GET,POST,PUT,PATCH,DELETE,HEAD,CONNECT,OPTIONS, TRACE). It is recommended that this property be set to "GET" to disable remote code submission or other modifications via the REST API. This property will default to "GET" for all Cloud Dataproc clusters in a future release (see Important cross-update notes, above).
• Image 1.4 preview only:
  • Upgraded Kafka to 1.1.1
  • Added a new yarn-site.xml property (yarn.resourcemanager.webapp.methods-allowed) which is set with a comma- separated list of the names of HTTP methods that the YARN Resource Manager will allow to be called in its web UI and REST API (default port 8088). The current default value is "ALL", which allows all HTTP methods (GET,POST,PUT,PATCH,DELETE,HEAD,CONNECT,OPTIONS, TRACE). It is recommended that this property be set to "GET" to disable remote code submission or other modifications via the REST API. This property will default to "GET" for all Cloud Dataproc clusters in a future release (see Important cross-update notes, above).

• New sub-minor versions of Cloud Dataproc images - 1.0.109-deb9, 1.1.100-deb9, 1.2.63-deb9, 1.3.23-deb9, 1.4.0-RC7-deb9.

• New sub-minor versions of Cloud Dataproc images - 1.0.108-deb9, 1.1.99-deb9, 1.2.62-deb9, 1.3.22-deb9, 1.4.0-RC6-deb9.

• New sub-minor versions of Cloud Dataproc images - 1.0.107-deb9, 1.1.98-deb9, 1.2.61-deb9, 1.3.21-deb9, 1.4.0-RC5-deb9.
• Do not throttle scheduling of submitted jobs based on master node CPU load.
• Set yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs to 86400 (1 day) on Cloud Dataproc 1.3+ clusters. This fixes the bug where trying to graceful decommission nodes with a timeout on Cloud Dataproc 1.3+ would still terminate at 1 hour.
• Upgrade the Presto Component to version 0.215.
• Change property parameter in Kerberos Component "dataproc:kerberos.cross-realm-trust.shared-key.uri" to "dataproc:kerberos.cross-realm-trust.shared-password.uri".
• Make the Presto Component compatible with the Kerberos Component, so if both Components are enabled, Presto will run in secure mode.
• Explicitly set the property hadoop.ssl.enabled.protocols to "TLSv1,TLSv1.1,TLSv1.2". This change will affect any newly created clusters using a Dataproc image 1.2 and up. This is to help avoid issues caused by incompatibility between SSLv2Hello (enabled by default in Hadoop 2.8.4 and up) and the Conscrypt security provider, the default security provider used by the Cloud Storage connector.
• Make 1.3 default image version.
• Image 1.0 only:
  • Fix Hive Metastore/Hive Server2/Spark History Server systemd services exited problem.
• Image 1.2 only:
  • Upgrade Hue to 4.3.0.
  • Upgrade Hadoop to 2.8.5.
  • Patch [SPARK-21960][STREAMING] Spark Streaming Dynamic Allocation should respect spark.executor.instances.
• Image 1.3 only:
  • Upgrade Hadoop to 2.9.2.
  • Upgrade Hue to 4.3.0.
  • Patch Spark-25959 GBTClassifier picks wrong impurity stats on loading.
  • Patch HIVE-20395 Parallelize files move in the ql.metadata.Hive#replaceFiles
  • Upgrade Hive to 2.3.4.
  • Cloud Storage connector upgrade (for more information, review the change notes in the GitHub repository).
• Image 1.4 - Preview only:
  • Upgrade Hadoop to 2.9.2.
  • Upgrade Flink to 1.6.2.
  • Patch Spark-25959 GBTClassifier picks wrong impurity stats on loading.
  • Upgrade Hive to 2.3.4.
  • Cloud Storage connector upgrade (for more information, review the change notes in the GitHub repository).
• Remove the logic in Kerberos optional component to set property hadoop.ssl.enabled.protocols. It will be set by the Dataproc service.
• Collect Zookeeper logs as part of the diagnose command.
• Export Zookeeper logs to stackdriver.
• Run Dataproc Agent process at higher priority.
• Add parquet (parquet-cli) binary to PATH for debugging convenience.
• Fix Hive-WebHCat to use effective python profile, and fix issues with logging directory.
• Publish up-to-date HDFS and YARN metrics from the Dataproc agent. Earlier, there was an up to 30 second delay between collection and publishing.
• Reduce the minimum autoscaling cooldown period to 2 minutes. The cooldown period still starts after a previous scaling operation completes.
• In Kerberos optional component activation process, the KMS key existence verification is collapsed into decryption while using the key.

• New sub-minor versions of Cloud Dataproc images - 1.0.105-deb9, 1.1.96-deb9, 1.2.59-deb9, 1.3.19-deb9, 1.4.0-RC3-deb9.
• Allow properties in zoo.cfg to be configured through 'zookeeper:' property prefix.
• Allow users to parameterize the name of a managed cluster in workflow templates.

• New sub-minor versions of Cloud Dataproc images - 1.0.104-deb9, 1.1.95-deb9, 1.2.58-deb9, 1.3.18-deb9, 1.4.0-RC2-deb9.
• Upgraded Spark from 2.3.1 to 2.3.2 on image version 1.3.
• Extra ' character in /etc/hadoop/conf.empty/yarn-site.xml under yarn.application.classpath prevents running Hadoop jobs.
• Image 1.2 only:
  • Fix issue where difference in featureImportances results on computed vs. saved models. (SPARK-25959).
• Image 1.3 and later only:
  • Jupyter Optional Component changes default storage location of notebooks from gs://staging-bucket/notebooks to gs://staging-bucket/notebooks/jupyter. This can be changed by setting dataproc:jupyter.notebook.gcs.dir cluster property.
  • Zeppelin Optional Component will now save notebooks to the staging Google Cloud Storage bucket in gs://staging-bucket/notebooks/zeppelin. This can be changed by setting zeppelin:zeppelin.notebook.gcs.dir cluster property.
  • Zeppelin Optional Component can now be configured via zeppelin and zeppelin-env property prefixes.

• New sub-minor versions of Cloud Dataproc images - 1.0.102-deb9, 1.1.93-deb9, 1.2.56-deb9, 1.3.16-deb9.
• Creating Dataproc clusters will now issue a warning if we detect a potential security vulnerability due to misconfigured firewall rules allowing public access to YARN ports.
• Looking up the details of a job will show who submitted that job in the submittedBy field.
• Image 1.3 only:
  • Upgraded Cloud Storage Connector to version 1.9.10. See the GitHub release notes.

• New sub-minor versions of Cloud Dataproc images - 1.0.101-deb9, 1.1.92-deb9, 1.2.55-deb9, 1.3.15-deb9.
• Minor image versions now redirect to Debian 9-based images. For example, 1.2 now points to 1.2-deb9. There will be no new Debian 8-based images.
• Job UUIDs are now exposed to allow job runs to be uniquely identified.
• The Cloud Storage Connector now sets fadvise to SEQUENTIAL for Hadoop DistCp jobs. This mode is optimized for streaming reads, which are most efficient for these workloads.
• Removed the ALPN boot jar from Cloud Dataproc versions 1.0 and 1.1 due to incompatibility with the latest OpenJDK 8 distributed with Debian. gRPC users must use a form of netty-tcnative, for example, io.grpc:grpc-netty-shaded. This already applies to 1.2 and 1.3.
• Reduced the Linux process priority of user jobs.
• dfs.namenode.datanode.registration.retry-hostname-dns-lookup is now set to true.
• Increased the maximum number of DistCp tasks scheduled per node. This improves DistCp performance.
• Image 1.3 only:
  • Ported HDFS-13056 to Hadoop 2.9.
  • Upgraded Cloud Storage Connector to version 1.9.9. See the GitHub release notes.
  • Presto is now supported as an optional top-level component.

• New sub-minor versions of Cloud Dataproc images - 1.0.100-deb8, 1.1.91-deb8, 1.2.54-deb8, 1.3.14-deb8, 1.0.100-deb9, 1.1.91-deb9, 1.2.54-deb9, 1.3.14-deb9.

• New sub-minor versions of Cloud Dataproc images - 1.0.99-deb8, 1.1.90-deb8, 1.2.53-deb8, 1.3.13-deb8, 1.0.99-deb9, 1.1.90-deb9, 1.2.53-deb9, 1.3.13-deb9.

• New sub-minor versions of Cloud Dataproc images - 1.0.98-deb8, 1.1.89-deb8, 1.2.52-deb8, 1.3.12-deb8, 1.0.98-deb9, 1.1.89-deb9, 1.2.52-deb9, 1.3.12-deb9.
• Image 1.3 only: Cloud Storage connector upgrade (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.9.8 version.
• Image 1.0 only: Upgrade Hadoop to 2.7.4.

• New sub-minor versions of Cloud Dataproc images - 1.0.97-deb8, 1.1.88-deb8, 1.2.51-deb8, 1.3.11-deb8, 1.0.97-deb9, 1.1.88-deb9, 1.2.51-deb9, 1.3.11-deb9.
• Image 1.1 only: Upgraded Zeppelin to 0.7.3.
• Image 1.1 only: Published YARN and HDFS metrics to Stackdriver (except HDFS PendingDeletionBlocks) from clusters using image version 1.1.82 and above.

• New sub-minor versions of Cloud Dataproc images - 1.0.97-deb8, 1.1.88-deb8, 1.2.51-deb8, 1.3.11-deb8, 1.0.97-deb9, 1.1.88-deb9, 1.2.51-deb9, 1.3.11-deb9.
• Image 1.3 only: Cloud Storage connector upgrade (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.9.7 version
• Image 1.0-1.2 only: Cloud Storage and BigQuery connector upgrades (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.6.10 version
  • BigQuery connector has been upgraded to 0.10.11 version

• New sub-minor versions of Cloud Dataproc images - 1.0.95-deb8, 1.1.86-deb8, 1.2.49-deb8, 1.3.9-deb8, 1.0.95-deb9, 1.1.86-deb9, 1.2.49-deb9, 1.3.9-deb9.
• Changed initialization actions to run inside a login shell so that environment profile changes can be seen by subsequent init actions.
• Image 1.3 only: Cloud Storage connector upgrade (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.9.6 version
• Image 1.0-1.2 only: Cloud Storage and BigQuery connector upgrades (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.6.9 version
  • BigQuery connector has been upgraded to 0.10.10 version

• New sub-minor versions of Cloud Dataproc images - 1.0.94-deb8, 1.1.85-deb8, 1.2.48-deb8, 1.3.8-deb8, 1.0.94-deb9, 1.1.85-deb9, 1.2.48-deb9, 1.3.8-deb9.
• Added Flink 1.5.0 and HBase 1.3.2 to 1.3-deb8 images.

• New sub-minor versions of Cloud Dataproc images - 1.0.93-deb8, 1.1.84-deb8, 1.2.47-deb8, 1.3.7-deb8, 1.0.93-deb9, 1.1.84-deb9, 1.2.47-deb9, 1.3.7-deb9.

• New sub-minor versions of Cloud Dataproc images - 1.0.92-deb8, 1.1.83-deb8, 1.2.46-deb8, 1.3.6-deb8, 1.0.92-deb9, 1.1.83-deb9, 1.2.46-deb9, 1.3.6-deb9.
• Image 1.0-1.2 only: Cloud Storage and BigQuery connector upgrades (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to 1.6.8 version
  • BigQuery connector has been upgraded to 0.10.9 version
• Image 1.3 only: Cloud Storage connector upgrade to 1.9.5 version (for more information, review the change notes in the GitHub repository)
• Image 1.3 with Debian 9 only:
  • Upgrade Spark to 2.3.1.
  • Add HBase 1.3.2.
  • Add Flink 1.5.0.

• New sub-minor versions of Cloud Dataproc images - 1.0.91-deb8, 1.0.91-deb9, 1.1.82-deb8, 1.1.82-deb9, 1.2.45-deb8, 1.2.45-deb9, 1.3.5-deb8, 1.3.5-deb9.
• Security fix: Install Linux Kernel 4.9 in all image versions to get security fixes for CVE-2018-3590 and CVE-2018-3591 in all new Debian 8 images.

• New sub-minor versions of Cloud Dataproc images - 1.0.90, 1.1.81, 1.2.45, 1.3.5
• Set maximum number of open files to 65535 for all Systemd services.

• New sub-minor versions of Cloud Dataproc images - 1.0.89, 1.1.80, 1.2.44, 1.3.4
• In HA clusters, hadoop.security.token.service.use_ip is now set to false.
• Upgraded Hadoop to 2.8.4. (Dataproc 1.2)

• New sub-minor versions of Cloud Dataproc images - 1.3.3
• Changes to 1.3 image only:
  • Disabled node blacklisting in Tez jobs (set tez.am.node-blacklisting.enabled=false). This affects all Hive jobs, which run on Tez by default.

• New sub-minor versions of Cloud Dataproc images - 1.0.88, 1.1.79, 1.2.43, 1.3.2
• Cloud Dataproc now adds resource location to generated cloud audit logs.

• New sub-minor versions of Cloud Dataproc images - 1.0.87, 1.1.78, 1.2.42, 1.3.1
• Changes to 1.3 image only:
  • The per-driver Spark web UI has been re-enabled.
  • The HCatalog library is installed by default.
• MapReduce job history server recovery is now enabled by default.

• New sub-minor versions of Cloud Dataproc images - 1.0.86, 1.1.77, 1.2.41, 1.3.0
• Changes to 1.3 image only:
  • Cloud Storage connector was upgraded to version 1.9.0 (see the change notes in the GitHub repository).
  • The NFS Kernel Server is no longer installed.

• New sub-minor versions of Cloud Dataproc images - 1.0.85, 1.1.76, 1.2.40
• Upgraded the Cloud Storage and BigQuery connectors in 1.0.85, 1.1.76, 1.2.40 (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to version 1.6.7
  • BigQuery connector has been upgraded to version 0.10.8

• New sub-minor versions of Cloud Dataproc images - 1.0.84, 1.1.75, 1.2.39
• Initialization Action output is now available in Stackdriver under google.dataproc.startup log.
• Cloud Dataproc will refuse to create new clusters based on most images created before 2018-02-14. Customers do not need to change minor versions, but if they specify a subminor version that falls into this group, they will need to use a newer subminor version (e.g. 1.1.39 can not be used for new clusters, but 1.1 and 1.1.73 are valid).

• New sub-minor versions of Cloud Dataproc images - 1.0.83, 1.1.74, 1.2.38
• Change the jdbc connect string passed to beeline when submitting Hive jobs to high availability clusters through the Cloud Dataproc Jobs API. The new connect string takes advantage of high availability of HiveServer2.

• New sub-minor versions of Cloud Dataproc images - 1.0.82, 1.1.73, 1.2.37
• Hive Server 2 now runs all three masters in High Availablility mode.
• Preview image changes (Dataproc 1.3):
  • Now requires a 15 GB minimum boot disk size.
  • The NameNode Service RPC port has changed from 8040 to 8051.
  • The SPARK_HOME environment variable is now globally set.

• New sub-minor versions of Cloud Dataproc images - 1.0.81, 1.1.72, 1.2.36
• Upgraded the Cloud Storage and BigQuery connectors in 1.0.81, 1.1.72, 1.2.36 (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to version 1.6.6
  • BigQuery connector has been upgraded to version 0.10.7
• New version of Cloud Dataproc 1.3 preview image:
  • Remove BigQuery connector from image. Users should instead include the BigQuery connector with jars for their jobs.
  • Cloud Dataproc 1.3 is not supported
  • See Cloud Dataproc version list for more information.
• Hive Metastore is configured to run on all three masters in High Availability mode

• New sub-minor versions of Cloud Dataproc images - 1.0.80, 1.1.71, 1.2.35

• New sub-minor versions of Cloud Dataproc images - 1.0.79, 1.1.70, 1.2.34

• New sub-minor versions of Cloud Dataproc images - 1.0.78, 1.1.69, 1.2.33

• New sub-minor versions of Cloud Dataproc images - 1.0.77, 1.1.68, 1.2.32
• Changed the Namenode HTTP port from 50070 to 9870 on HA clusters in preview image. WebHDFS, for example, is accessible at http://clustername-m-0:9870/webhdfs/v1/. This is consistent with standard and single node clusters in Dataproc 1.2+. Dataproc 1.0 and 1.1 clusters will continue to use port 50070 for all cluster modes.
• Upgraded the Cloud Storage and BigQuery connectors (for more information, review the change notes in the GitHub repository):
  • Cloud Storage connector has been upgraded to version 1.6.5
  • BigQuery connector has been upgraded to version 0.10.6

• New sub-minor versions of Cloud Dataproc images - 1.0.76, 1.1.67, 1.2.31
• Updated software versions in Cloud Dataproc image version 1.2 to the following:
  • Apache Spark 2.2.0 -> 2.2.1
  • Apache Hadoop 2.8.2 -> 2.8.3

• New sub-minor versions of Cloud Dataproc images - 1.0.75, 1.1.66, 1.2.30

• New sub-minor versions of Cloud Dataproc images - 1.0.74, 1.1.65, 1.2.29

• New sub-minor versions of Cloud Dataproc images - 1.0.73, 1.1.64, 1.2.28
• Cloud Storage and BigQuery connector upgrades: The Cloud Storage connector has been upgraded to gcs-connector-1.6.4 and the BigQuery connector has been upgraded to bigquery-connector-0.10.5. For more information, review the GitHub repository 2018-03-19 (Google Cloud Storage 1.6.4, BigQuery 0.10.5) changelog.

• New sub-minor versions of Cloud Dataproc images - 1.0.73, 1.1.64, 1.2.28 .
• Updated Conda initialization action to use latest Miniconda version if Spark is at least version 2.2.0.

• Cloud Dataproc Auto Zone is now generally available.
• New sub-minor versions of Cloud Dataproc images - 1.0.71, 1.1.62, 1.2.26.

• New sub-minor versions of Cloud Dataproc images - 1.0.70, 1.1.61, 1.2.25.
• An optional requestId field has been added to CreateCluster, UpdateCluster, DeleteCluster, and SubmitJob. The requestId field can be used to prevent the processing of duplicate requests (subsequent requests with a requestId that is the same as a previous requestId are ignored).
• Increased the MapReduce and Spark History Server heap sizes when running on large master nodes.

• New sub-minor versions of Cloud Dataproc images - 1.0.69, 1.1.60, 1.2.24.

• New sub-minor versions of Cloud Dataproc images - 1.0.68, 1.1.59, 1.2.23.
• Updating cluster labels now also updates labels on PDs attached to master and primary worker VMs.

• New sub-minor versions of the Cloud Dataproc images - 1.0.67, 1.1.58, 1.2.22.

• The High Availability Mode feature is now in public release (was Beta). Cloud Dataproc clusters can be created with high availability mode enabled. This is an optional feature when creating a cluster. In high availability mode, Cloud Dataproc clusters have three master nodes instead of one. This enables both HDFS High Availability and YARN High Availability to allow uninterrupted YARN and HDFS operations despite any single-node failures or reboots.

  This feature is available when creating clusters with the [gcloud](https://cloud.google.com/sdk/gcloud/reference/dataproc/) command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See High Availability Mode for more information.

• An Update Cluster operation now returns a DONE operation if the update request has no work to perform.

• New sub-minor versions of the Cloud Dataproc images - 1.0.66, 1.1.57, 1.2.21.
• Cloud Storage and BigQuery connector upgrades: The Cloud Storage connector has been upgraded to gcs-connector-1.6.3 and the Google BigQuery connector has been upgraded to bigquery-connector-0.10.4. For more information, review the GitHub repository Cloud Storage change notes and BigQuery change notes.
• Updates to BoringSSL and Conscrypt.
• User labels set on a cluster now propagate to attached disks.

• New sub-minor versions of the Cloud Dataproc images - 1.0.63, 1.1.54, 1.2.18.
• Automatic retry of commitJob (introduced in MAPREDUCE-5485) is now enabled by default; set mapreduce.fileoutputcommitter.failures.attempt to 1 to revert to the old behavior.

• The Google Cloud Dataproc Graceful Decommissioning feature is now in public release (was Beta). Graceful Decomissioning enables the removal of nodes from the cluster without interrupting jobs in progress. A user-specified timeout specifies how long to wait for jobs in progress to finish before forcefully removing nodes. This feature is available when updating clusters using the gcloud command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See Graceful Decommissioning for more information.

• The Single Node Clusters feature is now in public release (was Beta). Single node clusters are Cloud Dataproc clusters with only one node that acts as the master and worker. Single node clusters are useful for a number of activities, including development, education, and lightweight data science.

  This feature is available when creating clusters with the gcloud command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See Single node clustersfor more information.

• The Restartable Jobs feature is now in public release (was Beta). Cloud Dataproc jobs have an optional setting to restart failed jobs. When you set a job to restart, you specify the maximum number of retries per hour (maximum is 10). Restartable jobs allow you to mitigate common types of job failure, and are especially useful for long-running and streaming jobs. This feature is available when submitting jobs using the gcloud command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See Restartable jobs for more information.

• New sub-minor versions of the Cloud Dataproc images - 1.0.58, 1.1.49, 1.2.13.
• Added new optimization that increases the performance of list operations for operations and jobs when tags are used.

• New sub-minor versions of the Cloud Dataproc images - 1.0.57, 1.1.48, 1.2.12.
• Apache Hadoop has been upgraded to 2.8.2 in the Cloud Dataproc 1.2 image.

• When using workflow cluster selectors, if more than one cluster matches the specified label(s), Cloud Dataproc will select the cluster with the most free YARN memory. This change replaces the old behavior of choosing a random cluster with the matching label.
• New sub-minor versions of the Cloud Dataproc images - 1.0.56, 1.1.47, 1.2.11.
• HTTP 404 and 409 errors will now show the full resource name in order to provide more useful error messages.

• All WorkflowTemplate operations after October 27, 2017 will be logged in Cloud Audit Logging.
• New sub-minor versions of the Cloud Dataproc images - 1.0.55, 1.1.46, 1.2.10.

• New sub-minor versions of the Cloud Dataproc images - 1.0.54, 1.1.45, 1.2.9.

• The fluentd on Cloud Dataproc clusters has been reconfigured to concatenate multi-line error messages. This should make error messages easier to locate.
• Clusters created with Cloud Dataproc Workflows can now use auto-zone placement.
• Starting with this release, sub-minor releases to the Cloud Dataproc images will be mentioned in the release notes.
• New sub-minor versions of the Cloud Dataproc images - 1.0.53, 1.1.44, 1.2.8.

• New Subminor Image Versions – The latest subminor image versions for 1.0, 1.1, and 1.2 now map to 1.0.51, 1.1.42, 1.2.6, respectively.

• New Subminor Image Versions – The latest subminor image versions for 1.0, 1.1, and 1.2 now map to 1.0.49, 1.1.40, 1.2.4, respectively.
• All Cloud Dataproc clusters now have a goog-dataproc-cluster-name label that is propagated to underlying Google Compute Engine resources and can be used to determine combined Cloud Dataproc related costs in exported billing data.

Beginning today, Cloud Dataproc 1.2 will be the default version for new clusters. To use older versions of Cloud Dataproc, you will need to manually select the version on cluster creation.

Apache Hadoop on Cloud Dataproc 1.2 has been updated to version 2.8.1

Starting with the Cloud Dataproc 1.2 release, the ALPN boot jars are no longer provided on the Cloud Dataproc image. To avoid Spark job breakage, upgrade Cloud Bigtable client versions, and bundle boringssl-static with Cloud Dataproc jobs. Our initialization action repository contains initialization actions to revert to the previous (deprecated) behavior of including the jetty-alpn boot jar. This change should only impact you if you use Cloud Bigtable or other Java gRPC clients from Cloud Dataproc.

Cloud Storage and BigQuery connector upgrades – The Cloud Storage connector has been upgraded to gcs-connector-1.6.1 and the BigQuery connector has been upgraded to bigquery-connector-0.10.2. For more information, review the Cloud Storage or BigQuery change notes in the GitHub repository.

• In the Hadoop 2.8-based preview image, the YARN capacity scheduler has been set to use the fair ordering policy instead of FIFO.
• The Cloud Dataproc IAM role names have been updated to be consistent with other Google Cloud products.
• New logging and monitoring permissions have been added to the Dataproc/Dataproc Worker IAM role.

Apache Hive on Cloud Dataproc 1.1 has been updated to version 2.1.1.

The Conscrypt security provider has been temporarily changed from the default to an optional security provider. This change was made due to incompatibilities with some workloads. The Conscrypt provider will be re-enabled as the default with the release of Cloud Dataproc 1.2 in the future. In the meantime, you can re-enable the Conscrypt provider when creating a cluster by specifying this Cloud Dataproc property:

--properties dataproc:dataproc.conscrypt.provider.enable=true

Updates to user labels applied to Cloud Dataproc clusters will now be applied to managed instance group templates. Since preemptible virtual machines are included in a managed instance group, label updates are now applied to preemptible VMs.

The gcloud command now requires a double dash (--) between gcloud-specific arguments and arguments to those commands. For example, if you used this command in the past:

gcloud dataproc jobs submit spark --cluster example-cluster \
    --class sample_class --jars jar_file 1000

The new format for the command requires a double-dash with a space before and after the double dash:

gcloud dataproc jobs submit spark --cluster example-cluster \
    --class sample_class --jars jar_file -- 1000

.

• Cloud Dataproc Stackdriver Logging Changes
  • With new images, cluster logs are now exported to Stackdriver as resource type cloud_dataproc_cluster (was previously dataproc_cluster).
  • Cloud Audit logs will be emitted to both cloud_dataproc_cluster and dataproc_cluster (deprecated) until the March 9th release.
  • Stackdriver logs for new images are indexed first by cluster name and then cluster UUID to assist in filtering logs by cluster name or cluster instance.
• Cloud Dataproc Stackdriver Monitoring Changes
  • Logs-based metrics from Cloud Dataproc logs are now visible in Stackdriver.
• Cloud Dataproc User Labels Changes
  • User Labels on Dataproc jobs can now be updated with the gcloud command-line tool or the Cloud Dataproc REST API.

• The diagnose command has been updated to include the jstack output of the agent and spawned drivers.

• Increased number of situations where Cloud-Dataproc services are automatically restarted by systemd on clusters in the event of unexpected or unhealthy behavior.

• Fixed an issue where failures during a cluster update caused the cluster to fail. Now, update failures return the cluster to Running state.
• Fixed an issue where submitting a large number of jobs rapidly or over a long period of time caused a cluster to fail.
• Increased the maximum number of concurrent jobs per cluster.

• Fixed an issue where HiveServer2 was not healthy for up to 60 seconds after the cluster was deployed. Hive jobs should now successfully connect to the required HiveServer2 immediately after a cluster is deployed.

• Fixed an issue where CPU quota validation for preemptible virtual machines was being performed against the non-preemptible CPU quota even when preemptible CPU quota was set.

• Optimized the listing of resources by state and cluster UUID. This should reduce several list operations from seconds to milliseconds.

• Optimized how jobs are listed based on state or cluster uuid. This should significantly decrease the time required to list jobs.

• While Cloud Dataproc has been using Java 8 since its Beta launch in September 2015, there is now a hard dependency on Java 8 or higher.
• The --preemptible-worker-boot-disk-size command no longer requires that you specify 0 preemptible workers if you do not want to add preemptible machines when you create a cluster.

• Updated Debian guest environment with latest changes as outlined in the Linux Guest Environment for Google Compute Engine overview.

• Cluster resource definitions for newly deployed clusters now display a fully resolved sub-minor image-version (e.g. 1.0.11 instead of 1.0). This makes it easier to temporarily revert to an older sub-minor version. See the Cloud Dataproc versioning for more information.
• The message displayed after submitting a long-running operation in the GCP Console, such as creating or deleting a cluster, will now indicate the operation has been "submitted" rather than "has succeeded."

Cloud Dataproc 1.1 default – Cloud Dataproc 1.1 is now the default image version for new clusters.

Image versions released after Cloud Dataproc became generally available, starting with 1.0, will be subject to the Cloud Dataproc versioning policy.

Cloud SDK release 121.0.0 – Updated several gcloud dataproc arguments.

• The --preemptible-worker-boot-disk-size argument has been promoted to general availability and can be used to adjust the persistent disk size (in GB) of preemptible workers.
• The --master-boot-disk-size-gb and --worker-boot-disk-size-gb arguments have been removed. Use --master-boot-disk-size and --worker-boot-disk-size instead.