Cluster properties

Apache Hadoop YARN, HDFS, Spark, and related properties

The open source components installed on Dataproc clusters contain many configuration files. For example, Apache Spark and Apache Hadoop have several XML and plain text configuration files. You can use the ‑‑properties flag of the gcloud dataproc clusters create command to modify many common configuration files when creating a cluster.

Formatting

The gcloud dataproc clusters create --properties flag accepts the following string format:

file_prefix1:property1=value1,file_prefix2:property2=value2,...

• The file_prefix maps to a predefined configuration file as shown in the table below, and the property maps to a property within the file.

• The default delimiter used to separate multiple cluster properties is the comma (,). However, if a comma is included in a property value, you must change the delimiter by specifying a "^delimiter^" at the beginning of the property list (see gcloud topic escaping for more information).

  • Example using a "#" delimiter:

    --properties ^#^file_prefix1:property1=part1,part2#file_prefix2:property2=value2
    

Examples

--properties 'spark:spark.master=spark://example.com'

--properties 'spark:spark.master=spark://example.com,hdfs:dfs.hosts=/foo/bar/baz'

"properties": {
  "spark:spark.executor.memory": "10g"
}

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --properties=spark:spark.executor.memory=10g \
    --log-http \
    other args ...

...
== body start ==
{"clusterName": "my-cluster", "config": {"gceClusterConfig": ...
"masterConfig": {... "softwareConfig": {"properties": {"spark:spark.executor.memory": "10g"}},

...
== body end ==
...

Cluster vs. job properties

The Apache Hadoop YARN, HDFS, Spark, and other file-prefixed properties are applied at the cluster level when you create a cluster. These properties cannot be applied to a cluster after cluster creation. However, many of these properties can also be applied to specific jobs. When applying a property to a job, the file prefix is not used.

The following example sets Spark executor memory to 4g for a Spark job (spark: prefix omitted).

gcloud dataproc jobs submit spark \
    --region=REGION \
    --properties=spark.executor.memory=4g \
    other args ...

Job properties can be submitted in a file using the gcloud dataproc jobs submit job-type --properties-file flag (see, for example, the --properties-file description for an Hadoop job submission).

gcloud dataproc jobs submit JOB_TYPE \
    --region=REGION \
    --properties-file=PROPERTIES_FILE \
    other args ...

The PROPERTIES_FILE is a set of line-delimited key=value pairs. The property to be set is the key, and the value to set the property to is the value. See the java.util.Properties class for a detailed description of the properties file format.

The following is an example of a properties file that can be passed to the --properties-file flag when submitting a Dataproc job.

 dataproc:conda.env.config.uri=gs://some-bucket/environment.yaml
 spark:spark.history.fs.logDirectory=gs://some-bucket
 spark:spark.eventLog.dir=gs://some-bucket
 capacity-scheduler:yarn.scheduler.capacity.root.adhoc.capacity=5

File-prefixed properties table

Notes

• Some properties are reserved and cannot be overridden because they impact the functionality of the Dataproc cluster. If you try to change a reserved property, you will receive an error message when creating your cluster.
• You can specify multiple changes by separating each with a comma.
• The --properties flag cannot modify configuration files not shown above.
• Changes to properties will be applied before the daemons on your cluster start.
• If the specified property exists, it will be updated. If the specified property does not exist, it will be added to the configuration file.

Dataproc service properties

The properties listed in this section are specific to Dataproc. These properties can be used to further configure the functionality of your Dataproc cluster.

Formatting

The gcloud dataproc clusters create --properties flag accepts the following string format:

property_prefix1:property1=value1,property_prefix2:property2=value2,...

• The default delimiter used to separate multiple cluster properties is the comma (,). However, if a comma is included in a property value, you must change the delimiter by specifying "^delimiter^" at the beginning of the property list (see gcloud topic escaping for more information).

  • Example using a "#" delimiter:

    --properties ^#^property_prefix1:property1=part1,part2#property_prefix2:property2=value2
    

Example:

Create a cluster and set Enhanced Flexibility Mode to Spark primary worker shuffle.

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --properties=dataproc:efm.spark.shuffle=primary-worker \
    other args ...

Dataproc service properties table

¹Keystore file: The keystore file contains the SSL certificate. It should be in Java KeyStore (JKS) format. When copied to VMs, it is renamed to keystore.jks. The SSL certificate should be a wildcard certificate that applies to each node in the cluster.

²Truststore file: The truststore file should be in Java KeyStore (JKS) format. When copied to VMs, it is renamed to truststore.jks.