Skip to content

feat: [dataproc] add support for new Dataproc features #9127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
diegomarquezp merged 8 commits into from
Feb 28, 2023
Merged

feat: [dataproc] add support for new Dataproc features #9127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9ad0cda
fix: Add service_yaml_parameters to py_gapic_library BUILD.bazel targets
gcf-owl-bot[bot] Feb 16, 2023
225ed31
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 16, 2023
be86a86
feat: add support for new Dataproc features
gcf-owl-bot[bot] Feb 17, 2023
889506f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 17, 2023
16cc129
feat: ROLLBACK: add support for new Dataproc features
gcf-owl-bot[bot] Feb 21, 2023
90f90d9
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 21, 2023
be21529
feat: add support for new Dataproc features
gcf-owl-bot[bot] Feb 22, 2023
23bf753
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions java-dataproc/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,20 @@ If you are using Maven, add this to your pom.xml file:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-dataproc</artifactId>
<version>4.6.0</version>
<version>4.8.0</version>
</dependency>
```

If you are using Gradle without BOM, add this to your dependencies:

```Groovy
implementation 'com.google.cloud:google-cloud-dataproc:4.6.0'
implementation 'com.google.cloud:google-cloud-dataproc:4.8.0'
```

If you are using SBT, add this to your dependencies:

```Scala
libraryDependencies += "com.google.cloud" % "google-cloud-dataproc" % "4.6.0"
libraryDependencies += "com.google.cloud" % "google-cloud-dataproc" % "4.8.0"
```

## Authentication
Expand Down
18 changes: 14 additions & 4 deletions ...ogle-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/BatchControllerClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,8 @@ public final UnaryCallable<CreateBatchRequest, Operation> createBatchCallable()
* }
* }</pre>
*
* @param name Required. The name of the batch to retrieve.
* @param name Required. The fully qualified name of the batch to retrieve in the format
* "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID"
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final Batch getBatch(BatchName name) {
Expand All @@ -421,7 +422,8 @@ public final Batch getBatch(BatchName name) {
* }
* }</pre>
*
* @param name Required. The name of the batch to retrieve.
* @param name Required. The fully qualified name of the batch to retrieve in the format
* "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID"
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final Batch getBatch(String name) {
Expand Down Expand Up @@ -561,6 +563,8 @@ public final ListBatchesPagedResponse listBatches(String parent) {
* .setParent(LocationName.of("[PROJECT]", "[LOCATION]").toString())
* .setPageSize(883849137)
* .setPageToken("pageToken873572522")
* .setFilter("filter-1274492040")
* .setOrderBy("orderBy-1207110587")
* .build();
* for (Batch element : batchControllerClient.listBatches(request).iterateAll()) {
* // doThingsWith(element);
Expand Down Expand Up @@ -593,6 +597,8 @@ public final ListBatchesPagedResponse listBatches(ListBatchesRequest request) {
* .setParent(LocationName.of("[PROJECT]", "[LOCATION]").toString())
* .setPageSize(883849137)
* .setPageToken("pageToken873572522")
* .setFilter("filter-1274492040")
* .setOrderBy("orderBy-1207110587")
* .build();
* ApiFuture<Batch> future =
* batchControllerClient.listBatchesPagedCallable().futureCall(request);
Expand Down Expand Up @@ -626,6 +632,8 @@ public final ListBatchesPagedResponse listBatches(ListBatchesRequest request) {
* .setParent(LocationName.of("[PROJECT]", "[LOCATION]").toString())
* .setPageSize(883849137)
* .setPageToken("pageToken873572522")
* .setFilter("filter-1274492040")
* .setOrderBy("orderBy-1207110587")
* .build();
* while (true) {
* ListBatchesResponse response = batchControllerClient.listBatchesCallable().call(request);
Expand Down Expand Up @@ -665,7 +673,8 @@ public final UnaryCallable<ListBatchesRequest, ListBatchesResponse> listBatchesC
* }
* }</pre>
*
* @param name Required. The name of the batch resource to delete.
* @param name Required. The fully qualified name of the batch to retrieve in the format
* "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID"
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final void deleteBatch(BatchName name) {
Expand Down Expand Up @@ -693,7 +702,8 @@ public final void deleteBatch(BatchName name) {
* }
* }</pre>
*
* @param name Required. The name of the batch resource to delete.
* @param name Required. The fully qualified name of the batch to retrieve in the format
* "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID"
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final void deleteBatch(String name) {
Expand Down
36 changes: 18 additions & 18 deletions ...proc/google-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/gapic_metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,24 @@
}
}
},
"NodeGroupController": {
"clients": {
"grpc": {
"libraryClient": "NodeGroupControllerClient",
"rpcs": {
"CreateNodeGroup": {
"methods": ["createNodeGroupAsync", "createNodeGroupAsync", "createNodeGroupAsync", "createNodeGroupOperationCallable", "createNodeGroupCallable"]
},
"GetNodeGroup": {
"methods": ["getNodeGroup", "getNodeGroup", "getNodeGroup", "getNodeGroupCallable"]
},
"ResizeNodeGroup": {
"methods": ["resizeNodeGroupAsync", "resizeNodeGroupAsync", "resizeNodeGroupOperationCallable", "resizeNodeGroupCallable"]
}
}
}
}
},
"WorkflowTemplateService": {
"clients": {
"grpc": {
Expand Down Expand Up @@ -142,24 +160,6 @@
}
}
}
},
"NodeGroupController": {
"clients": {
"grpc": {
"libraryClient": "NodeGroupControllerClient",
"rpcs": {
"CreateNodeGroup": {
"methods": ["createNodeGroupAsync", "createNodeGroupAsync", "createNodeGroupAsync", "createNodeGroupOperationCallable", "createNodeGroupCallable"]
},
"GetNodeGroup": {
"methods": ["getNodeGroup", "getNodeGroup", "getNodeGroup", "getNodeGroupCallable"]
},
"ResizeNodeGroup": {
"methods": ["resizeNodeGroupAsync", "resizeNodeGroupAsync", "resizeNodeGroupOperationCallable", "resizeNodeGroupCallable"]
}
}
}
}
}
}
}
32 changes: 16 additions & 16 deletions ...taproc/google-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/package-info.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,43 +97,43 @@
* }
* }</pre>
*
* <p>======================= WorkflowTemplateServiceClient =======================
* <p>======================= NodeGroupControllerClient =======================
*
* <p>Service Description: The API interface for managing Workflow Templates in the Dataproc API.
* <p>Service Description: The `NodeGroupControllerService` provides methods to manage node groups
* of Compute Engine managed instances.
*
* <p>Sample for WorkflowTemplateServiceClient:
* <p>Sample for NodeGroupControllerClient:
*
* <pre>{@code
* // This snippet has been automatically generated and should be regarded as a code template only.
* // It will require modifications to work:
* // - It may require correct/in-range values for request initialization.
* // - It may require specifying regional endpoints when creating the service client as shown in
* // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
* try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
* WorkflowTemplateServiceClient.create()) {
* LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
* WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
* WorkflowTemplate response =
* workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
* try (NodeGroupControllerClient nodeGroupControllerClient = NodeGroupControllerClient.create()) {
* NodeGroupName name = NodeGroupName.of("[PROJECT]", "[REGION]", "[CLUSTER]", "[NODE_GROUP]");
* NodeGroup response = nodeGroupControllerClient.getNodeGroup(name);
* }
* }</pre>
*
* <p>======================= NodeGroupControllerClient =======================
* <p>======================= WorkflowTemplateServiceClient =======================
*
* <p>Service Description: The `NodeGroupControllerService` provides methods to manage node groups
* of Compute Engine managed instances.
* <p>Service Description: The API interface for managing Workflow Templates in the Dataproc API.
*
* <p>Sample for NodeGroupControllerClient:
* <p>Sample for WorkflowTemplateServiceClient:
*
* <pre>{@code
* // This snippet has been automatically generated and should be regarded as a code template only.
* // It will require modifications to work:
* // - It may require correct/in-range values for request initialization.
* // - It may require specifying regional endpoints when creating the service client as shown in
* // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
* try (NodeGroupControllerClient nodeGroupControllerClient = NodeGroupControllerClient.create()) {
* NodeGroupName name = NodeGroupName.of("[PROJECT]", "[REGION]", "[CLUSTER]", "[NODE_GROUP]");
* NodeGroup response = nodeGroupControllerClient.getNodeGroup(name);
* try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
* WorkflowTemplateServiceClient.create()) {
* LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
* WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
* WorkflowTemplate response =
* workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
* }
* }</pre>
*/
Expand Down
2 changes: 2 additions & 0 deletions ...dataproc/src/main/java/com/google/cloud/dataproc/v1/stub/HttpJsonBatchControllerStub.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,8 @@ public class HttpJsonBatchControllerStub extends BatchControllerStub {
Map<String, List<String>> fields = new HashMap<>();
ProtoRestSerializer<ListBatchesRequest> serializer =
ProtoRestSerializer.create();
serializer.putQueryParam(fields, "filter", request.getFilter());
serializer.putQueryParam(fields, "orderBy", request.getOrderBy());
serializer.putQueryParam(fields, "pageSize", request.getPageSize());
serializer.putQueryParam(fields, "pageToken", request.getPageToken());
serializer.putQueryParam(fields, "$alt", "json;enum-encoding=int");
Expand Down
12 changes: 8 additions & 4 deletions ...d-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/WorkflowTemplateServiceGrpc.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,8 @@ public void instantiateWorkflowTemplate(
* <pre>
* Instantiates a template and begins execution.
* This method is equivalent to executing the sequence
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate], [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate],
* [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [DeleteWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.DeleteWorkflowTemplate].
* The returned Operation can be used to track execution of
* workflow by polling
Expand Down Expand Up @@ -734,7 +735,8 @@ public void instantiateWorkflowTemplate(
* <pre>
* Instantiates a template and begins execution.
* This method is equivalent to executing the sequence
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate], [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate],
* [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [DeleteWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.DeleteWorkflowTemplate].
* The returned Operation can be used to track execution of
* workflow by polling
Expand Down Expand Up @@ -897,7 +899,8 @@ public com.google.longrunning.Operation instantiateWorkflowTemplate(
* <pre>
* Instantiates a template and begins execution.
* This method is equivalent to executing the sequence
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate], [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate],
* [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [DeleteWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.DeleteWorkflowTemplate].
* The returned Operation can be used to track execution of
* workflow by polling
Expand Down Expand Up @@ -1049,7 +1052,8 @@ protected WorkflowTemplateServiceFutureStub build(
* <pre>
* Instantiates a template and begins execution.
* This method is equivalent to executing the sequence
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate], [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [CreateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.CreateWorkflowTemplate],
* [InstantiateWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.InstantiateWorkflowTemplate],
* [DeleteWorkflowTemplate][google.cloud.dataproc.v1.WorkflowTemplateService.DeleteWorkflowTemplate].
* The returned Operation can be used to track execution of
* workflow by polling
Expand Down
42 changes: 21 additions & 21 deletions ...oogle-cloud-dataproc-v1/src/main/java/com/google/cloud/dataproc/v1/AcceleratorConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,10 @@ public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down Expand Up @@ -115,10 +115,10 @@ public java.lang.String getAcceleratorTypeUri() {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down Expand Up @@ -535,10 +535,10 @@ public Builder mergeFrom(
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down Expand Up @@ -569,10 +569,10 @@ public java.lang.String getAcceleratorTypeUri() {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down Expand Up @@ -603,10 +603,10 @@ public com.google.protobuf.ByteString getAcceleratorTypeUriBytes() {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down Expand Up @@ -636,10 +636,10 @@ public Builder setAcceleratorTypeUri(java.lang.String value) {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand All @@ -665,10 +665,10 @@ public Builder clearAcceleratorTypeUri() {
* Full URL, partial URI, or short name of the accelerator type resource to
* expose to this instance. See
* [Compute Engine
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/beta/acceleratorTypes).
* AcceleratorTypes](https://cloud.google.com/compute/docs/reference/v1/acceleratorTypes).
* Examples:
* * `https://www.googleapis.com/compute/beta/projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/us-east1-a/acceleratorTypes/nvidia-tesla-k80`
* * `https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80`
* * `nvidia-tesla-k80`
* **Auto Zone Exception**: If you are using the Dataproc
* [Auto Zone
Expand Down
Loading