Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(@aws-cdk/aws-glue-alpha): Add partition_filtering.enabled support #20825

Closed
1 of 2 tasks
atsaki opened this issue Jun 22, 2022 · 3 comments · Fixed by #21081
Closed
1 of 2 tasks

(@aws-cdk/aws-glue-alpha): Add partition_filtering.enabled support #20825

atsaki opened this issue Jun 22, 2022 · 3 comments · Fixed by #21081
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. good first issue Related to contributions. See CONTRIBUTING.md p2

Comments

@atsaki
Copy link

atsaki commented Jun 22, 2022

Describe the feature

Table construct support partition indexes. However, the feature to set partition_filtering.enabled: true to enable partition filtering is missing.

https://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.html#glue-best-practices-partition-index

It would be nice if Table construct provide the way to set partition_filtering.enabled property.

Use Case

I always manually add partition_filtering.enabled: true after creating the table.
It would helpful if I can add the property with Table construct.

Proposed Solution

How about adding a partitionFilteringEnable prop to TableProps and setting parameters like classification and has_encrypted_data?

        parameters: {
          classification: props.dataFormat.classificationString?.value,
          has_encrypted_data: this.encryption !== TableEncryption.UNENCRYPTED,
          'partition_filtering.enabled': props.partitionFilteringEnabled,
        },

aws-cdk/packages/@aws-cdk/aws-glue/lib/table.ts

Lines 304 to 307 in 74318c7

parameters: {
classification: props.dataFormat.classificationString?.value,
has_encrypted_data: this.encryption !== TableEncryption.UNENCRYPTED,
},

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.28.1

Environment details (OS name and version, etc.)

Windows
@atsaki atsaki added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jun 22, 2022
@github-actions github-actions bot added the @aws-cdk/aws-glue Related to AWS Glue label Jun 22, 2022
@kaizencc kaizencc added p2 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Jun 30, 2022
@kaizencc kaizencc removed their assignment Jun 30, 2022
@kaizencc
Copy link
Contributor

The proposed solution is good. I would be happy to take a contribution on this. I'd call the prop enablePartitionFiltering. @atsaki do you want to work on this?

@kaizencc kaizencc added the good first issue Related to contributions. See CONTRIBUTING.md label Jun 30, 2022
@atsaki
Copy link
Author

atsaki commented Jul 2, 2022

@kaizencc Yes, I would like to try it!

@daschaa daschaa mentioned this issue Jul 10, 2022
Merged
4 tasks
@mergify mergify bot closed this as completed in #21081 Jul 12, 2022
mergify bot pushed a commit that referenced this issue Jul 12, 2022
@daschaa
feat(glue): enable partition filtering on tables (#21081)
bf35048 
Fixes #20825

Adds `partition_filtering.enabled` to `TableProps` in AWS Glue.

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/small Small work item – less than a day of effort feature-request A feature should be added or improved. good first issue Related to contributions. See CONTRIBUTING.md p2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(glue): enable partition filtering on tables daschaa/aws-cdk
2 participants
@atsaki @kaizencc