Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bigquery/storage/managedwriter/adapt): add RANGE support to adapt #9836

Merged
merged 4 commits into from
Apr 24, 2024

Conversation

shollyman
Copy link
Contributor

This PR adds support for schema conversions and proto conversions to the adapt subpackage. For proto conversion, we build a static name for range message types rather than using the normal first-encountered-message, as the schema for a range field will also be the consistent for a given element type.

This PR also moves adapt to using subtests for table-driven testing to make it easier to isolate specific test cases.

Towards: #9017

This PR adds support for schema conversions and proto conversions to the
adapt subpackage.  For proto conversion, we build a static name for range
message types rather than using the normal first-encountered-message,
as the schema for a range field will also be the consistent for a given
element type.

This PR also moves adapt to using subtests for table-driven testing to
make it easier to isolate specific test cases.
@shollyman shollyman requested review from a team as code owners April 18, 2024 21:45
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label Apr 18, 2024
@shollyman
Copy link
Contributor Author

The interesting question here is how to register the message types for the range values when converting a table schema to the descriptor proto. At a high level, there's three options:

  • publish a set of messages similar to the well-known types, and only add the reference. This makes compilation more awkward, as these types aren't general enough to warrant include in the common protobuf libraries, nor even in the google API type extensions.

  • [current implementation] generate type names independently of where they occur in the message hierarchy. This makes it easier to recognize a range message via the TypeName, since we lose the ability to recognize the RANGE purely from the descriptor.

  • inject range messages into the common dependency cache as we're walking the referenced messages. This consolidates the dependency cache a bit, but it means the TypeName of the message will be based on the first encountered reference (something like root.nested_message.range_field). It also means that any user messages with a similar two-field schema (start/end) could potentially be re-used in the cache.

@shollyman shollyman added the automerge Merge the pull request once unit tests and other checks pass. label Apr 24, 2024
@gcf-merge-on-green gcf-merge-on-green bot merged commit ae25253 into googleapis:main Apr 24, 2024
8 checks passed
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Apr 24, 2024
@shollyman shollyman deleted the more-write-work branch April 24, 2024 19:49
gcf-merge-on-green bot pushed a commit that referenced this pull request Apr 24, 2024
🤖 I have created a release *beep* *boop*
---


## [1.61.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.60.0...bigquery/v1.61.0) (2024-04-24)


### Features

* **bigquery/storage/managedwriter/adapt:** Add RANGE support to adapt ([#9836](https://togithub.com/googleapis/google-cloud-go/issues/9836)) ([ae25253](https://togithub.com/googleapis/google-cloud-go/commit/ae252533375b21dd41a0ea13e85462bbcad291af))
* **bigquery:** RANGE support for basic data movement ([#9762](https://togithub.com/googleapis/google-cloud-go/issues/9762)) ([07f0806](https://togithub.com/googleapis/google-cloud-go/commit/07f0806a945c2cf0fbc431b63d9c8a30ed3a22fd))
* **bigquery:** RANGE support when reading Arrow format ([#9795](https://togithub.com/googleapis/google-cloud-go/issues/9795)) ([da245fa](https://togithub.com/googleapis/google-cloud-go/commit/da245fac5ee335e86c63bfa5f165b0ab84960847))
* **bigquery:** RANGE type StandardSQLDataType support ([#9754](https://togithub.com/googleapis/google-cloud-go/issues/9754)) ([33666cf](https://togithub.com/googleapis/google-cloud-go/commit/33666cfeaefdebc474045894af069ca7172e836b))


### Bug Fixes

* **bigquery/datatransfer:** Mark parent/name fields with the REQUIRED field_behavior annotation ([8892943](https://togithub.com/googleapis/google-cloud-go/commit/8892943b169060f8ba7be227cd65680696c494a0))
* **bigquery:** Bump x/net to v0.24.0 ([ba31ed5](https://togithub.com/googleapis/google-cloud-go/commit/ba31ed5fda2c9664f2e1cf972469295e63deb5b4))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants