feat(bigquery-jdbc): add EnableProjectDiscovery connection property for metadata methods#13344
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces an EnableProjectDiscovery configuration property to automatically discover and list all accessible Google Cloud projects as catalogs. To support this, schema fetching in BigQueryDatabaseMetaData has been refactored to run concurrently using an executor service. Feedback on these changes highlights two main areas for improvement: first, replacing the fragile use of reflection to access the low-level BigQuery client with the standard public BigQuery.listProjects() API; second, ensuring that outstanding asynchronous tasks are properly cancelled if the schema fetching loop is interrupted to prevent resource leaks.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via a new EnableProjectDiscovery connection property. It also updates BigQueryDatabaseMetaData to parallelize schema fetching across multiple projects. The review feedback highlights several areas for improvement: reusing a shared ExecutorService instead of creating a new one on every getSchemas() call, avoiding caching empty results on transient exceptions in getDiscoveredProjects(), using GsonFactory.getDefaultInstance() for better resource reuse, and preserving the stack trace when logging execution exceptions.
There was a problem hiding this comment.
I'd suggest some refactoring for this method, few ideas to simplify:
- Move code to fetch list of schemas in a specific catalog to a separate function;
- Keep this method simple - single catalog only, essentially calls into helper & transforms to jsonResultSet. No need for background threads since it is single catalog
- Refactor
getSchemas()to be the one that fans out multiple requests & assembles data.
Probably breaking up CLs in 2 will be easier:
- Add proper threadPool to connection-layer (and remove static threadPool in statement, it is not used)
- update getSchemas
Also I'd suggest to reuse more code. There are 4 metadata methods that can generate large # of rest calls:
- getCatalogs()
- getSchemas()
- getTables()
- getColumns()
- getProcedures()
In a way, they build results on top of each other. Right now we duplicate a lot, e.g. listDatasets() is called in 7 different places.
Also please don't tackle all of them in a single CL, lets do it incrementally :)
(Otherwise it is pain to review)
There was a problem hiding this comment.
Yeah, makes sense. I have created 3 bugs:
- b/520400589 - Refactor Metadata Thread Pool Management to Reuse Connection-Scoped Executor
- b/520407325 - Refactor getSchemas for Catalog-Based Routing (Synchronous & Async Fan-out)
- b/520406763 - Deduplicate metadata API calls
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via the EnableProjectDiscovery connection property. The implementation includes caching of discovered projects and error handling for transient and non-transient BigQuery exceptions. The review feedback correctly identifies compilation errors in BigQueryConnection.java where invalid LOG.warning(e, ...) calls are used instead of LOG.log(Level.WARNING, ..., e).
5187968 to
47a47ac
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to enable the discovery of all accessible Google Cloud projects as catalogs via the new EnableProjectDiscovery property. Feedback on the changes highlights that non-transient errors (e.g., HTTP 400, 401, 403) should cache an empty list to prevent redundant failing calls, and corresponding unit tests should be updated to verify this behavior. Additionally, it is recommended to avoid using getMaxResults() as the page size when listing projects, as it may be configured with values inappropriate for project listing.
… for metadata methods (#13344) b/499078725 This PR implements the `EnableProjectDiscovery` connection property into the BigQuery JDBC driver. Enabling this property (default `false`) allows JDBC database metadata methods (like `getCatalogs()` and `getSchemas()`) to discover and query datasets across all Google Cloud projects accessible to the client credentials, rather than being confined to the single default `ProjectId` specified in the connection URL. #### Changes - **Connection Parameter Parsing**: Added parsing for `EnableProjectDiscovery` in `BigQueryJdbcUrlUtility` and configured it in `DataSource`. - **SDK Integration**: Implemented `BigQueryConnection.getDiscoveredProjects()` to utilize the `BigQuery.listProjects()` core SDK method - **Resilience and Caching**: - Connection-scoped caching is implemented via `discoveredProjectsCache`. - **Metadata Integration**: Integrated discovered projects inside `BigQueryDatabaseMetaData.getAccessibleCatalogNames()` so that catalog and schema metadata fetches automatically query across all accessible projects when `EnableProjectDiscovery` is enabled.
🤖 I have created a release *beep* *boop* --- <details><summary>1.88.0</summary> ## [1.88.0](v1.87.1...v1.88.0) (2026-07-06) ### Features * **auth:** Add support for Regional Access Boundaries ([#13499](#13499)) ([b721e43](b721e43)) * automate libraries-bom tag and release creation on release publish ([#13655](#13655)) ([7a7e2ff](7a7e2ff)) * **bigquery-jdbc:** add `EnableProjectDiscovery` connection property for metadata methods ([#13344](#13344)) ([ab9669a](ab9669a)) * **bigquery-jdbc:** migrate statement execution thread tracking to connection-scoped executor ([#13454](#13454)) ([341e51a](341e51a)) * **bigquery-jdbc:** respect standard JVM trustStore properties by default ([#13435](#13435)) ([7af3224](7af3224)) * **bigquery-jdbc:** support connection-scoped executor isolation and dynamic queue warnings ([#13413](#13413)) ([79e26b8](79e26b8)) * **bigquery:** add internal listProjects API to core client ([#13429](#13429)) ([3580407](3580407)) * **bigtable:** route point read rows to shim ([#13542](#13542)) ([2c7e5f5](2c7e5f5)) * **deps:** add jspecify to shared dependencies ([#13412](#13412)) ([582053d](582053d)) * enable self-signed JWTs by default in ServiceOptions ([#13338](#13338)) ([110d2b7](110d2b7)) * **gapic-generator:** Add NullMarked annotation to generated classes ([#13517](#13517)) ([825dadd](825dadd)) * **google/cloud/agentregistry/v1:** onboard a new library ([#13509](#13509)) ([6728816](6728816)) * scale up connection worker pool based on latency ([#13384](#13384)) ([eb379b3](eb379b3)) * **spanner:** add settings for gRCP keep-alive ([#13643](#13643)) ([9d78307](9d78307)) * **spanner:** auth login support for Spanner Omni endpoints ([#13470](#13470)) ([55930b4](55930b4)) * **storage:** add checksum validation in the json read channel ([#13270](#13270)) ([872d7b7](872d7b7)) * **storage:** add checksum validation on json read paths ([#13269](#13269)) ([396b042](396b042)) * **storage:** add full object checksum validation for bidi flow ([#13266](#13266)) ([9113d80](9113d80)) * **storage:** add full object checksum validation for grpc flow ([#13265](#13265)) ([a5ba606](a5ba606)) * **storage:** Adding CumulativeHasher wrapper class for Full object … ([#13239](#13239)) ([bd40324](bd40324)) * **storage:** adding disabling option for bidi reads ([#13506](#13506)) ([591cae0](591cae0)) * **storage:** log additional bytes received from GCS in read path ([#13427](#13427)) ([9492aa2](9492aa2)) * **storage:** update compose sample to support deleteSourceObjects option ([#13493](#13493)) ([50a8658](50a8658)) * use self-signed JWTs in Spanner MutableCredentials ([#13381](#13381)) ([29543a2](29543a2)) ### Bug Fixes * Add logging-logback to gapic-libraries-bom ([#13663](#13663)) ([512e370](512e370)) * **auth:** Fix UserCredentials serialization clientSecret leak ([#13465](#13465)) ([2d9d01c](2d9d01c)) * **auth:** handle missing APPDATA on Windows gracefully ([#13471](#13471)) ([77e84a9](77e84a9)), closes [#12565](#12565) * **bigquery-jdbc:** add proper version to BigQueryDriver ([#13294](#13294)) ([9fd84fc](9fd84fc)) * **bigquery-jdbc:** avoid rollback on statement close in manual commit mode ([#13503](#13503)) ([79bbee1](79bbee1)) * **bigquery-jdbc:** correct `FilterTablesOnDefaultDataset` fallback logic ([#13625](#13625)) ([f7eb23d](f7eb23d)) * **bigquery-jdbc:** ensure largeResults are handled in PreparedStatement ([#13496](#13496)) ([d3e7a19](d3e7a19)) * **bigquery-jdbc:** ensure test uses unique dataset & cleans up ([#13587](#13587)) ([a6bb362](a6bb362)) * **bigquery-jdbc:** explicitly set location when available for temporary dataset ([#13508](#13508)) ([40cd0a7](40cd0a7)) * **bigquery-jdbc:** propagate connection proxy settings to auth library ([#13539](#13539)) ([87dda5d](87dda5d)) * **bigquery-jdbc:** shade org.slf4j ([#13547](#13547)) ([7ef1312](7ef1312)) * **bigquery-jdbc:** update format validation tests to create tables for the test ([#13588](#13588)) ([4bb3a61](4bb3a61)) * **bigquery-jdbc:** update Maven Central artifact link in the README ([#13563](#13563)) ([8707a65](8707a65)) * **bigquery:** retry cancel job and routine creation on transient HTTP 5xx errors ([#13416](#13416)) ([35eb041](35eb041)) * **bigquery:** route JOB_CREATION_REQUIRED through fast query path ([#13437](#13437)) ([64cde7f](64cde7f)) * **bigquery:** Update fast query path to allow jobTimeoutMs in the request ([#13502](#13502)) ([1a6f4d5](1a6f4d5)) * **bigtable:** honour session_load override when server-returned session_load is 0 ([#13629](#13629)) ([b56f9b8](b56f9b8)) * **bqjdbc:** pass rowsInPage with TableResult ([#13238](#13238)) ([203a91e](203a91e)) * **ci:** build dependencies before convergence check ([#13638](#13638)) ([9de3209](9de3209)) * **ci:** ignore SNAPSHOT suffix in flatten-plugin check ([#13639](#13639)) ([bb65627](bb65627)) * **ci:** run root install in dependency convergence check ([#13666](#13666)) ([86d35e4](86d35e4)) * **ci:** skip existing-versions-check for gapic-generator-java-root ([#13590](#13590)) ([a1f3b8a](a1f3b8a)) * **datastore:** Configure TraceExporter with Datastore credentials in E2E tests ([#13627](#13627)) ([d281a62](d281a62)) * **datastore:** disable built-in OpenTelemetry SDK when metrics export is disabled ([#13543](#13543)) ([53b7142](53b7142)) * **datastore:** reduce flakiness of E2E tracing tests ([#13645](#13645)) ([8afc0ab](8afc0ab)) * **deps:** update dependency com.google.apis:google-api-services-bigquery to v2-rev20260612-2.0.0 ([#13570](#13570)) ([ca5ff79](ca5ff79)) * **deps:** update dependency com.google.apis:google-api-services-dns to v1-rev20260616-2.0.0 ([#12796](#12796)) ([0dd60ca](0dd60ca)) * **deps:** update dependency com.google.cloud:google-cloud-pubsub-bom to v1.151.0 ([#12097](#12097)) ([d169006](d169006)) * **deps:** update dependency com.google.cloud:libraries-bom to v26.84.0 ([#12663](#12663)) ([95c7774](95c7774)) * **deps:** update first-party storage dependencies to v1-rev20260524-2.0.0 ([#13406](#13406)) ([bdeedb8](bdeedb8)) * exclude java-vertexai from existing versions check ([#13632](#13632)) ([edaa4e3](edaa4e3)) * fallback on VPC ([#13567](#13567)) ([e8c428d](e8c428d)) * **java-cloud-bom:** execute pre-install pass in GHA release notes generator ([#13608](#13608)) ([c365c10](c365c10)) * **release:** check remote tags using git ls-remote to prevent already exists error ([#13574](#13574)) ([6cf0567](6cf0567)) * **spanner:** fail fast when inline-begin DML returns no transaction id ([#13536](#13536)) ([94315ce](94315ce)) * **spanner:** pin inline read-write transactions to default endpoint ([#13562](#13562)) ([6908dee](6908dee)) * **spanner:** remove explicit tink version to resolve dependency convergence conflict ([#13602](#13602)) ([863c26e](863c26e)) * **spanner:** update dynamic channel pool default configuration ([#13646](#13646)) ([37b77c3](37b77c3)) ### Dependencies * Add org.bouncycastle:bcprov-jdk18on to java-shared-dependencies ([#13531](#13531)) ([e3d2082](e3d2082)) * **auth:** Update commons-codec to v1.18.0 ([#13598](#13598)) ([fd89957](fd89957)) * Update gapic-showcase to v0.41.0 ([#13582](#13582)) ([09faaac](09faaac)) * Upgrade google-http-client to v2.1.1 ([#13578](#13578)) ([6abef19](6abef19)) ### Documentation * add instructions for manual code generation checks in CI ([#13596](#13596)) ([f6162e2](f6162e2)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
b/499078725
This PR implements the
EnableProjectDiscoveryconnection property into the BigQuery JDBC driver. Enabling this property (defaultfalse) allows JDBC database metadata methods (likegetCatalogs()andgetSchemas()) to discover and query datasets across all Google Cloud projects accessible to the client credentials, rather than being confined to the single defaultProjectIdspecified in the connection URL.Changes
EnableProjectDiscoveryinBigQueryJdbcUrlUtilityand configured it inDataSource.BigQueryConnection.getDiscoveredProjects()to utilize theBigQuery.listProjects()core SDK methoddiscoveredProjectsCache.BigQueryDatabaseMetaData.getAccessibleCatalogNames()so that catalog and schema metadata fetches automatically query across all accessible projects whenEnableProjectDiscoveryis enabled.