[Feature](iceberg) Support Iceberg JDBC Catalog #59502
Open
+789
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Pull request overview
This PR adds support for Iceberg JDBC Catalog, enabling metadata storage in relational databases (PostgreSQL, MySQL, SQLite) as an alternative to HMS or other catalog types. The implementation follows the existing pattern used by other Iceberg catalog types and integrates seamlessly with Doris's catalog framework.
Key Changes:
- Added JDBC catalog type with comprehensive property handling and S3-compatible storage support
- Integrated JDBC catalog into factory registration and serialization infrastructure
- Added unit tests for property validation and JDBC-specific configuration passthrough
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| IcebergJdbcMetaStoreProperties.java | Core implementation handling JDBC catalog properties, URI configuration, and storage integration |
| IcebergJdbcExternalCatalog.java | External catalog class following standard pattern for JDBC catalog type |
| IcebergJdbcMetaStorePropertiesTest.java | Unit tests validating property handling, passthrough, and required field checks |
| IcebergPropertiesFactory.java | Registered "jdbc" catalog type in the factory |
| IcebergExternalCatalogFactory.java | Added JDBC catalog creation in the factory switch statement |
| GsonUtils.java | Registered IcebergJdbcExternalCatalog for JSON serialization |
| IcebergScanNode.java | Added JDBC catalog type to supported scan sources |
| IcebergExternalCatalog.java | Added ICEBERG_JDBC constant definition |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
157
to
191
| private static void toFileIOProperties(List<StorageProperties> storagePropertiesList, | ||
| Map<String, String> fileIOProperties, Configuration conf) { | ||
| for (StorageProperties storageProperties : storagePropertiesList) { | ||
| if (storageProperties instanceof AbstractS3CompatibleProperties) { | ||
| toS3FileIOProperties((AbstractS3CompatibleProperties) storageProperties, fileIOProperties); | ||
| } else if (storageProperties.getHadoopStorageConfig() != null) { | ||
| conf.addResource(storageProperties.getHadoopStorageConfig()); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private static void toS3FileIOProperties(AbstractS3CompatibleProperties s3Properties, | ||
| Map<String, String> options) { | ||
| // Set S3FileIO as the FileIO implementation for S3-compatible storage | ||
| options.put(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.aws.s3.S3FileIO"); | ||
|
|
||
| if (StringUtils.isNotBlank(s3Properties.getEndpoint())) { | ||
| options.put(S3FileIOProperties.ENDPOINT, s3Properties.getEndpoint()); | ||
| } | ||
| if (StringUtils.isNotBlank(s3Properties.getUsePathStyle())) { | ||
| options.put(S3FileIOProperties.PATH_STYLE_ACCESS, s3Properties.getUsePathStyle()); | ||
| } | ||
| if (StringUtils.isNotBlank(s3Properties.getRegion())) { | ||
| options.put(AwsClientProperties.CLIENT_REGION, s3Properties.getRegion()); | ||
| } | ||
| if (StringUtils.isNotBlank(s3Properties.getAccessKey())) { | ||
| options.put(S3FileIOProperties.ACCESS_KEY_ID, s3Properties.getAccessKey()); | ||
| } | ||
| if (StringUtils.isNotBlank(s3Properties.getSecretKey())) { | ||
| options.put(S3FileIOProperties.SECRET_ACCESS_KEY, s3Properties.getSecretKey()); | ||
| } | ||
| if (StringUtils.isNotBlank(s3Properties.getSessionToken())) { | ||
| options.put(S3FileIOProperties.SESSION_TOKEN, s3Properties.getSessionToken()); | ||
| } | ||
| } |
There was a problem hiding this comment.
The methods toFileIOProperties and toS3FileIOProperties are duplicated from IcebergRestProperties with only minor differences (the problematic FILE_IO_IMPL line). Consider extracting these common methods to the AbstractIcebergProperties base class to eliminate code duplication and ensure consistency across all Iceberg catalog implementations. This would make maintenance easier and prevent similar issues in the future.
Comment on lines
170
to
172
| // Set S3FileIO as the FileIO implementation for S3-compatible storage | ||
| options.put(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.aws.s3.S3FileIO"); | ||
|
|
There was a problem hiding this comment.
Setting FILE_IO_IMPL unconditionally to S3FileIO when S3 storage is detected may break functionality for non-S3 storage types. Looking at IcebergRestProperties, the toS3FileIOProperties method does not set FILE_IO_IMPL and relies on Iceberg's automatic FileIO selection via CatalogUtil.buildIcebergCatalog. This line should be removed to maintain consistency with other catalog implementations and allow Iceberg to handle FileIO selection automatically based on the storage properties provided.
Suggested change
| // Set S3FileIO as the FileIO implementation for S3-compatible storage | |
| options.put(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.aws.s3.S3FileIO"); |
morningman
reviewed
Jan 4, 2026
| private String uri = ""; | ||
|
|
||
| @ConnectorProperty( | ||
| names = {"jdbc.user"}, |
Contributor
There was a problem hiding this comment.
How about adding iceberg. prefix to all properties?
| private String jdbcStrictMode; | ||
|
|
||
| @ConnectorProperty( | ||
| names = {"driver_url"}, |
Contributor
There was a problem hiding this comment.
Suggested change
| names = {"driver_url"}, | |
| names = {"iceberg.driver-url"}, |
I know it is diff from JDBC catalog's prop, but we need to keep name consistency within this iceberg jdbc catalog.
| private String driverUrl; | ||
|
|
||
| @ConnectorProperty( | ||
| names = {"driver_class"}, |
Contributor
There was a problem hiding this comment.
Suggested change
| names = {"driver_class"}, | |
| names = {"iceberg.driver-class"}, |
| private String jdbcPassword; | ||
|
|
||
| @ConnectorProperty( | ||
| names = {"jdbc.init-catalog-tables"}, |
Contributor
There was a problem hiding this comment.
For init-catalog-tables, schema-version and strict-mode, do we have default values for these config?
Contributor
Author
There was a problem hiding this comment.
yes,init-catalog-tables=true,schema-version=v0,strict-mode=false
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31726 ms |
TPC-DS: Total hot run time: 172056 ms |
ClickBench: Total hot run time: 26.96 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 32120 ms |
TPC-DS: Total hot run time: 172679 ms |
ClickBench: Total hot run time: 26.69 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32014 ms |
TPC-DS: Total hot run time: 175551 ms |
ClickBench: Total hot run time: 26.79 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31461 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-DS: Total hot run time: 172303 ms |
ClickBench: Total hot run time: 27.81 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Currently, Doris supports multiple Iceberg catalog types (HMS, REST, Hadoop, Glue, DLF, S3Tables) but lacks support for JDBC Catalog. This PR adds support for Iceberg JDBC Catalog, which allows users to store Iceberg metadata in relational databases like PostgreSQL, MySQL, and SQLite.
Key Changes:
IcebergJdbcMetaStorePropertiesclass to handle JDBC catalog configurationsIcebergJdbcExternalCatalogclass for JDBC catalog operationsBenefits:
Release note
Features
Check List (For Author)
Manual Test Steps:
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)