41 Pull requests merged by 19 people

• Flink: Test both "new" Flink Avro planned reader and "deprecated" Avro reader

  #11430 merged Nov 26, 2024

• Build: Delete branch automatically on PR merge

  #11635 merged Nov 26, 2024

• Build: Bump nessie from 0.100.0 to 0.100.2

  #11637 merged Nov 25, 2024

• Docs: add WHEN NOT MATCHED BY SOURCE to Spark MERGE INTO doc

  #11636 merged Nov 25, 2024

• Docs: Use DataFrameWriterV2 in Spark's example

  #11647 merged Nov 25, 2024

• Flink: Add table.exec.iceberg.use-v2-sink option

  #11244 merged Nov 25, 2024

• Build: Bump com.google.errorprone:error_prone_annotations from 2.35.1 to 2.36.0

  #11638 merged Nov 25, 2024

• Core,Open-API: Don't expose the last-column-id

  #11514 merged Nov 25, 2024

• Build: Bump software.amazon.awssdk:bom from 2.29.15 to 2.29.20

  #11639 merged Nov 25, 2024

• Spark 3.3: Correct the two-stage parsing strategy of antlr parser

  #11630 merged Nov 25, 2024

• Build: Bump mkdocs-material from 9.5.44 to 9.5.45

  #11641 merged Nov 25, 2024

• Build: Bump testcontainers from 1.20.3 to 1.20.4

  #11640 merged Nov 25, 2024

• Spark 3.3: IcebergSource extends SessionConfigSupport

  #11625 merged Nov 23, 2024

• Spark 3.5: IcebergSource extends SessionConfigSupport

  #11624 merged Nov 23, 2024

• Spark 3.4: IcebergSource extends SessionConfigSupport

  #7732 merged Nov 23, 2024

• Docs: Mention look-free requires HIVE-28121 for MySQL/MariaDB-based HMS

  #11631 merged Nov 23, 2024

• 1.7.x cherry pick #11526

  #11629 merged Nov 22, 2024

• Docs: Add new blog post to Iceberg Blogs

  #11627 merged Nov 22, 2024

• Spark 3.4: Correct the two-stage parsing strategy of antlr parser

  #7734 merged Nov 22, 2024

• Spark 3.5: Correct the two-stage parsing strategy of antlr parser

  #11628 merged Nov 22, 2024

• Add REST Catalog tests to Spark 3.5 integration test

  #11093 merged Nov 21, 2024

• 1.7.x apply PR #11220

  #11622 merged Nov 21, 2024

• Revert "Core: Update TableMetadataParser to ensure all streams closed (#11220)"

  #11621 merged Nov 21, 2024

• 1.7.1rc0 cherry pick PR #11564

  #11613 merged Nov 21, 2024

• Core: Optimize MergingSnapshotProducer to use referenced manifests to determine if manifest needs to be rewritten

  #11131 merged Nov 21, 2024

• Upgrade to Gradle 8.11.1

  #11619 merged Nov 21, 2024

• Spark 3.5: Fix flaky TestRemoveOrphanFilesAction3

  #11616 merged Nov 21, 2024

• Spark: Fix changelog table bug for start time older than current snapshot

  #11564 merged Nov 21, 2024

• Parquet: Use native getRowIndexOffset support instead of calculating it

  #11520 merged Nov 21, 2024

• Procedure to compute table stats

  #10986 merged Nov 20, 2024

• Docs: add iceberg-go to doc site

  #11607 merged Nov 20, 2024

• Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError

  #11576 merged Nov 20, 2024

• Spark 3.3: Deprecate support

  #11596 merged Nov 20, 2024

• 1.7.1rc0 cherry pick PR #11157

  #11605 merged Nov 20, 2024

• 1.7.0rc0 cherry picks #2

  #11603 merged Nov 20, 2024

• Spark 3.5: Fix NotSerializableException when migrating Spark tables

  #11157 merged Nov 20, 2024

• API, Core: Remove unnecessary casts to Iterable<T>

  #11601 merged Nov 20, 2024

• Core: Fix CCE when retrieving TableOps

  #11585 merged Nov 20, 2024

• core: Filter on live entries when reading the manifest

  #9996 merged Nov 20, 2024

• Build: Bump Apache Parquet 1.14.4

  #11502 merged Nov 20, 2024

• Core: delete temp metadata file when version already exists

  #11350 merged Nov 20, 2024

23 Pull requests opened by 18 people

• Kafka Connect: Add config to prefix the control consumer group

  #11599 opened Nov 20, 2024

• Ignore partition fields that are dropped from the current-schema

  #11604 opened Nov 20, 2024

• Document procedure for stats collection

  #11606 opened Nov 20, 2024

• Core: Fix a bug in streams closing while read or write metadata files

  #11609 opened Nov 21, 2024

• Spark: remove ROW_POSITION from project schema

  #11610 opened Nov 21, 2024

• Spark 3.5: Refactor scanning changelog table with timestamps

  #11612 opened Nov 21, 2024

• Spark : Derive Stats From Manifest on the Fly

  #11615 opened Nov 21, 2024

• Core: Add TableUtil to provide access to a table's format version

  #11620 opened Nov 21, 2024

• Kafka Connect: Add mechanisms for routing records by topic name

  #11623 opened Nov 22, 2024

• Core,API: Set `503: added_snapshot_id` as required

  #11626 opened Nov 22, 2024

• Create publish-docker.yml

  #11632 opened Nov 22, 2024

• Docs: Add RisingWave

  #11642 opened Nov 25, 2024

• Hadoop: Log where the missing metadata file is located

  #11643 opened Nov 25, 2024

• Core: Set missing table-default property in RESTSessionCatalog

  #11646 opened Nov 25, 2024

• Adding ComputeTableStats Procedure to Spark 3.4

  #11652 opened Nov 26, 2024

• Parquet: add variant type support

  #11653 opened Nov 26, 2024

• Core, Spark3.5: Fix tests failure due to timeout

  #11654 opened Nov 26, 2024

• Parquet: Bump to Apache Parquet 1.15.0

  #11656 opened Nov 26, 2024

• Spark: Read DVs when reading from .position_deletes table

  #11657 opened Nov 26, 2024

• Replace use of deprecated methods

  #11658 opened Nov 26, 2024

• Spec: Document Snapshot Summary Optional Fields for Standardization

  #11660 opened Nov 26, 2024

• Reduce code duplication in VectorizedParquetDefinitionLevelReader

  #11661 opened Nov 27, 2024

• Flink: Fix range distribution npe when value is null

  #11662 opened Nov 27, 2024

13 Issues closed by 6 people

• Renamed column returns null values from 'appended' Parquet file not originally created by Iceberg

  #11650 closed Nov 26, 2024

• How an application can communicate with the Iceberg REST catalog like Dremio Arctic (Nessie), Snowflake’s Polaris , Gravitino

  #11649 closed Nov 26, 2024

• Interaction with the Iceberg REST catalog like Dremio Arctic (Nessie), Snowflake’s Polaris , Gravitino

  #11655 closed Nov 26, 2024

• Allow to configure thread-pool while using Iceberg to read the data (plan files/tasks)

  #10335 closed Nov 26, 2024

• REST Catalog to support custom-catalog name like HMS/Glue

  #10205 closed Nov 26, 2024

• Using subdirectory to dave data in ICEBERG.

  #10327 closed Nov 24, 2024

• catalog issue

  #10324 closed Nov 24, 2024

• Improvements to Iceberg Catalog Descriptions

  #10316 closed Nov 24, 2024

• User ID information in Iceberg Table's snapshot

  #11474 closed Nov 22, 2024

• Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError

  #11575 closed Nov 21, 2024

• how do you guys back up your iceberg table?

  #10299 closed Nov 21, 2024

• procedure add_files parallelism > 1 -> NotSerializableException

  #11147 closed Nov 20, 2024

• Cannot resolve method 'predicate' with Expression.transform

  #11600 closed Nov 20, 2024

12 Issues opened by 11 people

• Document Snapshot Summary Optional Fields for Standardization

  #11659 opened Nov 26, 2024

• Flaky test `TestCopyOnWriteDelete > testDeleteWithSnapshotIsolation()`

  #11651 opened Nov 26, 2024

• SparkExecutorCache causes slowness of RewriteDataFilesSparkAction

  #11648 opened Nov 25, 2024

• How to move Iceberg table from one location to another

  #11645 opened Nov 25, 2024

• Flink Use distribution-mode: RANGE , null partition bucket will case error

  #11644 opened Nov 25, 2024

• Efficient Addition of New Columns in Large-Scale Feature Datasets

  #11634 opened Nov 23, 2024

• java.lang.IllegalStateException: Connection pool shut down in Spark

  #11633 opened Nov 22, 2024

• Best Practices for Storing and Querying Full History and Latest Versions

  #11618 opened Nov 21, 2024

• Is iceberg support "Predicate Pushdown" when spark read data from it?

  #11617 opened Nov 21, 2024

• java.io.IOException: can not read class org.apache.iceberg.shaded.org.apache.parquet.format.PageHeader: Required field 'num_values' was not found in serialized data

  #11614 opened Nov 21, 2024

• Using the Struct type as the primary key in equalDelete operation will cause data reading errors.

  #11611 opened Nov 21, 2024

• REST Catalog S3 Signer Endpoint should be Catalog specific

  #11608 opened Nov 20, 2024

71 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Hive: Optimize tableExists API in hive catalog

  #11597 commented on Nov 26, 2024 • 20 new comments

• API: Support removeUnusedSpecs in ExpireSnapshots

  #10755 commented on Nov 25, 2024 • 18 new comments

• Spark: Write DVs for V3 MoR tables

  #11561 commented on Nov 26, 2024 • 17 new comments

• Spec: add variant type

  #10831 commented on Nov 26, 2024 • 15 new comments

• Materialized View Spec

  #11041 commented on Nov 25, 2024 • 11 new comments

• API, Core: Add scan planning apis to REST Catalog

  #11180 commented on Nov 25, 2024 • 9 new comments

• REST: Docker file for Rest catalog adapter image

  #11283 commented on Nov 26, 2024 • 8 new comments

• Data: Add partition stats writer and reader

  #11216 commented on Nov 26, 2024 • 8 new comments

• Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required

  #10962 commented on Nov 26, 2024 • 8 new comments

• Azure: Support vended credentials refresh in ADLSFileIO.

  #11577 commented on Nov 22, 2024 • 6 new comments

• GCP: Implement SupportsRecoveryOperations for GCSFileIO

  #11565 commented on Nov 25, 2024 • 6 new comments

• Core: Fix caching table with metadata table names

  #11123 commented on Nov 26, 2024 • 5 new comments

• Add scan planning api request and response models, parsers

  #11369 commented on Nov 26, 2024 • 4 new comments

• Spark: add property to disable client-side purging in spark

  #11317 commented on Nov 25, 2024 • 3 new comments

• Core: Add support for `view-default` property in catalog

  #11064 commented on Nov 25, 2024 • 3 new comments

• Core,Format: Deprecate embedded manifests

  #11586 commented on Nov 22, 2024 • 2 new comments

• Flink: Add RowConverter for Iceberg Source

  #11301 commented on Nov 26, 2024 • 2 new comments

• Docs: Use the correct YAML text block indicator to prevent formatting issues

  #11552 commented on Nov 20, 2024 • 2 new comments

• REST: AuthManager API

  #10753 commented on Nov 21, 2024 • 2 new comments

• Core, Spark: Refactor RewriteFileGroup planner to core

  #11513 commented on Nov 21, 2024 • 2 new comments

• API, Core: Add formatVersion() to Table

  #11587 commented on Nov 26, 2024 • 1 new comment

• [Views] Update view spec with table identifier requirements

  #11365 commented on Nov 21, 2024 • 1 new comment

• Spec: Support geo type

  #10981 commented on Nov 24, 2024 • 1 new comment

• Spark 3.5: Implement RewriteTablePath

  #11555 commented on Nov 20, 2024 • 1 new comment

• [WIP] Spark: DVs + Positional Deletes

  #11545 commented on Nov 25, 2024 • 1 new comment

• Spark: Add view support to SparkSessionCatalog

  #11388 commented on Nov 25, 2024 • 1 new comment

• Build: Bump calcite from 1.10.0 to 1.38.0

  #11361 commented on Nov 22, 2024 • 0 new comments

• Flink Support for TIMESTAMP_NANOS

  #11348 commented on Nov 24, 2024 • 0 new comments

• Add optional Glue Schema configuration to exclude Non-Current Fields

  #11334 commented on Nov 24, 2024 • 0 new comments

• API: Align CharSequenceSet impl with Data/DeleteFileSet

  #11322 commented on Nov 26, 2024 • 0 new comments

• (AWS) Docs: List all AWS S3 properties from all language impl.

  #11321 commented on Nov 22, 2024 • 0 new comments

• add_files with RestCatalog, S3FileIO

  #11558 commented on Nov 21, 2024 • 0 new comments

• Kafka Connect: Add config to route to tables using topic name

  #11313 commented on Nov 22, 2024 • 0 new comments

• Handling NO Coordinator Scenario and Data Loss in the current Design

  #11298 commented on Nov 22, 2024 • 0 new comments

• OpenAPI: Define REST Catalog models for Snapshot Production

  #11287 commented on Nov 23, 2024 • 0 new comments

• When write.object-storage.enabled=true, it is difficult to gather information for individual partition of partitioned tables

  #11488 commented on Nov 21, 2024 • 0 new comments

• Fix when reading struct-type data without an id in iceberg-parquet

  #11378 commented on Nov 26, 2024 • 0 new comments

• Core: Fix drop partition field and schema field error

  #11387 commented on Nov 24, 2024 • 0 new comments

• Core, Rest: Read the max connection for rest client from properties

  #11522 commented on Nov 25, 2024 • 0 new comments

• Core, Rest: Enable useSystemProperties on RESTClient

  #11548 commented on Nov 25, 2024 • 0 new comments

• API: Follow up on adding Variant data type to implement sanitizing for Variant

  #11479 commented on Nov 21, 2024 • 0 new comments

• Iceberg 1.7.0 java.lang.IllegalStateException: Connection pool shut down

  #11582 commented on Nov 21, 2024 • 0 new comments

• remove orphan file question

  #10363 commented on Nov 21, 2024 • 0 new comments

• NullPointerException when using VectorizedArrowReader to read a null column

  #10275 commented on Nov 20, 2024 • 0 new comments

• Add SparkSessionCatalog support for views

  #9845 commented on Nov 25, 2024 • 0 new comments

• Review new DangerousJavaDeserialization error-prone check

  #10853 commented on Nov 25, 2024 • 0 new comments

• Iceberg TTL setting

  #10372 commented on Nov 26, 2024 • 0 new comments

• Handling Updates on Partition Columns in Iceberg with Flink CDC

  #11573 commented on Nov 26, 2024 • 0 new comments

• SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog

  #10003 commented on Nov 27, 2024 • 0 new comments

• Support for loading different hive-metastore versions at Runtime

  #10401 commented on Nov 27, 2024 • 0 new comments

• Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action

  #9731 commented on Nov 22, 2024 • 0 new comments

• Core: add support to add custom schemes via properties in ResolvingFileIO

  #9884 commented on Nov 24, 2024 • 0 new comments

• Support convert orc timestamptz

  #9905 commented on Nov 22, 2024 • 0 new comments

• [draft] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC

  #10233 commented on Nov 22, 2024 • 0 new comments

• Kafka Connect: Add table to topics mapping property

  #10422 commented on Nov 27, 2024 • 0 new comments

• ERROR when executing UPDATE/DELETE queries in Iceberg 1.6.0: "Cannot add fieldId 1 as an identifier field"

  #11341 commented on Nov 25, 2024 • 0 new comments

• Quick notes how to update docs and javadoc at release publication time.

  #10810 commented on Nov 24, 2024 • 0 new comments

• Iceberg Roadmap is 404

  #10390 commented on Nov 25, 2024 • 0 new comments

• Support changelog scan for table with delete files

  #10935 commented on Nov 25, 2024 • 0 new comments

• Copy iceberg table from hdfs to GCS and register table to BLMS

  #10389 commented on Nov 25, 2024 • 0 new comments

• Remove Hive 2

  #10996 commented on Nov 20, 2024 • 0 new comments

• Add view support for Hadoop catalog

  #10387 commented on Nov 25, 2024 • 0 new comments

• Introduce a parameter to control whether the flink writer is linked with the previous operator

  #10371 commented on Nov 24, 2024 • 0 new comments

• Using the Iceberg catalog in your file system

  #10326 commented on Nov 24, 2024 • 0 new comments

• Config for deciding whether to use Iceberg Time type

  #11174 commented on Nov 22, 2024 • 0 new comments

• Support different JDBC backend in the `JdbcCatalog`

  #9733 commented on Nov 24, 2024 • 0 new comments

• [Bug] Iceberg tables break when they're named any of the metadata table names (e.g. `files`, `history`, `manifests`)

  #10550 commented on Nov 22, 2024 • 0 new comments

• More accurate estimate on parquet row groups size

  #11258 commented on Nov 25, 2024 • 0 new comments

• Spec: Add cross-region bucket access property to config

  #11260 commented on Nov 25, 2024 • 0 new comments

• Added support for evolving the partition of the table

  #11275 commented on Nov 22, 2024 • 0 new comments

• Variant Data Type Support

  #10392 commented on Nov 22, 2024 • 0 new comments