Skip to content

GH-2926: Fix issue reading buggy parquet files with incorrect compressed size #2927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pyckle wants to merge 1 commit into
base: master
Choose a base branch
from
Open

GH-2926: Fix issue reading buggy parquet files with incorrect compressed size #2927

pyckle wants to merge 1 commit into from

Conversation

@pyckle
Copy link
Contributor

@pyckle pyckle commented Jun 23, 2024

Make sure you have checked all steps below.

Jira

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    Unit test with malformed file is added

Commits

  • My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines
    from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Style

  • My contribution adheres to the code style guidelines and Spotless passes.
    • To apply the necessary changes, run mvn spotless:apply -Pvector-plugins

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

@wgtmac wgtmac changed the title #2926 Fix issue reading buggy parquet files with incorrect compressed size GH-2926: Fix issue reading buggy parquet files with incorrect compressed size Jun 24, 2024
emkornfield
emkornfield reviewed Jun 26, 2024
View reviewed changes
parquet-avro/src/test/resources/test-bad-compressed-size.parquet
Copy link
Contributor

@emkornfield emkornfield Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typically parquet files are checked into the parquet-testing repo

Copy link
Contributor Author

@pyckle pyckle Aug 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks for taking a look. I encountered this issue while working on an early version of a new parquet driver built on top of parquet-java (https://github.com/Earnix/parquetforge)
I'm happy to put this generated corrupted file into the parquet-testing repo. But that leaves a few questions:

  • Should this go in the bad data folder? It is indeed invalid parquet, but parquet-java attempts to read it.
  • This flow is very difficult to unit test without the corrupted file. Can (or should) this be merged without an automated test?

As such, I think the sensible options are either to merge the fix without a test, or remove this code entirely. Removing the code entirely seems to make more sense as it's only purpose is to allow reading these corrupted files and it's been broken for a while so nobody seems to depend upon it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emkornfield emkornfield emkornfield left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pyckle @emkornfield