Skip to content

feat(ingestion/hive): Add lineage functionality for hive tables from/to file storage #11841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
anshbansal merged 39 commits into from
Dec 21, 2024
Merged

feat(ingestion/hive): Add lineage functionality for hive tables from/to file storage #11841

anshbansal merged 39 commits into from
Dec 21, 2024

Conversation

@acrylJonny
Copy link
Collaborator

@acrylJonny acrylJonny commented Nov 12, 2024

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Nov 12, 2024
@acrylJonny acrylJonny marked this pull request as draft November 12, 2024 21:39
@acrylJonny
Copy link
Collaborator Author

acrylJonny commented Dec 6, 2024

Reopening as this PR as it was closed in error

@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 6, 2024
treff7es
treff7es approved these changes Dec 17, 2024
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/sql/hive.py
default=False,
description="Whether to emit storage-to-Hive lineage",
)
hive_storage_lineage_direction: str = Field(
Copy link
Contributor

@treff7es treff7es Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't storage always upstream of the hive dataset?

Copy link
Contributor

@treff7es treff7es Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or a sibling?

Copy link
Collaborator Author

@acrylJonny acrylJonny Dec 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be to have parity with the glue_s3_lineage_direction parameter in the Glue config, so this was added to ensure that there is consistency.
If it should be removed, we might want to look at the Glue connector also.

@treff7es treff7es self-assigned this Dec 17, 2024
@datahub-cyborg datahub-cyborg bot added merge-pending-ci A PR that has passed review and should be merged once CI is green. and removed needs-review Label for PRs that need review from a maintainer. labels Dec 17, 2024
treff7es
treff7es reviewed Dec 17, 2024
View reviewed changes
@anshbansal anshbansal merged commit 8350a4e into datahub-project:master Dec 21, 2024
74 checks passed
brock-acryl pushed a commit to brock-acryl/datahub that referenced this pull request Jan 30, 2025
@acrylJonny @anshbansal @chakru-r
feat(ingest/hive): lineage from/to file storage (datahub-project#11841)
d6749ad 
Co-authored-by: Aseem Bansal <[email protected]>
@anilreddygollapalli anilreddygollapalli mentioned this pull request Apr 16, 2025
Open
kartikey-visa pushed a commit to kartikey-visa/datahub that referenced this pull request Jul 23, 2025
@acrylJonny @anshbansal @kartikey-visa
feat(ingest/hive): lineage from/to file storage (datahub-project#11841)
06d4884 
Co-authored-by: Aseem Bansal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@treff7es treff7es treff7es approved these changes

Assignees

@treff7es treff7es

Labels

ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@acrylJonny @deepgarg760 @treff7es @anshbansal