Skip to content

fix(openapi_parser): add ability to parse openapi 3.0+ schemas #13624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sgomezvillamor merged 9 commits into from
Jun 1, 2025
Merged

fix(openapi_parser): add ability to parse openapi 3.0+ schemas #13624

sgomezvillamor merged 9 commits into from
Jun 1, 2025

Conversation

@chatterjee-atanu
Copy link
Contributor

@chatterjee-atanu chatterjee-atanu commented May 26, 2025 •
edited
Loading

fixes #10197

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels May 26, 2025
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label May 26, 2025
@chatterjee-atanu chatterjee-atanu mentioned this pull request May 26, 2025
Closed
@sgomezvillamor
Copy link
Contributor

sgomezvillamor commented May 30, 2025

We have an integration test for openapi source

datahub/metadata-ingestion/tests/integration/openapi/openapi_to_file.yml

Lines 1 to 12 in 075c57b

source:
type: openapi
config:
name: test_openapi
url: https://raw.githubusercontent.com/OAI/learn.openapis.org/refs/heads/main/examples/
swagger_file: v3.0/api-with-examples.yaml
sink:
type: file
config:
filename: "/tmp/openapi_mces.json"

That is, it uses https://github.com/OAI/learn.openapis.org/blob/main/examples/v3.0/api-with-examples.yaml for the testing. However, I see no parameters or servers here so your new feature is not being validated.

I do see parameters and servers in https://github.com/OAI/learn.openapis.org/blob/main/examples/v3.0/petstore-expanded.yaml Could you add a new test in https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/tests/integration/openapi/test_openapi.py with that one?

@chatterjee-atanu
Copy link
Contributor Author

chatterjee-atanu commented May 30, 2025

@sgomezvillamor sure let me add that

@chatterjee-atanu

This comment was marked as outdated.

Sign in to view
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels May 30, 2025
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels May 30, 2025
@sgomezvillamor
Copy link
Contributor

sgomezvillamor commented May 30, 2025

Almost there @chatterjee-atanu ! 💪

Would reformat: src/datahub/ingestion/source/openapi_parser.py
1 file would be reformatted, 1214 files already formatted

This should be automatically fixed with ./gradlew :metadata-ingestion:lintFix

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels May 30, 2025
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels May 30, 2025
@chatterjee-atanu
Copy link
Contributor Author

chatterjee-atanu commented May 30, 2025 •
edited
Loading

Almost there @chatterjee-atanu ! 💪

Would reformat: src/datahub/ingestion/source/openapi_parser.py
1 file would be reformatted, 1214 files already formatted

This should be automatically fixed with ./gradlew :metadata-ingestion:lintFix

done 💯

@sgomezvillamor sgomezvillamor merged commit 487d52a into datahub-project:master Jun 1, 2025
57 of 59 checks passed
@chatterjee-atanu chatterjee-atanu deleted the fix/10197_openapi_ingestion_fix branch June 1, 2025 18:10
kartikey-visa pushed a commit to kartikey-visa/datahub that referenced this pull request Jul 23, 2025
@chatterjee-atanu @kartikey-visa
fix(openapi_parser): add ability to parse openapi 3.0+ schemas (datah…
642f529 
…ub-project#13624)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgomezvillamor sgomezvillamor sgomezvillamor approved these changes

Assignees

No one assigned

Labels

community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

OpenAPI ingest breaks on valid definition version 3.0.1

2 participants

@chatterjee-atanu @sgomezvillamor