fix(airflow): set minimum supported version to 2.7.0 (#13357) #13619

harishkesavarao · 2025-05-25T07:13:45Z

Part of a 2-step fix for #13357

hsheth2

The airflow.md docs file needs to be updated

hsheth2 · 2025-05-27T16:24:10Z

metadata-ingestion-modules/airflow-plugin/setup.py

 base_requirements = {
    f"acryl-datahub[datahub-rest]{_self_pin}",
-    # We require Airflow 2.3.x at minimum, since we need the new DAG listener API.
+    # We require Airflow 2.7.x at minimum, since we need the new DAG listener API.


this comment is a bit inaccurate now - the DAG listener API was added in 2.3. We required 2.7+ for a different reason, which should be listed here

@hsheth2 Agreed, will change. Can you please point me to the unit and integration tests in this repo? I am new to this setup, and tried digging my way around the tests which need to be updated to the new Airflow version, but just wanted to confirm.

it's in metadata-ingestion-modules/airflow-plugin/tests

Changed the comment.

harishkesavarao · 2025-06-03T12:13:45Z

The airflow.md docs file needs to be updated

Will update it. To reflect the fact that the V2 plugin will support only Airflow versions >= 2.7.

…#13357)

hsheth2 · 2025-06-09T23:55:00Z

docs/lineage/airflow.md


-We no longer officially support Airflow <2.5. However, you can use older versions of `acryl-datahub-airflow-plugin` with older versions of Airflow.
+We no longer officially support Airflow <2.7. However, you can use older versions of `acryl-datahub-airflow-plugin` with older versions of Airflow.
 The first two options support Python 3.7+, and the others require Python 3.8+.


the list below should also be updated to explain the older versions that have support

@hsheth2 Added more details related to older versions. Let me know if this looks good or if more info needs to be added.

hsheth2 · 2025-06-09T23:55:25Z

.github/workflows/airflow-plugin.yml

-            extra_pip_constraints: "-c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt"
+          #          - python-version: "3.10"
+          #            extra_pip_requirements: "apache-airflow~=2.6.3"
+          #            extra_pip_constraints: "-c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt"


let's remove these lines entirely

.github/workflows/airflow-plugin.yml

metadata-ingestion-modules/airflow-plugin/setup.py

harishkesavarao · 2025-06-28T18:14:17Z

@harishkesavarao looks like CI is complaining about some lint errors

Thanks for letting me know. Mostly due to formatting imports. Got past that now. Can you please take another look?

docs/lineage/airflow.md

hsheth2 · 2025-06-30T19:06:20Z

metadata-ingestion-modules/airflow-plugin/tox.ini

    # For older versions of Airflow (where we can't use constraints),
    # we need to add a couple extra constraints.
-    airflow25: test-airflow25
+    airflow27: test-airflow27


this does not look right

probably can just be removed

Removed the constraint.

hsheth2 · 2025-06-30T19:06:43Z

metadata-ingestion-modules/airflow-plugin/setup.py

can remove the test-airflow25 extra as well

hsheth2 · 2025-06-30T19:07:43Z

metadata-ingestion-modules/airflow-plugin/tox.ini


 [tox]
-envlist = py38-airflow25, py310-airflow26, py310-airflow27, py310-airflow28, py311-airflow29, py311-airflow210
+envlist = py38-airflow27,py310-airflow27, py310-airflow28, py311-airflow29, py311-airflow210


tox.ini should be in sync with the airflow-plugin.yml

this say it's testing both py38 and py310 with airflow 2.7, but the other file has something different

Agree, changed it.

metadata-ingestion-modules/airflow-plugin/setup.py

Co-authored-by: Harshal Sheth <[email protected]>

…#13619) Co-authored-by: Harshal Sheth <[email protected]>

…13619) (datahub-project#13946)

github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels May 25, 2025

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label May 25, 2025

vercel bot deployed to Preview May 25, 2025 07:32 View deployment

vercel bot deployed to Preview May 25, 2025 14:17 View deployment

hsheth2 suggested changes May 27, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels May 27, 2025

vercel bot had a problem deploying to Preview May 31, 2025 16:04 Failure

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jun 3, 2025

harishkesavarao added 2 commits June 9, 2025 23:08

fix(airflow): set minimum supported version to 2.7.0 (datahub-project…

efef193

…#13357)

fix(airflow): remove tests for airflow < 2.7

7da9489

harishkesavarao force-pushed the fix/airflow-versions branch from 963d1a5 to 7da9489 Compare June 9, 2025 17:38

vercel bot deployed to Preview June 9, 2025 17:57 View deployment

harishkesavarao added 3 commits June 9, 2025 23:28

fix(airflow): Comment change to reflect requirement of Airflow 2.7.x

2ef5843

fix(airflow): Removed tests for older Airflow versions

1212138

fix(airflow): changes to airflow.md file

0abe29e

harishkesavarao requested a review from hsheth2 June 9, 2025 18:06

vercel bot deployed to Preview June 9, 2025 18:38 View deployment

hsheth2 reviewed Jun 9, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jun 10, 2025

harishkesavarao added 3 commits June 22, 2025 23:46

Update setup.py to reflect the Airflow version

5d82a9f

Update airflow-plugin.yml to remove old Airflow versions

b61cd7e

Compatibility changes

e95897e

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jun 22, 2025

Update test_airflow.py

78e8483

vercel bot deployed to Preview June 28, 2025 14:53 View deployment

harishkesavarao added 5 commits June 28, 2025 22:37

Removed an extra comma

a40009b

Re-format imports

7b7c224

Update test_airflow.py

e8e2780

Add Python 3.8 env to Airflow 2.7

9848f87

Remove Python 3.8 for Airflow 2.7

64fa0f2

vercel bot deployed to Preview June 28, 2025 18:22 View deployment

hsheth2 suggested changes Jun 30, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jun 30, 2025

harishkesavarao and others added 2 commits July 1, 2025 20:25

Update metadata-ingestion-modules/airflow-plugin/setup.py

03e527b

Co-authored-by: Harshal Sheth <[email protected]>

Update docs/lineage/airflow.md

7319ee8

Co-authored-by: Harshal Sheth <[email protected]>

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jul 1, 2025

vercel bot deployed to Preview July 1, 2025 15:34 View deployment

harishkesavarao added 4 commits July 1, 2025 22:36

Added Python 3.8 to sync with tox.ini

8e13da5

Added py38-airflow27 constraint file

8b31205

Remove extra deps

f6749e2

Merge branch 'master' into fix/airflow-versions

809bb20

harishkesavarao requested a review from hsheth2 July 1, 2025 17:28

vercel bot deployed to Preview July 1, 2025 18:06 View deployment

hsheth2 approved these changes Jul 2, 2025

View reviewed changes

hsheth2 merged commit 77a5671 into datahub-project:master Jul 2, 2025
52 checks passed

hsheth2 added a commit that referenced this pull request Jul 2, 2025

docs(airflow): follow-ups for dropping airflow <2.7 (#13619)

af708ae

hsheth2 mentioned this pull request Jul 2, 2025

docs(airflow): follow-ups for dropping airflow <2.7 (#13619) #13946

Merged

treff7es pushed a commit that referenced this pull request Jul 4, 2025

docs(airflow): follow-ups for dropping airflow <2.7 (#13619) (#13946)

308c6bd

Linux-oiD mentioned this pull request Jul 7, 2025

Make Airflow plugin fully compatible with Airflow >=2.7 #13357

Open

kartikey-visa pushed a commit to kartikey-visa/datahub that referenced this pull request Jul 23, 2025

fix(airflow): set minimum supported version to 2.7.0 (datahub-project…

214b05c

…#13619) Co-authored-by: Harshal Sheth <[email protected]>

kartikey-visa pushed a commit to kartikey-visa/datahub that referenced this pull request Jul 23, 2025

docs(airflow): follow-ups for dropping airflow <2.7 (datahub-project#…

9e61c6e

…13619) (datahub-project#13946)

fix(airflow): set minimum supported version to 2.7.0 (#13357) #13619

fix(airflow): set minimum supported version to 2.7.0 (#13357) #13619

Uh oh!

Conversation

harishkesavarao commented May 25, 2025

Uh oh!

hsheth2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harishkesavarao commented Jun 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

harishkesavarao commented Jun 28, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants