Support Task execution interface (AIP-72) in Airflow 3 in EdgeExecutor #44982
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jscheffl
requested review from
ashb,
XD-DENG,
o-nikolas,
pierrejeambrun and
hussein-awala
as code owners
boring-cyborg
bot
added
area:Executors-core
jscheffl
requested review from
Copilot,
kaxil and
ashb
and removed request for
ashb,
XD-DENG,
pierrejeambrun,
hussein-awala,
o-nikolas and
kaxil
jscheffl
changed the title
There was a problem hiding this comment.
Copilot reviewed 5 out of 16 changed files in this pull request and generated no comments.
Files not reviewed (11)
- docs/apache-airflow-providers-edge/edge_executor.rst: Language not supported
- providers/src/airflow/providers/edge/CHANGELOG.rst: Language not supported
- airflow/jobs/scheduler_job_runner.py: Evaluated as low risk
- providers/src/airflow/providers/edge/worker_api/routes/jobs.py: Evaluated as low risk
- airflow/executors/base_executor.py: Evaluated as low risk
- airflow/executors/local_executor.py: Evaluated as low risk
- providers/tests/edge/plugins/test_edge_executor_plugin.py: Evaluated as low risk
- providers/src/airflow/providers/edge/worker_api/datamodels.py: Evaluated as low risk
- providers/src/airflow/providers/edge/init.py: Evaluated as low risk
- providers/src/airflow/providers/edge/provider.yaml: Evaluated as low risk
- providers/src/airflow/providers/edge/worker_api/routes/_v2_compat.py: Evaluated as low risk
Comments suppressed due to low confidence (2)
providers/src/airflow/providers/edge/executors/edge_executor.py:111
- [nitpick] The comment should be updated to reflect that this method is used for executing tasks asynchronously in both Airflow 2.10 and Airflow 3.0.
"""Execute asynchronously. Airflow 2.10 entry point to execute a task."""
providers/src/airflow/providers/edge/executors/edge_executor.py:133
- Ensure that the new queue_workload method is covered by tests to validate its behavior with the new workload type.
def queue_workload(
This was referenced Dec 17, 2024
jscheffl
added
the
area:task-execution-interface-aip72
jscheffl
force-pushed
the
feature/add-airflow-2-aip-72-support-to-edge
branch
2 times, most recently
from
5b6d05a to
2e5e993
Compare
jscheffl
force-pushed
the
feature/add-airflow-2-aip-72-support-to-edge
branch
from
851b897 to
6316d14
Compare
jscheffl
force-pushed
the
feature/add-airflow-2-aip-72-support-to-edge
branch
2 times, most recently
from
2e771b9 to
0156a2f
Compare
jscheffl
force-pushed
the
feature/add-airflow-2-aip-72-support-to-edge
branch
from
0156a2f to
3910484
Compare
ashb
approved these changes
Jan 7, 2025
jscheffl
force-pushed
the
feature/add-airflow-2-aip-72-support-to-edge
branch
from
3910484 to
f11e1b8
Compare
agupta01
pushed a commit
to agupta01/airflow
that referenced
this pull request
Jan 13, 2025
apache#44982) * Extend the queue_workload() call with an ORM session * Support Task execution interface (AIP-72) in Airflow 3 * Fix CI errors, typos and static checks * Fix pytest backcompat * Apply Copilot feedback from other PR * Review Feedback * Add missing session to CLI call * Review Feedback
HariGS-DB
pushed a commit
to HariGS-DB/airflow
that referenced
this pull request
Jan 16, 2025
apache#44982) * Extend the queue_workload() call with an ORM session * Support Task execution interface (AIP-72) in Airflow 3 * Fix CI errors, typos and static checks * Fix pytest backcompat * Apply Copilot feedback from other PR * Review Feedback * Add missing session to CLI call * Review Feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The introduction of AIP-72 and removal of AIP-44 in main broke the EdgeExecutor/EdgeWorker on main.
This PR makes it working again as the second executor supporting AIP-72 for the first "edge cases" and check.
Note: Two commits, first one is an extension in the Executor API which is needed to ship a session context to EdgeExecutor not to break DB lock during scheduling as EdgeExecutor need to write to DB. Let me know if I should separate into two PRs... (as of dev rules core and provider changes bundled here)
@ashb Would very much favor a review from you here if the design anticipated for AIP-72 is correctly understood how integrated.
If you want to test with Airflow 3, use:
breeze start-airflow --python 3.12 --load-example-dags --backend postgres --executor EdgeExecutor --answer yIn Airflow 3 the "example_bash_operator" is working, not all other DAGs.
If somebody wants to test Airflow 2.10 as regression, this can be made via:
breeze down && rm dist/* && breeze release-management prepare-provider-packages --include-not-ready-providers edge && breeze start-airflow --python 3.12 --load-example-dags --backend postgres --executor EdgeExecutor --answer y --use-airflow-version 2.10.4 --use-packages-from-distFYI @AutomationDev85