Enforce Best Practices for all your Airflow DAGs. ⭐
- Use function-level imports instead of top-level imports12 (see Top level Python Code)
- Use jinja template syntax instead of
Variable.get
(see Airflow Variables)
based on official Best Practices
airflint is tested with:
Main version (dev) | Released version (0.3.2-alpha) | |
---|---|---|
Python | 3.9, 3.10, 3.11.0-alpha - 3.11.0 | 3.9, 3.10 |
Apache Airflow | >= 2.0.0 | >= 2.0.0 |
To install it from PyPI run:
pip install airflint
NOTE: It is recommended to install airflint into your existing airflow environment with all your providers included. This way
UseJinjaVariableGet
rule can detect alltemplate_fields
and airflint works as expected.
Then just call it like this:
Alternatively you can add the following repo to your pre-commit-config.yaml
:
- repo: https://github.com/feluelle/airflint
rev: v0.3.2-alpha
hooks:
- id: airflint
args: ["-a"] # Use -a to apply the suggestions
additional_dependencies: # Add all package dependencies you have in your dags, preferable with version spec
- apache-airflow
- apache-airflow-providers-cncf-kubernetes
To complete the UseFunctionlevelImports
rule, please add the autoflake
hook after the airflint
hook, as below:
- repo: https://github.com/pycqa/autoflake
rev: v1.4
hooks:
- id: autoflake
args: ["--remove-all-unused-imports", "--in-place"]
This will remove unused imports.
I am looking for contributors who are interested in..
- testing airflint with real world Airflow DAGs and reporting issues as soon as they face them
- optimizing the ast traversing for existing rules
- adding new rules based on best practices or bottlenecks you have experienced during Airflow DAGs authoring
- documenting about what is being supported in particular by each rule
- defining supported airflow versions i.e. some rules are bound to specific Airflow features and version
For questions, please don't hesitate to open a GitHub issue.
Footnotes
-
There is a PEP for Lazy Imports targeted to arrive in Python 3.12 which would supersede this rule. ↩
-
To remove top-level imports after running
UseFunctionLevelImports
rule, use a tool such as autoflake. ↩