feat(integration/dagster): dagster connector to emit dagster pipeline #8384
Conversation
github-actions
bot
added
ingestion
|
@shubhamjagtap639, please, can you fix the linter errors? |
@treff7es There is no linter errors. CI is getting failed bacause of below error: ERROR: Cannot install apache-airflow[snowflake]==2.0.2, apache-airflow[snowflake]==2.1.0, apache-airflow[snowflake]==2.1.1, apache-airflow[snowflake]==2.1.2, apache-airflow[snowflake]==2.1.3, apache-airflow[snowflake]==2.1.4, apache-airflow[snowflake]==2.2.5, apache-airflow[snowflake]==2.3.0, apache-airflow[snowflake]==2.3.1, apache-airflow[snowflake]==2.3.2, apache-airflow[snowflake]==2.3.3, apache-airflow[snowflake]==2.3.4, apache-airflow[snowflake]==2.4.0, apache-airflow[snowflake]==2.4.1, apache-airflow[snowflake]==2.4.2, apache-airflow[snowflake]==2.4.3, apache-airflow[snowflake]==2.5.0, apache-airflow[snowflake]==2.5.1, apache-airflow[snowflake]==2.5.2, apache-airflow[snowflake]==2.5.3, apache-airflow[snowflake]==2.6.0, apache-airflow[snowflake]==2.6.1, apache-airflow[snowflake]==2.6.2, apache-airflow[snowflake]==2.6.3 and apache-airflow~=2.2.0 because these package versions have conflicting dependencies. |
…ubhamjagtap639/datahub into dagster-ingestion-integration
.github/workflows/dagster-plugin.yml
Outdated
| - python-version: "3.10" | ||
| extraPythonRequirement: "dagster>=1.3.3" | ||
| - python-version: "3.10" | ||
| extraPythonRequirement: "dagster>=1.3.3" |
There was a problem hiding this comment.
This matrix doesn't look right. We've added such test matrix for airflow as different versions of airflow have very different dependencies and some changes in logic.
For dagster, looks like its always dagster<=1.3.3 so extraPythonRequirement is not needed. The only variations are in python-version to run on python 3.7 and python 3.10 - similar to this - https://github.com/datahub-project/datahub/blob/master/.github/workflows/metadata-ingestion.yml#L34.
metadata-ingestion/setup.py
Outdated
| @@ -392,6 +392,9 @@ def get_long_description(): | |||
| "powerbi-report-server": powerbi_report_server, | |||
| "vertica": sql_common | {"vertica-sqlalchemy-dialect[vertica-python]==0.0.8"}, | |||
| "unity-catalog": databricks | sqllineage_lib, | |||
| "dagster": { | |||
There was a problem hiding this comment.
This should not be required. I think, we should allow only - pip install acryl-datahub-dagster-plugin and not pip install acryl-datahub[dagster]. Its there in case of airflow for backward compatibility.
metadata-ingestion/setup.py
Outdated
| @@ -647,6 +650,7 @@ def get_long_description(): | |||
| "file = datahub.ingestion.reporting.file_reporter:FileReporter", | |||
| ], | |||
| "apache_airflow_provider": ["provider_info=datahub_provider:get_provider_info"], | |||
| "dagster_provider": ["provider_info=datahub_dagster_plugin:get_provider_info"], | |||
There was a problem hiding this comment.
I think, this is airflow specific thing and can be removed unless dagster also needs it.
| @@ -0,0 +1,48 @@ | |||
| from dagster import ( | |||
There was a problem hiding this comment.
Is this required to be included in plugin source code that gets included in pypi package ?
Checklist