Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CT-983] Support dbt Python models in OSS Apache Spark #415

Open
jtcohen6 opened this issue Aug 3, 2022 · 5 comments
Open

[CT-983] Support dbt Python models in OSS Apache Spark #415

jtcohen6 opened this issue Aug 3, 2022 · 5 comments
Labels
enhancement New feature or request help_wanted Extra attention is needed python_models issues related to python model

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Aug 3, 2022 •
edited by dataders
Loading

Context: #407

The current implementation depends on Databricks APIs that are not available in OSS Apache Spark. We would like help from knowledgeable and interested community members, who could spec out an implementation using Spark-only functionality.

The entry point is submit_python_job:

dbt-spark/dbt/adapters/spark/impl.py

Line 392 in 7f6cffe

def submit_python_job(self, parsed_model: dict, compiled_code: str, timeout=None):

potentially useful Spark doc: Submitting Applications
@jtcohen6 jtcohen6 added enhancement New feature or request help_wanted Extra attention is needed python_models issues related to python model labels Aug 3, 2022
@github-actions github-actions bot changed the title Support dbt Python models in OSS Apache Spark [CT-983] Support dbt Python models in OSS Apache Spark Aug 3, 2022
@lostmygithubaccount
Copy link

let us know if you'd like to help on this issue!

@Waltherr
Copy link

Waltherr commented Nov 3, 2022 via email

@Adricu8
Copy link

Adricu8 commented Feb 28, 2023

Hi, is there any update or current plan on this?

@huydeelll
Copy link

Would like to vouch that this will be an important features, and open dbt up to a different data engineers who mostly work with Spark but at the same time wanted the rigor and data quality framework of dbt.

@timvw
Copy link

timvw commented Nov 21, 2023

I have some non-production grady sample that uses the same approach as duckdb to run python models.. https://github.com/timvw/dbt-spark/tree/support-sparksession-python-local

@ben-schreiber ben-schreiber mentioned this issue Feb 11, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help_wanted Extra attention is needed python_models issues related to python model
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@timvw @jtcohen6 @Adricu8 @Waltherr @lostmygithubaccount @huydeelll