Are functions only:

• user-definable inside of the same file
• importable from a public PyPI package
  ?

Or can you do something chaotic like import add_one from my_python_model in my_second_python_model.py?

Related but broader question: I assume there are no Python macros? Either Jinja to template python or a global space for add_one to be defined once

Currently you cannot do that. It is possible to support(not very pretty way). @jtcohen6 maybe that's something to call out that we don't support it.

I really consider this an open question for us!

I'm disinclined to do this one, given the chaos it could lead to, and over-reliance on the local file system:

Or can you do something chaotic like import add_one from my_python_model in my_second_python_model.py?

I think we have two good options to pursue:

• Some platforms support registering these functions as "named" / persistent UDFs. This would be the case for "dbt should know about functions." These UDFs have some drawbacks; inconsistent support; and the main use case seems to be defining a Python UDF that then gets called from within a SQL model. That seems valuable for exposing a prediction function output of a trained ML model, less so for a utility like add_one.
• Potentially more promising: most of these platforms allow you to upload your own home-spun "Python package," as a .zip, ==. You could then import it and use its functions, as if it were a public PyPI package.

I agree that you shouldn't be able to do arbitrary imports from functions defined in other models in the models/ directory.

most of these platforms allow you to upload your own home-spun "Python package," as a .zip, ==. You could then import it and use its functions, as if it were a public PyPI package

Where would the code that gets zipped and uploaded be source controlled? This feels very analogous to the before-state of https://discourse.getdbt.com/t/using-dbt-to-manage-user-defined-functions/18 - code is magically available but no one really knows where it came from.

If we can resolve that, I do like the idea of an internal functions package.

Would finding .py functions defined in the macros directory (or add functions/ to your macro-paths) and embedding them all into an internal package be good enough at breaking reliance on the filesystem? This assumes that you're disinclined to enable imports from inside a model because it would be bad practice, not that it would be hard to implement technically.

Where would the code that gets zipped and uploaded be source controlled?

Yup — 100% needs to live within the dbt project, and be version-controlled along with it.

I think that "upload" step will require different behind-the-scenes implementations across different data warehouses. We could even opt for UDFs as the backing implementation, if a given warehouse doesn't support arbitrary package upload. There's risk of a leaky abstraction here, but it feels like a place where dbt can help in a big way, and where we won't see the full value of Python models without some answer here.

Questions we need to answer:

• Is this "upload" step part of dbt deps? What if these reusable methods live in the root project, rather than a dbt package?
• How to support the transition from local → cloud development (if we want to preserve that workflow)?
• Those Python models should ideally be (a) pure functions with (b) thorough unit testing. Can dbt help here? Or should we encourage standard best practices Python development (pytest etc)?

@lostmygithubaccount @ChenyuLInx This feels like one of our highest-priority spikes during the v1.3 beta → final period.

Currently you cannot do that. It is possible to support(not very pretty way). @jtcohen6 maybe that's something to call out that we don't support it.

Should we add that we don't delete the notebook so you can use it for interactive development of model and bring the code back?

Good call!

Might be worth mentioning also that what says .sql below would need to be replaced by .py for Python. No need to go into too much detail in that page but just calling it out.

I tried a few different permutations of this, but I wasn't happy with any. I'm going to stick with minimal note for now. We have a lot of future work to do, to figure out the right ways to integrate Python capabilities / references into the existing docs that say SQL, SQL, .sql

@dataders This reads a bit too harsh right now. My goal here is to set clear expectations for any BQ users around both (a) the setup experience, (b) the possibility that this functionality is cut from v1.3 (final). Definitely open to your wordsmithing

not sure if this was a typo so just flagging

consider using Philadelphia unstead of Philly to be less US centric, less colloquial

does the end user ever have to modify these parameters to be different than dbt and session? If not, why expose them to users at all?

The end user needs to access classmethods of dbt, and some properties of session as well. We think, better to document those and make it clearer where they're coming from, rather than pulling them out of the aether.

We could have gone for a "script"-style approach, where dbt and session appear to be available "globally":

def some_fn:
    ...
    
some_df = dbt.ref("some_model")
final_df = some_df.apply(some_fn)

final_df

Where really we just take users' code, and turn it into:

def model(dbt, session):
    def some_fn:
        ...
    
    some_df = dbt.ref("some_model")
    final_df = some_df.apply(some_fn)

    return final_df  # we add this return?

It's still an option we could consider, if people like that a lot better! (@ChenyuLInx)

are these lazily-evaluated references? or does dbt.source() read the reference into memory at time of definition?

Lazy evaluation if you're using a DataFrame API worth its salt — so yes! As soon as you convert to Pandas, though, that will read everything into memory.

are we recommending to initially install all the packages you might use before creating python models in lieu of specifying requirements as part of a Python model's specifification? As in, it's not easy/possible/practical to configure model-specific python environments, so just do it once manually outside of dbt?

This totally varies based on the backend / implementation. On Databricks, it is possible to tell a cluster to run pip install <some_package> in a first notebook cell. That can take quite a bit of time, though. Their recommendation (and it makes sense to me) is to configure the needed packages on the cluster ahead of time.

By configuring packages in that way, it does mean that two models cannot have conflicting package version requirements. I think that's a recipe for disaster, anyway...

I was trying to be a little cute here. Not sure if I'm achieving it, might be better to cut altogether: