Is your feature request related to a problem? Please describe.
When running a PythonTransformation for offline feature retrieval, an exception is raised here. I understand that python transformations on very large datasets do not scale well. However, in some cases, a user might still want to apply a python transformation to a smaller dataset in the offline store. Due to this error (and the missing implementation of PythonTransformation().transform_arrow()), this is currently not possible.
Describe the solution you'd like
Instead of raising an exception, we could still run the transformation, and also log a warning indicating that it may not scale well. On top of that we can implement the functionality in transform_arrow to make the transformation happen.
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
When running a
PythonTransformationfor offline feature retrieval, an exception is raised here. I understand thatpythontransformations on very large datasets do not scale well. However, in some cases, a user might still want to apply apythontransformation to a smaller dataset in the offline store. Due to this error (and the missing implementation ofPythonTransformation().transform_arrow()), this is currently not possible.Describe the solution you'd like
Instead of raising an exception, we could still run the transformation, and also log a warning indicating that it may not scale well. On top of that we can implement the functionality in
transform_arrowto make the transformation happen.Describe alternatives you've considered
Additional context