Skip to content

MarkusSagen/kedro-neptune

 
 

Repository files navigation

Neptune + Kedro integration

Kedro plugin for experiment tracking and metadata management. It lets you browse, filter and sort runs in a nice UI, visualize node/pipeline metadata, and compare pipelines.

What will you get with this integration?

  • browse, filter, and sort your model training runs
  • compare nodes and pipelines on metrics, visual node outputs, and more
  • display all pipeline metadata including learning curves for metrics, plots, and images, rich media like video and audio or interactive visualizations from Plotly, Altair, or Bokeh
  • and do whatever else you would expect from a modern ML metadata store

image Kedro pipeline metadata in custom dashboard in the Neptune web app

Note: The Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow.

Resources

Example

On the command line:

pip install kedro neptune[kedro]
kedro new --starter=pandas-iris

In your Kedro project directory:

kedro neptune init

In a pipeline node, in nodes.py:

import neptune

# Add a Neptune run handler to the report_accuracy() function

def report_accuracy(
    y_pred: pd.Series,
    y_test: pd.Series,
    neptune_run: neptune.handler.Handler,
) -> None:
    accuracy = (y_pred == y_test).sum() / len(y_test)
    logger = logging.getLogger(__name__)
    logger.info("Model has accuracy of %.3f on test data.", accuracy)

    # Log metrics to the Neptune run
    neptune_run["nodes/report/accuracy"] = accuracy * 100

# Add the Neptune run handler to the Kedro pipeline
node(
    func=report_accuracy,
    inputs=["y_pred", "y_test", "neptune_run"],
    outputs=None,
    name="report_accuracy",
)

On the command line, run the Kedro pipeline:

kedro run

Support

If you got stuck or simply want to talk to us, here are your options:

  • Check our FAQ page
  • You can submit bug reports, feature requests, or contributions directly to the repository.
  • Chat! When in the Neptune application click on the blue message icon in the bottom-right corner and send a message. A real person will talk to you ASAP (typically very ASAP),
  • You can just shoot us an email at [email protected]

About

📌 Track & manage metadata, visualize & compare Kedro pipelines in a nice UI.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%