Neptune Fetcher is designed to separate data retrieval capabilities from the regular neptune package. This separation makes data fetching more efficient and improves performance.

pip install neptune-fetcher

1. Set your Neptune API token and project name as environment variables:

   export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

   export NEPTUNE_PROJECT="workspace-name/project-name"

   For help, see https://docs-beta.neptune.ai/setup.

2. In your Python code, create a ReadOnlyProject instance:

   from neptune_fetcher import ReadOnlyProject
   
   my_project = ReadOnlyProject()

Now you have a Neptune project to operate on.

If you don't set the Neptune environment variables, you can pass your credentials through arguments when creating a project or run object.

To fetch experiments in bulk, call a fetching method on the project:

experiments_df = my_project.fetch_experiments_df(
    names_regex="tree/.*",
    columns=["sys/custom_run_id", "sys/modification_time"],
    query='(last(`accuracy`:floatSeries) > 0.88) AND (`learning_rate`:float < 0.01)',
)

To fetch metadata from an individual experiment or run, create and use a ReadOnlyRun object:

from neptune_fetcher import ReadOnlyRun

run = ReadOnlyRun(
    read_only_project=my_project,
    experiment_name="seagull-flying-skills",
)

# Fetch value
print(run["parameters/optimizer"].fetch())

# Fecth last value of metric
print(run["metrics/loss"].fetch_last())

# Fetch all metric values, with optional pre-fetching to speed up subsequent access to the field
run.prefetch_series_values(["metrics/accuracy"])
print(run["metrics/accuracy"].fetch_values())

For details, see the Neptune documentation:

• Fetch runs or experiments
• Fetch metadata from a run or experiment
• Neptune Query Language (NQL)

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for run in project.list_runs():
    print(run)  # dicts with identifiers

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for experiment in project.list_experiments():
    print(experiment)  # dicts with identifiers

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

runs_df = project.fetch_runs_df(
    columns=["sys/custom_run_id", "sys/modification_time"],
    columns_regex="tree/.*",  # added to columns specified with the "columns" parameter
)

from neptune_fetcher import ReadOnlyProject

project = ReadOnlyProject("workspace/project")

for run in project.fetch_read_only_runs(with_ids=["RUN-1", "RUN-2"]):
    run.prefetch(["parameters/optimizer", "parameters/init_lr"])

    print(run["parameters/optimizer"].fetch())
    print(run["parameters/init_lr"].fetch())

from neptune_fetcher import ReadOnlyProject, ReadOnlyRun

project = ReadOnlyProject("workspace/project")
run = ReadOnlyRun(project, with_id="TES-1")

run.prefetch(["parameters/optimizer", "parameters/init_lr"])
run.prefetch_series_values(["metrics/loss", "metrics/accuracy"], use_threads=True)

print(run["parameters/optimizer"].fetch())
print(run["parameters/init_lr"].fetch())
print(run["metrics/loss"].fetch_values())
print(run["metrics/accuracy"].fetch_values())

Neptune uses the RE2 regular expression library. For supported regex features and limitations, see the official syntax guide.

Representation of a Neptune project in a limited read-only mode.

Initialize with the ReadOnlyProject class constructor:

project = ReadOnlyProject("workspace/project", api_token="...")

Parameters:

Lists all runs of a project.

Each run is identified by Neptune ID (sys/id), custom ID (sys/custom_run_id) and, if set, name (sys/name).

Returns: Iterator of dictionaries with Neptune run identifiers, custom identifiers and names.

Example:

project = ReadOnlyProject()

for run in project.list_runs():
    print(run)

Lists all experiments of a project.

Each experiment is identified by:

• Neptune ID: sys/id
• (If set) Custom ID: sys/custom_run_id
• Name: sys/name

Example:

for experiment in project.list_experiments():
    print(experiment)

Returns: Iterator of dictionaries with Neptune experiment identifiers, custom identifiers and names.

Fetches a table containing Neptune IDs, custom run IDs and names of runs in the project.

Returns: pandas.DataFrame pandas.DataFrame with three columns (sys/id, sys/name and sys/custom_run_id) and one row for each run.

Example:

project = ReadOnlyProject()
df = project.fetch_runs()

Fetches a table containing Neptune IDs, custom IDs and names of experiments in the project.

Example:

df = project.fetch_experiments()

Returns: pandas.DataFrame with three columns (sys/id, sys/custom_run_id, sys/name) and one row for each experiment.

Fetches the runs' metadata and returns them as a pandas DataFrame.

Parameters:

Returns: pandas.DataFrame: A pandas DataFrame containing metadata of the fetched runs.

Examples:

Fetch all runs, with specific columns:

project = ReadOnlyProject()

runs_df = project.fetch_runs_df(
    columns=["sys/modification_time", "training/lr"]
)

Fetch all runs, with specific columns and extra columns that match a regex pattern:

runs_df = project.fetch_runs_df(
    columns=["sys/modification_time"],
    columns_regex="tree/.*",
)

Fetch runs by specific ID:

specific_runs_df = my_project.fetch_runs_df(custom_ids=["nostalgic_shockley", "high_albattani"])

Fetch runs by names that match a regex pattern:

specific_runs_df = my_project.fetch_runs_df(
    names_regex="tree_3[2-4]+"
)

Fetch runs with a complex query using NQL.

runs_df = my_project.fetch_runs_df(
    query='(last(`accuracy`:floatSeries) > 0.88) AND (`learning_rate`:float < 0.01)'
)

Fetches the experiments' metadata and returns them as a pandas DataFrame.

Parameters:

Returns: pandas.DataFrame: A pandas DataFrame containing metadata of the fetched experiments.

Examples:

Fetch all experiments with specific columns:

experiments_df = project.fetch_experiments_df(
    columns=["sys/custom_run_id", "sys/modification_time", "training/lr"]
)

Fetch all experiments with specific columns and extra columns that match a regex pattern:

experiments_df = project.fetch_experiments_df(
    columns=["sys/custom_run_id", "sys/modification_time"],
    columns_regex="tree/.*",
)

Fetch experiments by specific IDs:

specific_experiments_df = my_project.fetch_experiments_df(
    custom_ids=["nostalgic_shockley", "high_albattani"]
)

Use the Neptune Query Language to fetch experiments with a complex query. Note that for regular strings, the \ character needs to be escaped:

experiments_df = my_project.fetch_experiments_df(
    query='(`learning_rate`:float < 0.01) AND (`sys/name`:string MATCHES "experiment-\\\\d+")'
)

As a less cluttered alternative, pass a raw Python string to the query argument:

experiments_df = my_project.fetch_experiments_df(
    query=r'(`learning_rate`:float < 0.01) AND (`sys/name`:string MATCHES "experiment-\\d+")'
)

List runs of the project in the form of ReadOnlyRun.

Parameters:

Returns: Iterator of ReadOnlyRun objects.

Example:

project = ReadOnlyProject()

for run in project.fetch_read_only_runs(custom_ids=["nostalgic_shockley", "high_albattani"]):
    ...

Lists experiments of the project in the form of ReadOnlyRun.

Parameters:

Returns: Iterator of ReadOnlyRun objects.

Example:

project = ReadOnlyProject()

for run in project.fetch_read_only_experiments(names=["yolo-v2", "yolo-v3"]):
    ...

Representation of a Neptune run in a limited read-only mode.

Can be created

• with the class constructor:

  project = ReadOnlyProject()
  run = ReadOnlyRun(project, with_id="TES-1")

• or as a result of the fetch_read_only_runs() method of the ReadOnlyProject class:

  for run in project.fetch_read_only_runs(
      custom_ids=["nostalgic_shockley", "high_albattani"]):
      ...

Parameters:

Example:

from neptune_fetcher import ReadOnlyProject, ReadOnlyRun

project = ReadOnlyProject("workspace-name/project-name", api_token="...")
run = ReadOnlyRun(project, custom_id="high_albattani")

List of run field names.

A field is the location where a piece of metadata is stored in the run.

Returns: Iterator of run fields as strings.

Example:

for run in project.fetch_read_only_runs(custom_ids=["nostalgic_shockley", ...]):
    print(list(run.field_names))

Used to access a specific field of a run. See Available types.

Returns: An internal object used to operate on a specific field.

Example:

run = ReadOnlyRun(...)
custom_id = run["sys/custom_run_id"].fetch()

Pre-fetches a batch of fields to the internal cache.

Improves the performance of access to consecutive field values.

Supported Neptune field types:

• Boolean
• Datetime
• Float
• FloatSeries
• Integer
• ObjectState
• String
• StringSet

Parameters:

Example:

run = ReadOnlyRun(...)
run.prefetch(["parameters/optimizer", "parameter/init_lr"])
# No more calls to the API
print(run["parameters/optimizer"].fetch())
print(run["parameter/init_lr"].fetch())

Prefetches a batch of series to the internal cache. This method skips the non-existing attributes.

Improves the performance of access to consecutive field values. Works only for series (FloatSeries).

To speed up the fetching process, this method can use multithreading. To enable it, set the use_threads parameter to True.

By default, the maximum number of workers is 10. You can change this number by setting the NEPTUNE_FETCHER_MAX_WORKERS environment variable.

Parameters:

Example:

run.prefetch_series_values(["metrics/loss", "metrics/accuracy"])
# No more calls to the API
print(run["metrics/loss"].fetch_values())
print(run["metrics/accuracy"].fetch_values())

This section lists the available field types and data retrieval operations.

Retrieves a bool value either from the internal cache (see prefetch()) or from the API.

Example:

status = run["sys/failed"].fetch()

Retrieves a datetime.datetime value either from the internal cache (see prefetch()) or from the API.

Example:

created_at = run["sys/creation_time"].fetch()

Retrieves a float value either from the internal cache (see prefetch()) or from the API.

Example:

f1 = run["scores/f1"].fetch()

Retrieves the last value of a series, either from the internal cache (see prefetch()) or from the API.

Returns: Optional[float]

Example:

loss = run["loss"].fetch_last()

Retrieves all series values either from the internal cache (see prefetch_series_values()) or from the API.

Parameters:

Returns: pandas.DataFrame

Example:

values = run["loss"].fetch_values()

Retrieves an int value either from the internal cache (see prefetch()) or from the API.

Example:

batch_size = run["batch_size"].fetch()

Retrieves the state of a run either from the internal cache (see prefetch()) or from the API.

Returns: str

Example:

state = run["sys/state"].fetch()

Retrieves a str value either from the internal cache (see prefetch()) or from the API.

Example:

token = run["token"].fetch()

Retrieves a dict of str values either from the internal cache (see prefetch()) or from the API.

Example:

groups = run["sys/group_tags"].fetch()

This project is licensed under the Apache License Version 2.0. For more details, see Apache License Version 2.0.