Skip to content

Deprecation warning for sv.MeanAveragePrecision and other outdated metrics #1777

@patel-zeel

Description

@patel-zeel

Search before asking

  • I have searched the Supervision issues and found no similar feature requests.

Description

What's the problem?

I found two ways of computing mAP metric: sv.MeanAveragePrecision and sv.metrics.MeanAveragePrecision. However, I see here that sv.MeanAveragePrecision and a few other metrics will be deprecated.

Why is it important?

Both sv.MeanAveragePrecision and sv.metrics.MeanAveragePrecision give different results as shown in the following example.

import os
import numpy as np
import supervision as sv
from ultralytics import YOLO

# Download dataset
if not os.path.exists("/tmp/rf_animals"):
    !wget https://universe.roboflow.com/ds/1LLwpXz2td?key=8JnJML5YF6 -O /tmp/rf_animals.zip
    !unzip /tmp/dataset.zip -d /tmp/rf_animals

# Load dataset
dataset = sv.DetectionDataset.from_yolo("/tmp/rf_animals/train/images", "/tmp/rf_animals/train/labels", "/tmp/rf_animals/data.yaml")

# Inference
model = YOLO("yolov8s")
targets, detections = [], []
for image_path, image, target in dataset:
    targets.append(target)
    
    prediction = model(image, verbose=False)[0]
    detection = sv.Detections.from_ultralytics(prediction)
    detection = detection[np.isin(detection['class_name'], dataset.classes)]
    detection.class_id = np.array([dataset.classes.index(class_name) for class_name in detection['class_name']])
    detections.append(detection)
    
# Method #1
mAP = sv.metrics.MeanAveragePrecision().update(detections, targets).compute()
print(f"mAP50: {mAP.map50:.4f}")

# Method #2
mAP = sv.MeanAveragePrecision.from_detections(detections, targets)
print(f"mAP50: {mAP.map50:.4f}")

Output

mAP50: 0.1553
mAP50: 0.2100

As per the docstrings, Method 1 computes the average precision, given the recall and precision curves following https://github.com/rafaelpadilla/Object-Detection-Metrics and Method 2 uses 101-point interpolation (COCO) method. People may end up using a mixture of these methods and may derive wrong conclusions.

Use case

No response

Additional

No response

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions