generated from roboflow/template-python
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Search before asking
- I have searched the Supervision issues and found no similar feature requests.
Description
What's the problem?
I found two ways of computing mAP metric: sv.MeanAveragePrecision and sv.metrics.MeanAveragePrecision. However, I see here that sv.MeanAveragePrecision and a few other metrics will be deprecated.
Why is it important?
Both sv.MeanAveragePrecision and sv.metrics.MeanAveragePrecision give different results as shown in the following example.
import os
import numpy as np
import supervision as sv
from ultralytics import YOLO
# Download dataset
if not os.path.exists("/tmp/rf_animals"):
!wget https://universe.roboflow.com/ds/1LLwpXz2td?key=8JnJML5YF6 -O /tmp/rf_animals.zip
!unzip /tmp/dataset.zip -d /tmp/rf_animals
# Load dataset
dataset = sv.DetectionDataset.from_yolo("/tmp/rf_animals/train/images", "/tmp/rf_animals/train/labels", "/tmp/rf_animals/data.yaml")
# Inference
model = YOLO("yolov8s")
targets, detections = [], []
for image_path, image, target in dataset:
targets.append(target)
prediction = model(image, verbose=False)[0]
detection = sv.Detections.from_ultralytics(prediction)
detection = detection[np.isin(detection['class_name'], dataset.classes)]
detection.class_id = np.array([dataset.classes.index(class_name) for class_name in detection['class_name']])
detections.append(detection)
# Method #1
mAP = sv.metrics.MeanAveragePrecision().update(detections, targets).compute()
print(f"mAP50: {mAP.map50:.4f}")
# Method #2
mAP = sv.MeanAveragePrecision.from_detections(detections, targets)
print(f"mAP50: {mAP.map50:.4f}")Output
mAP50: 0.1553
mAP50: 0.2100
As per the docstrings, Method 1 computes the average precision, given the recall and precision curves following https://github.com/rafaelpadilla/Object-Detection-Metrics and Method 2 uses 101-point interpolation (COCO) method. People may end up using a mixture of these methods and may derive wrong conclusions.
Use case
No response
Additional
No response
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request