How to calculate metrics for a Roboflow exported YOLO11 dataset?

[robmarkcole]

I've found in the cheatsheet:

import supervision as sv
from supervision.metrics import F1Score

predictions = sv.Detections(...)
targets = sv.Detections(...)

f1_metric = F1Score()
f1_result = f1_metric.update(predictions, targets).compute()

print(f1_result)
print(f1_result.f1_50)
print(f1_result.small_objects.f1_50)           

How should I pass the image & labels to sv.Detection? My objective is to overlay the prediction on the truth, but more significantly, to calculate the metrics for that image (and also to filter e.g. by area of detection)

[galafis]

Great question, @robmarkcole! I'll provide a comprehensive guide for calculating metrics on YOLO11 datasets exported from Roboflow using supervision.

Complete Workflow:

1. Loading Ground Truth from YOLO Format

import supervision as sv
import numpy as np
from pathlib import Path

# Load dataset using DetectionDataset
dataset_path = "path/to/your/yolo/dataset"
dataset = sv.DetectionDataset.from_yolo(
    images_directory_path=f"{dataset_path}/images",
    annotations_directory_path=f"{dataset_path}/labels",
    data_yaml_path=f"{dataset_path}/data.yaml"
)

# Convert to Detections for a specific image
image_name = "your_image.jpg"
ground_truth = dataset[image_name].detections

2. Getting Predictions from YOLO11

from ultralytics import YOLO

# Load your trained model
model = YOLO("path/to/your/best.pt")

# Run inference
image_path = f"{dataset_path}/images/{image_name}"
results = model(image_path)

# Convert to supervision Detections
predictions = sv.Detections.from_ultralytics(results[0])

3. Calculate Metrics

from supervision.metrics import F1Score, MeanAveragePrecision

# Initialize metrics
f1_metric = F1Score()
map_metric = MeanAveragePrecision()

# Update with predictions and ground truth
f1_result = f1_metric.update(predictions, ground_truth).compute()
map_result = map_metric.update(predictions, ground_truth).compute()

# Print results
print(f"[email protected]: {f1_result.f1_50:.3f}")
print(f"[email protected]: {f1_result.f1_75:.3f}")
print(f"[email protected]: {map_result.map_50:.3f}")
print(f"[email protected]:0.95: {map_result.map_50_95:.3f}")

4. Filter by Detection Area

# Filter detections by area
def filter_by_area(detections, min_area=100, max_area=10000):
    areas = detections.area
    mask = (areas >= min_area) & (areas <= max_area)
    return detections[mask]

# Apply filtering
filtered_predictions = filter_by_area(predictions, min_area=500)
filtered_ground_truth = filter_by_area(ground_truth, min_area=500)

# Calculate metrics on filtered data
f1_filtered = f1_metric.update(filtered_predictions, filtered_ground_truth).compute()
print(f"[email protected] (filtered): {f1_filtered.f1_50:.3f}")

5. Visualization: Overlay Predictions on Ground Truth

import cv2

# Load image
image = cv2.imread(image_path)

# Create annotators
box_annotator = sv.BoxAnnotator(color_lookup=sv.ColorLookup.CLASS)
label_annotator = sv.LabelAnnotator(color_lookup=sv.ColorLookup.CLASS)

# Annotate ground truth (green boxes)
gt_image = box_annotator.annotate(image.copy(), ground_truth, color=sv.Color.GREEN)
gt_image = label_annotator.annotate(gt_image, ground_truth, labels=["GT"] * len(ground_truth))

# Annotate predictions (red boxes)
final_image = box_annotator.annotate(gt_image, predictions, color=sv.Color.RED)
final_image = label_annotator.annotate(final_image, predictions, labels=["PRED"] * len(predictions))

# Save result
cv2.imwrite("comparison.jpg", final_image)

6. Batch Processing Multiple Images

# Process entire validation set
f1_metric = F1Score()

for image_name in dataset.images.keys():
    # Get ground truth
    ground_truth = dataset[image_name].detections
    
    # Get predictions
    image_path = f"{dataset_path}/images/{image_name}"
    results = model(image_path)
    predictions = sv.Detections.from_ultralytics(results[0])
    
    # Update metrics
    f1_metric.update(predictions, ground_truth)

# Compute final metrics
final_f1 = f1_metric.compute()
print(f"Dataset [email protected]: {final_f1.f1_50:.3f}")
print(f"Small objects [email protected]: {final_f1.small_objects.f1_50:.3f}")
print(f"Medium objects [email protected]: {final_f1.medium_objects.f1_50:.3f}")
print(f"Large objects [email protected]: {final_f1.large_objects.f1_50:.3f}")

Key Points:

• DetectionDataset.from_yolo() handles YOLO format parsing automatically
• sv.Detections.from_ultralytics() converts YOLO11 results to supervision format
• Metrics are computed incrementally - call update() for each image, then compute() for final results
• Object size filtering (small/medium/large) is built into the metrics
• Visualization uses different colors to distinguish predictions from ground truth

This approach gives you comprehensive evaluation capabilities with professional-grade metrics computation!

Best regards,
Gabriel

[robmarkcole]

Many thanks @galafis !