Skip to content

fix(metrics): harden scoring edge cases#2411

Merged
Borda merged 3 commits into
developfrom
medium/Dataset-2
Jul 7, 2026
Merged

fix(metrics): harden scoring edge cases#2411
Borda merged 3 commits into
developfrom
medium/Dataset-2

Conversation

@Borda

@Borda Borda commented Jul 7, 2026

Copy link
Copy Markdown
Member

This pull request introduces several important improvements and bug fixes to the detection metrics, focusing on stricter class ID validation, aligning mean average precision (mAP) with the COCO 101-point AP standard, improved handling of ignore flags, and enhanced per-class recall reporting. These changes improve metric correctness, robustness, and user transparency, and include expanded tests to verify the new behaviors.

Metrics calculation and correctness:

  • MeanAveragePrecision now uses the COCO 101-point AP averaging method, which may change some reported mAP values to be more accurate and consistent with the COCO standard. [1] [2] [3] [4] [5] [6]
  • ConfusionMatrix now strictly validates class IDs, rejecting negative or out-of-bounds values instead of wrapping them via integer casting, preventing subtle bugs in confusion matrix indexing. [1] [2] [3]

Ignore flag and COCO compatibility:

  • The ignore flag in targets is now preserved and respected in COCO-style evaluation, allowing users to explicitly exclude certain ground truths from scoring. [1] [2] [3] [4]

Mean Average Recall improvements:

  • MeanAverageRecallResult.recall_per_class now exposes per-class recall for each max-detection cutoff (e.g., @1, @10, @100), providing more granular recall analysis. The string representation and result shape have been updated accordingly. [1] [2] [3] [4] [5] [6] [7] [8]

Expanded and improved tests:

  • New tests ensure that negative and overflowing class IDs are rejected, that the AP calculation matches the COCO standard exactly, and that ignore flags and per-class recall are handled as intended. [1] [2] [3] [4]

These changes improve the reliability and interpretability of detection metrics, and may require users to recalibrate thresholds or update their code if they previously relied on the old behaviors.


part of #2408

@Borda Borda requested a review from Copilot July 7, 2026 11:12
@Borda Borda requested a review from SkalskiP as a code owner July 7, 2026 11:12
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87%. Comparing base (6a69197) to head (9b1210f).

Additional details and impacted files
@@           Coverage Diff           @@
##           develop   #2411   +/-   ##
=======================================
  Coverage       87%     87%           
=======================================
  Files           70      70           
  Lines        10198   10218   +20     
=======================================
+ Hits          8823    8843   +20     
  Misses        1375    1375           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens dataset import/export round-trips (Pascal VOC, COCO, LabelMe) and tightens dataset/classification equality semantics so that equality checks reflect ordered classes and annotation/image content more predictably across IO boundaries.

Changes:

  • Make DetectionDataset / ClassificationDataset equality stricter (ordered classes, ordered image_paths, and pixel-wise comparison for in-memory images); add Classifications.__eq__ using NumPy value equality.
  • Improve Pascal VOC serialization/parsing: round float coordinates on export, make polygon parsing order-agnostic, and fill bbox-only masks when force_masks=True.
  • Enforce COCO category-name uniqueness and improve LabelMe multi-component mask grouping to round-trip as a single detection via group_id.

Assessment (n/5):

  • Code quality: 4/5
  • Testing: 5/5
  • Documentation: 4/5

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/supervision/dataset/core.py Tightens dataset equality semantics and compares in-memory images by key + pixel values.
src/supervision/classification/core.py Adds value-based equality for Classifications (NumPy-aware).
src/supervision/dataset/formats/pascal_voc.py Rounds exported coords, robust polygon parsing, and force_masks rasterization for bbox-only objects.
src/supervision/dataset/formats/coco.py Rejects duplicate COCO category names to avoid ambiguous mappings.
src/supervision/dataset/formats/labelme.py Groups multi-component masks via group_id and reconstructs masks from grouped polygons.
tests/dataset/test_core.py Adds regression tests for stricter dataset and Classifications equality behavior.
tests/dataset/formats/test_pascal_voc.py Adds/updates tests for rounding, polygon parsing order independence, and bbox-mask filling with force_masks=True.
tests/dataset/formats/test_coco.py Adds a test ensuring duplicate category names raise a targeted ValueError.
tests/dataset/formats/test_labelme.py Extends tests to assert grouped export and that multi-component masks round-trip as one detection.
docs/changelog.md Documents the dataset IO/equality and format-round-trip fixes.

Comment thread src/supervision/dataset/formats/coco.py Outdated
Comment thread src/supervision/dataset/formats/labelme.py Outdated
- Use COCO 101-point AP averaging in the legacy mAP path so perfect and imperfect curves score consistently.

- Validate confusion-matrix class ids before indexing and preserve target ignore flags in the COCO-style evaluator.

- Keep mAR per-class recall for each max-detection cutoff and cover the scoring fixes with focused regressions.

---

Co-authored-by: Codex <codex@openai.com>
@Borda Borda force-pushed the medium/Dataset-2 branch from 87be35f to 3ecc719 Compare July 7, 2026 11:29
@Borda Borda changed the title fix(dataset): harden dataset export round trips fix(metrics): harden scoring edge cases Jul 7, 2026
@Borda Borda requested a review from Copilot July 7, 2026 11:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comment thread src/supervision/metrics/mean_average_recall.py
Borda and others added 2 commits July 7, 2026 16:30
- Return empty mAR scores with the same max-detection axis as non-empty results.
- Add an empty-input regression covering recall score and per-class result shapes.
- Update the public mAR docstring to describe per-image detection limits.

---

Co-authored-by: Codex <codex@openai.com>
@Borda Borda merged commit 814a226 into develop Jul 7, 2026
26 checks passed
@Borda Borda deleted the medium/Dataset-2 branch July 7, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants