fix(metrics): harden scoring edge cases#2411
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #2411 +/- ##
=======================================
Coverage 87% 87%
=======================================
Files 70 70
Lines 10198 10218 +20
=======================================
+ Hits 8823 8843 +20
Misses 1375 1375 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens dataset import/export round-trips (Pascal VOC, COCO, LabelMe) and tightens dataset/classification equality semantics so that equality checks reflect ordered classes and annotation/image content more predictably across IO boundaries.
Changes:
- Make
DetectionDataset/ClassificationDatasetequality stricter (orderedclasses, orderedimage_paths, and pixel-wise comparison for in-memory images); addClassifications.__eq__using NumPy value equality. - Improve Pascal VOC serialization/parsing: round float coordinates on export, make polygon parsing order-agnostic, and fill bbox-only masks when
force_masks=True. - Enforce COCO category-name uniqueness and improve LabelMe multi-component mask grouping to round-trip as a single detection via
group_id.
Assessment (n/5):
- Code quality: 4/5
- Testing: 5/5
- Documentation: 4/5
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/supervision/dataset/core.py |
Tightens dataset equality semantics and compares in-memory images by key + pixel values. |
src/supervision/classification/core.py |
Adds value-based equality for Classifications (NumPy-aware). |
src/supervision/dataset/formats/pascal_voc.py |
Rounds exported coords, robust polygon parsing, and force_masks rasterization for bbox-only objects. |
src/supervision/dataset/formats/coco.py |
Rejects duplicate COCO category names to avoid ambiguous mappings. |
src/supervision/dataset/formats/labelme.py |
Groups multi-component masks via group_id and reconstructs masks from grouped polygons. |
tests/dataset/test_core.py |
Adds regression tests for stricter dataset and Classifications equality behavior. |
tests/dataset/formats/test_pascal_voc.py |
Adds/updates tests for rounding, polygon parsing order independence, and bbox-mask filling with force_masks=True. |
tests/dataset/formats/test_coco.py |
Adds a test ensuring duplicate category names raise a targeted ValueError. |
tests/dataset/formats/test_labelme.py |
Extends tests to assert grouped export and that multi-component masks round-trip as one detection. |
docs/changelog.md |
Documents the dataset IO/equality and format-round-trip fixes. |
- Use COCO 101-point AP averaging in the legacy mAP path so perfect and imperfect curves score consistently. - Validate confusion-matrix class ids before indexing and preserve target ignore flags in the COCO-style evaluator. - Keep mAR per-class recall for each max-detection cutoff and cover the scoring fixes with focused regressions. --- Co-authored-by: Codex <codex@openai.com>
- Return empty mAR scores with the same max-detection axis as non-empty results. - Add an empty-input regression covering recall score and per-class result shapes. - Update the public mAR docstring to describe per-image detection limits. --- Co-authored-by: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several important improvements and bug fixes to the detection metrics, focusing on stricter class ID validation, aligning mean average precision (mAP) with the COCO 101-point AP standard, improved handling of ignore flags, and enhanced per-class recall reporting. These changes improve metric correctness, robustness, and user transparency, and include expanded tests to verify the new behaviors.
Metrics calculation and correctness:
MeanAveragePrecisionnow uses the COCO 101-point AP averaging method, which may change some reported mAP values to be more accurate and consistent with the COCO standard. [1] [2] [3] [4] [5] [6]ConfusionMatrixnow strictly validates class IDs, rejecting negative or out-of-bounds values instead of wrapping them via integer casting, preventing subtle bugs in confusion matrix indexing. [1] [2] [3]Ignore flag and COCO compatibility:
ignoreflag in targets is now preserved and respected in COCO-style evaluation, allowing users to explicitly exclude certain ground truths from scoring. [1] [2] [3] [4]Mean Average Recall improvements:
MeanAverageRecallResult.recall_per_classnow exposes per-class recall for each max-detection cutoff (e.g., @1, @10, @100), providing more granular recall analysis. The string representation and result shape have been updated accordingly. [1] [2] [3] [4] [5] [6] [7] [8]Expanded and improved tests:
These changes improve the reliability and interpretability of detection metrics, and may require users to recalibrate thresholds or update their code if they previously relied on the old behaviors.
part of #2408