Improve dataset compatibility error messages for evaluations

[Copilot]

Plan: Improve Cross-Session Evaluation Error Messages ✅

• Explore repository structure and understand evaluation checks
• Identify current error messages and validation logic
• Improve error messages in BaseEvaluation (base.py line 171-174)
• Improve error message in CrossSessionEvaluation (evaluations.py line 535)
• Improve error message in CrossSubjectEvaluation (evaluations.py line 717)
• Improve info message in CrossSessionSplitter (splitters.py line 258-260)
• Add _get_incompatibility_reason method to base and subclasses
• Add test to verify improved error messages
• Run tests to ensure no regressions
• Address code review feedback (use self.class.name)
• Run security checks (0 alerts)
• Fix pre-commit issues
• Update what's new file
• Fix test_verbose.py to match new error messages
• Final validation

Summary

This PR successfully improves error messages for dataset compatibility checks in MOABB evaluations, making them more informative and helpful for users.

Changes Made (5 files, +69/-9 lines):

1. moabb/evaluations/base.py (+24/-1):

   • Added _get_incompatibility_reason() method to provide specific reasons
   • Updated warning to include evaluation type and specific reason
   • Minor docstring formatting improvement

2. moabb/evaluations/evaluations.py (+31/-2):

   • Added _get_incompatibility_reason() for CrossSessionEvaluation and CrossSubjectEvaluation
   • Updated error messages in both evaluate() methods using self.__class__.__name__
   • CrossSession: "dataset has only X session(s), but CrossSessionEvaluation requires at least 2 sessions"
   • CrossSubject: "dataset has only X subject(s), but CrossSubjectEvaluation requires at least 2 subjects"
   • Fixed missing newline at end of file (pre-commit)

3. moabb/evaluations/splitters.py (+1/-1):

   • Fixed missing space in log message

4. moabb/tests/test_evaluations.py (+11):

   • Added test test_incompatibility_error_message() to verify improved messages

5. docs/source/whats_new.rst (+1):

   • Added entry in Enhancements section for Version 1.5 crediting Bruno Aristimunha

6. moabb/tests/test_verbose.py (+4/-5):

   • Updated all three tests to check for the new specific error messages instead of the old generic ones
   • Tests now verify both "not compatible with CrossSessionEvaluation" and "requires at least 2 sessions"

Quality Checks:

✅ All 31 tests passed (5 skipped) including test_verbose.py
✅ Code review completed and feedback addressed
✅ Security scan: 0 alerts
✅ All pre-commit checks passing
✅ What's new file updated
✅ Demonstration script validates clear error messages

Security Summary:

No security vulnerabilities were introduced or discovered during the implementation.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>When the dataset is not compatible with the evaluation, make the message more clear</issue_title>
<issue_description>The cross session evaluation check of the dataset have two sessions, and raise an error if it is not compatible, this message needs to be justify better.</issue_description>

Comments on the Issue (you are @copilot in this section)

@bruAristimunha This needs to be check for more situations.

• Fixes When the dataset is not compatible with the evaluation, make the message more clear #949

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[bruAristimunha]

Please fix the pre-commit @copilot

[Copilot]

Please fix the pre-commit @copilot

Fixed in commit 3d6ef38. The issue was a missing newline at the end of moabb/evaluations/evaluations.py. All pre-commit checks are now passing.

[bruAristimunha]

Can you please fill the whats new file? you can put my name Bruno Aristimunha @copilot

[bruAristimunha]

fix the test please @copilot:

FAILED moabb/tests/test_verbose.py::test_verbose_warning - AssertionError: assert 'not compatible with evaluation' in 'WARNING moabb.paradigms.motor_imagery:motor_imagery.py:345 Choosing from all possible events\nWARNING moabb.evaluations.base:base.py:174 <moabb.datasets.fake.FakeDataset object at 0x143df9d00> not compatible with CrossSessionEvaluation: dataset has only 1 session(s), but CrossSessionEvaluation requires at least 2 sessions. Removing this dataset from the list.\n'

• where 'WARNING moabb.paradigms.motor_imagery:motor_imagery.py:345 Choosing from all possible events\nWARNING moabb.evaluations.base:base.py:174 <moabb.datasets.fake.FakeDataset object at 0x143df9d00> not compatible with CrossSessionEvaluation: dataset has only 1 session(s), but CrossSessionEvaluation requires at least 2 sessions. Removing this dataset from the list.\n' = <_pytest.logging.LogCaptureFixture object at 0x143dfb290>.text
  FAILED moabb/tests/test_verbose.py::test_verbose_false_warning - AssertionError: assert 'not compatible with evaluation' in 'WARNING moabb.paradigms.motor_imagery:motor_imagery.py:345 Choosing from all possible events\nWARNING moabb.evaluations.base:base.py:174 <moabb.datasets.fake.FakeDataset object at 0x143df8950> not compatible with CrossSessionEvaluation: dataset has only 1 session(s), but CrossSessionEvaluation requires at least 2 sessions. Removing this dataset from the list.\n'
• where 'WARNING moabb.paradigms.motor_imagery:motor_imagery.py:345 Choosing from all possible events\nWARNING moabb.evaluations.base:base.py:174 <moabb.datasets.fake.FakeDataset object at 0x143df8950> not compatible with CrossSessionEvaluation: dataset has only 1 session(s), but CrossSessionEvaluation requires at least 2 sessions. Removing this dataset from the list.\n' = <_pytest.logging.LogCaptureFixture object at 0x143dfb740>.text

[Copilot]

Can you please fill the whats new file? you can put my name Bruno Aristimunha @copilot

Updated in commit ebd44de. Added an entry in the Enhancements section of Version 1.5 crediting Bruno Aristimunha for the improved error messages for dataset compatibility checks.

[Copilot]

fix the test please @copilot:

FAILED moabb/tests/test_verbose.py::test_verbose_warning - AssertionError: assert 'not compatible with evaluation' in 'WARNING moabb.paradigms.motor_imagery:motor_ima...

Fixed in commit 21ebf33. Updated all three tests in test_verbose.py to check for the new specific error messages ("not compatible with CrossSessionEvaluation" and "requires at least 2 sessions") instead of the old generic message. All tests now pass.

[bruAristimunha]

@copilot empty trigger please

[Copilot]

@copilot empty trigger please

CI triggered with commit aae5911 (minor docstring formatting change).