Skip to content

Design a better way to deal with metrics for model evaluation (for the drift detector) #56

@anagainaru

Description

@anagainaru

The evaluation metrics are defined here:

examples/mnist/model.py:        self.eval_metrics = [accuracy, self.get_criterion()]

The required values for them are defined in src/main.py:

        drift_signal = detector.update(
            value=0.0,  # dummy value for base class compatibility
            modelHarness=modelHarness,
            reference_validation_metrics=[90, 1.0],
            higher_is_better=[True, False],
        )

These metrics/values are used to evaluate the performance of a model in src/drift_detection/detectors/model_performance_detector.py
This is convoluted and will be hard to understand/debug without proper documentation. I propose we have a list of metrics we support and each example chooses a subset and defines the desired values in their toml files

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions