The evaluation metrics are defined here:
examples/mnist/model.py: self.eval_metrics = [accuracy, self.get_criterion()]
The required values for them are defined in src/main.py:
drift_signal = detector.update(
value=0.0, # dummy value for base class compatibility
modelHarness=modelHarness,
reference_validation_metrics=[90, 1.0],
higher_is_better=[True, False],
)
These metrics/values are used to evaluate the performance of a model in src/drift_detection/detectors/model_performance_detector.py
This is convoluted and will be hard to understand/debug without proper documentation. I propose we have a list of metrics we support and each example chooses a subset and defines the desired values in their toml files
The evaluation metrics are defined here:
The required values for them are defined in
src/main.py:These metrics/values are used to evaluate the performance of a model in
src/drift_detection/detectors/model_performance_detector.pyThis is convoluted and will be hard to understand/debug without proper documentation. I propose we have a list of metrics we support and each example chooses a subset and defines the desired values in their toml files