There's duplicate code in different LightningModules that calculates metrics.
To avoid this metric setup, update, logging, reset can be moved to a lightning callback which can then be used by different modules.
The callback should have an init where you specify what metrics you want to track: f1, recall precision, accuracy, ROC-AUC, PR-AUC x averaging micro, macro, weighted, per-class.
Macro averaging should happen on the full validation dataset, not minibatched like it currently is.
By tracking validation_step outputs and not relying on on_validation_epoch_end results for preds we can address #1 as well
One wrinkle is, how do we manage plot callbacks. Many of them rely on averaging=None metrics. Do they become helper functions to this one main metrics callback? That doesn't seem right. I suppose they should also use aggregate validation_step outputs.
There's duplicate code in different LightningModules that calculates metrics.
To avoid this metric setup, update, logging, reset can be moved to a lightning callback which can then be used by different modules.
The callback should have an init where you specify what metrics you want to track: f1, recall precision, accuracy, ROC-AUC, PR-AUC x averaging micro, macro, weighted, per-class.
Macro averaging should happen on the full validation dataset, not minibatched like it currently is.
By tracking validation_step outputs and not relying on on_validation_epoch_end results for preds we can address #1 as well
One wrinkle is, how do we manage plot callbacks. Many of them rely on averaging=None metrics. Do they become helper functions to this one main metrics callback? That doesn't seem right. I suppose they should also use aggregate validation_step outputs.