Exclude Classes from Average (e.g. macro f1-score)

[plonerma]

I would like to exclude certain classes from the averaging in the classification metrics (accuracy, f1, recall, precision, etc). However, I would want to still include classes in the prediction (e.g. in an NER problem, a tagger might predict that a token is not part of a named entity and therefor predict an "out" or "O"). As they still affect the false positives and negatives, ignore_index does not work in this case. What would be the best way to solve this using the torchmetrics framework?

As far as I can see, the only option at the moment would be to set average to none and compute the desired averages post-hoc. However, this would require writing a wrapper for all of the used classification metrics (instead of addressing the underlying StatScores)

I would appreciate any pointer in how to solve this, or how to best address this in a PR if that is a feature which might be interesting in other use-cases as well.

[plonerma]

I opened a feature-request in the issues regarding this.