[FR] Interactive Confidence Thresholding for the Model Evaluation Panel

[KostadinovShalon]

Proposal Summary

While I'm not 100% if this is not really possible, it'd be great to get object detection evaluation metrics updated by modifying the confidence threshold for different models, so different recall and precision values are obtained for different confidence thresholds.

Motivation

There are some metrics that depend on the confidence threshold and other than not. For instance, COCO mAP shouldn't depend on the confidence threshold because it's calculated using the precision-recall curve (which consideres different confidence thresholds virtually from 0 to 1). However, other metrics such as recall, precision or f1-score depend on the chosen confidence score. The examples in the Docs consider using only high-confidence views of the predictions, but it'd be nice to have a way to interactively see these metrics by changing the confidence threshold in the App.

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

Basically, updating confidence-score dependant metrics (recall, precision, f1 score...) when modifying the confidence score in the App but (potentially) keeping global metrics (such as mAP) unchanged. Alternatively, it'd also be useful to have something like the 'Scenarios' but filtering predictions also (such as using confidence threshold).

If I'm not wrong, the current way to mimic this behaviour would be to create different prediciton views considering different confidence thresholds and then create different evaluations for each one.

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• No. I cannot contribute this feature at this time (sorry, I'm probably not entirely familiar with FiftyOne development to contribute at this time)

I'm sorry if this is something that can be done already!