Leaderboard Results after new PR

[TheItCrOw]

Hey @liamdugan,

I've seen you updated the metrics, which is good! But I'm not getting why Luminar is now - at least on the leaderboard - seemingly broken with a steady 0.5 score over all categories for AUROC and null for FPR? I've also checked the PR log of the change and it seems Luminar produces its old values normally? Also, the way I understand it is that FPR@5% should just be like in the old evaluation?

I'm asking because literally yesterday, we got reviewer responses for our paper and I wanted to link the leaderboard, as that was a request, but I can't really do that right now, which is extremely awkward for us.

[liamdugan]

Hey @TheItCrOw,

I apologize for not proactively emailing you about this. I should have let you know in advance of the update going live.

As part of the new update, we filter out all detectors that do not achieve the target false positive rate. Unfortunately both of your previous evaluation results on the older leaderboard did not achieve the target 5% false positive rate and were therefore on the leaderboard as an error.

The Luminar submission had the following distribution of scores on the dataset
Counter({1.0: 672000})

And the Gaussian Extreme submission had the following distribution of scores on the dataset
Counter({1.0: 652512, 0.0: 19488}) (with 0.0 scores not always on human data)

Their false positive rates were:

"0.05": {
      "abstracts": 1.0,
      "books": 1.0,
      "news": 1.0,
      "poetry": 1.0,
      "recipes": 1.0,
      "reddit": 1.0,
      "reviews": 1.0,
      "wiki": 1.0
    },

and

"0.05": {
      "abstracts": 0.98,
      "books": 0.97,
      "news": 0.995,
      "poetry": 0.98,
      "recipes": 0.97,
      "reddit": 0.97,
      "reviews": 0.975,
      "wiki": 0.955
    },

respectively.

The comment printed by the bot on the PR here is using the old evaluation code and is a bug. It shouldn't have been printed

Please let me know if you have further questions about this. I'm happy to correspond more.

[TheItCrOw]

Hey @liamdugan,

Thanks for the rundown - I get it now after seeing the prediction scores and your explanation!

Correct me if I'm wrong, but this update was also a major bug fix, wasn’t it? Shouldn’t the FPR have always been checked, since that was the whole point of the metric? Without it, we got a false sense of our model's capabilities. I mean, we got a perfect score with just a sequence of 1s, which means I could’ve generated my submission programmatically without even building a model.

[liamdugan]

Yes this is a pretty major bug fix. We should always have been checking for this. Surprisingly, neither I nor any of our users caught this until now.

Thankfully the only two submissions that were affected were Luminar and Gaussian Extreme. Every other submission on the leaderboard (and ones we tested internally for the paper) were able to achieve at least 5% FPR on each individual sub-domain of RAID.

I'm sorry again that our evaluation script gave you a false impression of your model capabilities. I'm happy to re-run evaluations on new submissions of yours in the meantime.

[TheItCrOw]

I see, thanks!

It's not a big problem in the long run, as I'll just submit more models, but I have to figure out how to deal with the rebuttals for now. From my side, this issue can be closed.