How can I optimize it

[DannyHuuuuu]

Your research is really great, so I replaced my private dataset in your project, but the experimental performance is not good. The figure shows one of the more typical and obvious defects, but the predict mask does not predict. How can I optimize it?

[blaz-r]

Hi, I will need a bit more information to help you as best as I can. Did you keep the hyperparemetrs as default? How do the loss and metrics look like? Do all images look visually very similar to this one?

[DannyHuuuuu]

Yes, I have retained all hyperparameters, losses, and metrics as follows. I have additionally provided several figures, as shown.

AP-det, AP-loc, P-AUROC, I-AUROC, AUPRO, seg-AP-det, seg-I-AUROC, category
0.8756066560745239, 0.32398414611816406, 0.9768444299697876, 0.953463613986969, 0.7664461731910706, 0.9303401708602905, 0.9813289642333984, avg

Based on the poor performance of my tests using existing data, I suspected background noise might be the issue. I first used a segmentation model to clean the data by removing the pure black background (excluding the edges, as they might be flawed).
As I had hypothesized, the performance of the cropped data improved, but it still fell short of achieving satisfactory detection results, as shown in the figure. Do you have any suggestions?

[blaz-r]

Good to see that the pre-processing has improved the results to some degree. Given the metrics it seems the model is good at detecting the anomalies but not good at localisng them.
I see two options here:

1. Your images seem to change quite a lot (different position of your object) and the object is quite complext. Therefore I would suggest a small experiment to see if the performance improves when model is trained and tested on a images where the object is always in the same position (i.e., aligned).
2. Usually for the best results it is benefitial to have some labeled defect cases in the train set. So if there is a possiblity of utilising just a few such samples I think that would be quite useful, especially for segmentation precision.

Hope this helps.

[DannyHuuuuu]

Hello, I'm glad to see your reply.
Regarding the first experiment, I don't think it would be very helpful for me. Here's why: even if we verify that both training and testing data are from the same location, improving performance wouldn't really matter.
The second one interests me more. I'd like to ask: your source code currently only uses unlabeled positive samples during training. If we now want to incorporate labeled negative samples into the training process, how significant would the changes to the source code be? Would modifications to the loss function or other components be necessary? (Would unsupervised training transition to supervised training?)
Thank you again for your response.

[blaz-r]

Hi,
besides the dataloading code there shouldn't be any changes required. One of the main benefits of SuperSimpleNet is that it works for many supervision regimes:
unsupervised (only good samples)
weakly supervised (good samples and anomalous with just image level label (i.e., is the object on image defective)
mixed supervision (good samples and anomalous where only some anomalous have pixel level labels but the rest have image level)
fully supervised (good samples and anomalous with pixel level labels)

Here I would note that in paper we refer to anomalous samples as "positive" and not negative.
Also when running the trainign please use the config for either sensum or ksdd2, since those are supervised datasets. That way you will get the best performance.