Testing on subset of imagenet seems to give less-than-optimal results

[chaosfengz]

Nice work! I am testing the performance on a subset of imagenet (https://tiny-imagenet.herokuapp.com/) using InfoMin_800.pth. I changed the number of clusters to 200. I only tested accuracy on training set. I run the minibatch kmeans for several epochs and so far the accuracy seems to be around 10%. Tho the first couple dozen clusters can get very high accuracy (80~90%). Does this seem reasonable? I would imaging if it produced good results on the whole Imagenet with 1000 classes it should produce similar results on a subset. The only caveat I can think of is that I have smaller number of images (100k) so maybe if will affect the quality of k-means clustering.

[Randl]

Hi! Thanks for reaching out. I agree with you that I'd expect to achieve better results on subset of imagenet (of course it depends on what is the subset). But, if I recall correctly, the image dimensions is 64*64 in tiny imagenet which might be the reason of performance drop.

One possible thing to try is to train linear classifier and compare the performance to "big" ImageNet. If there is large performance drop (i.e., you get less then, say, 60% accuracy) then probably the problem is not in the clustering.

For debugging k-means, first, make sure that you perform PCA and use large enough amount of PCA components. Try to increase batch size and check if overclustering provides any improvements.

I'll try to find some time to take a look on if neither of this helps.