Multiple Positive Samples in one batch

[beitong95]

Hi, I notice you set the batch size to 12 so there is a high chance to have multiple positive samples in one batch. It seems you did not explicitly handle this problem. Instead, these positive samples are treated as negative samples. Is this design intentional? Maybe it will not harm the accuracy?

[fabawi]

Handling this situation any other way would result in supervised contrastive learning. Yes, it may harm the output, but the class is not known here following the authors' description. This is problematic only for this toy dataset (although it still works in practice), but it shouldn't be an issue for larger datasets and learning other modalities, since every instance should be different from others

[lizeyujack]

I have the same problem. First, the clip text needs to keep each image with a unique corresponding text description. Because (text, image) pairs are one-to-one relationships. If a text message corresponds to more than one image, then there will be cases where some texts are both positive sample and negative sample in the same batch. Therefore, the computation using the contrast learning method will lead to some problems. I think there are potential problems even with very large datasets. Because you can't guarantee that data in the same batch size doesn't come from the same class. The easiest way to use comparative learning, I think, is to write each image with a unique hard prompt. In that way you can put your data directly into the code for tuning. Otherwise it may not be properly to use infoNCE loss directly to calculate the loss. That's my opinion, not sure if it's right.
Zeyu Li

[fabawi]

@lizeyujack @beitong95 I added support for multi-label masking. Essentially, all occurrences of the same class are set to infinitesimally low values so that they are not considered as negatives. I haven't fully tested it, but you can try it out on #7982038f3ddaf378066b53c05689d431ef5baccc by passing --class_masking as an argument to train.py

[lizeyujack]

@lizeyujack @beitong95 I added support for multi-label masking. Essentially, all occurrences of the same class are set to infinitesimally low values so that they are not considered as negatives. I haven't fully tested it, but you can try it out on #7982038f3ddaf378066b53c05689d431ef5baccc by passing --class_masking as an argument to train.py

I am glad to receive your reply! I think it would be a smart approach to add masks to take the multi-label similarity out of the equation. This would ensure that we continue to train using the contrast learning approach, while still perfectly solving the tricky case of the missing unique HARD prompt for each image. Thank you very much for your work.
Zeyu Li