Scaling validation to large datasets

[jn2clark]

Hi, I had a quick question about the validation. I seem to be running into memory issues with my validation set. After checking the code this seems to be expected (https://github.com/mlfoundations/open_clip/blob/main/src/training/train.py#L251). Just wondering if any work has been done here before I try and implement something? What would the ideal approach be? I was thinking of just batching the validation data and reporting the metrics from each batch.

[mehdidc]

@jn2clark there is a PR on distributed validation that could help #176, it is still not finished

[jn2clark]

Thanks @mehdidc ! I will take a look