LPIPS turn slow during training

[amcs-echo-0]

🐛 the append operation in update() cause training turing slow

To Reproduce

Here is a simple demo for reimplementation. When using torchmetrics==1.7.0, it works well, but with 1.8.1 the time per iteration gradually increases. After checking the commit history, I suspect this is caused by the append operation in update() at line 148.

# Ideally attach a minimal code sample to reproduce the decried issue.
# Minimal means having the shortest code but still preserving the bug.

class Loss(nn.Module):
    def __init__(self, dssim_weight, codebook_weight):
        super().__init__()
            
        self.lpips = LearnedPerceptualImagePatchSimilarity(net_type='vgg')

    def forward(self, colors, rec_colors):
        img_lpips = self.lpips(rec_colors, colors)
        return img_lpips

• TorchMetrics version (if build from source, add commit SHA): 1.8.1
• Python & PyTorch Version (e.g., 1.0): ???
• Any other relevant information such as OS (e.g., Linux): ???

Additional context

[murinmat]

I was just debugging my entire training pipeline for an entire day, figuring out why the training is slowing over time despite no overheating or anything, until I ended up with figuring out that using LPIPS from torchmetrics.image.lpip as part of the loss. I can confirm that reverting to version 1.7.0 (from 1.8.2) fixed this. By the way, here is a graph of GPU utilization and temperature during the training I left overnight, which ended up going from 20 minutes first epoch to 2 hours 6th epoch. It's a massive gradual slowdown.