Performance issue with data loading

[mfagerlund]

Hi, I'm playing around with MNIST, and the code in TorchSharp that loads the mnist test dataset takes 17.5 seconds - my code from previously load them from the gzipped files instead of the raw binaries and it takes 0.8 seconds, which is... well... less. Iteration time is everything when it comes to these kinds of experiments and 17.5s for MNIST hurts.

I think the issue is that the batches are grouped into pre-packaged batches - which is rife with round-trips and general sadness. I stack all the data into two large arrays of floats (features and labels) and then load that entire array into a tensor. Also, I don't believe the samples are shuffled between the epochs? It think that could hurt performance.

To get batches out of that very large tensor, I have code that looks like below. Obviously, for a larger dataset, this would be problematic because keeping all the data in one tensor would explode the GPU, but that could be worked around. I could make my stuff available, but I'd have to import a bit of my supporting code to do that.

Anyway, the only slow thing that's left is that the batch tensor is created anew every iteration - but since it's the same size (almost) every time, we could load data into the same tensor over and over again, the indicesTensor could be reused also. What's stopping me is that index_select doesn't support the out parameter, though there is such a parameter in the API, but I'm not sure how to add an overload for that. Perhaps someone could help me?

Also something to consider, I used small batches for a test I was running and the performance was horrible. Turns out I had copied a line of code from the MNIST or some other example, that really kicked me in the pants;

GC.Collect()

I spent a long time looking for the issue until I realized that with GC.Collect, my epochs took 2000ms each, without it, they take 950ms. Not sure why, but in the MNIST example it doesn't seem to make any difference even when it's run every epoch.

    public static IEnumerable<Samples> CreateBatches(torch.Tensor features, torch.Tensor labels, int batchSize = 100, bool randomize = true)
    {
        if (features.shape[0] <= batchSize)
        {
            yield return new Samples(features, labels);
            yield break;
        }

        // TODO: index_select doesn't support the out parameter
        var indices = Enumerable.Range(0, (int)features.shape[0]);
        foreach (var batchIndices in indices.Batched(batchSize, randomize))
        {
            using var indicesTensor = torch.tensor(batchIndices, device: features.device);
            var batchFeatures = features.index_select(0, indicesTensor);
            var batchLabels = labels.index_select(0, indicesTensor);
            yield return new Samples(batchFeatures, batchLabels);
        }
    }

[mfagerlund]

For some strange reason, when I run the loader from TorchSharp in my project, it only takes 5 seconds, not 17. Maybe a debug/release issue?