Handling zero batches in calc_loss_loader to avoid Division Errors.

[rmaestre]

Hi folks !
I notice an issue in the calc_loss_loader def when model evaluation batch are empty (but not None); it attempts to divide by zero, resulting in an exception.

I suggest to adding a guard clause into calc_loss_loader to handle this simple scenario.

    if num_batches == 0:
        return 0.0
    return total_loss / num_batches

Hope this help improve the code and learning experience.

[rasbt]

Thanks for raising this, I appreciate the comment!

Before changing the code, may I ask in which contexts you are encountering this issue? I.e., when/why are your data loaders empty?

Instead of the if-statement, I suppose you could also have return total_loss / max(1., num_batches), which would also return 0 in this case due to 0.0/1.

However, thinking about that more, isn't a loss of 0.0 misleading when your data loaders are empty? I.e., a loss of 0.0 implies that the model makes perfect predictions. However, since there is no data loader, we can't say anything about that.

Or, in other words, isn't it preferred if there is a division by zero error, which implies there's something not right with the data loader?

Let me know your thoughts.

[rmaestre]

Thanks Sebastian, for you feedback. Your point about the potential confusion when 0.0 loss is reported is absolutely right. Perhaps, in order to avoid zero divisions for a smother experience, a print info could be displayed and NaN returned when DataLoader is empty.

Ep 1 (Step 000000): Train loss 9.848, Val loss nan

(A validation) DataLoader could be empty if you setup train_ratio = 0.96 for instance; where the remaining data does not meet the minimum token threshold (< max_len) in GPTDatasetV1.

I hope it is helpful.

[rasbt]

Just implemented it here in #124