Can .test() be used inside of .fit()?

[khari998]

I was trying to run my testing alongside my validation phase so I could test the model on the entire training dataset. In my DataModule, I am correctly returning the DataLoader in from the test_dataloader method so my test() function is working when I call it, however I keep getting an error whenever I run the test method inside my fit() loop regardless of whether its with the use of a callback or not. For my current callback I have:

class PeriodicTestCallback(L.Callback):
    def __init__(self, eval_frequency):
        super().__init__()
        self.eval_frequency = eval_frequency

    def on_train_epoch_end(self, trainer: L.Trainer, pl_module: L.LightningModule):
        if (trainer.current_epoch + 1) % self.eval_frequency == 0:
            trainer.test(pl_module, datamodule=trainer.datamodule)

and im also using eval_frequency to modulate when my validation round runs with:

trainer = L.Trainer(
                        max_epochs=config["num_training_rounds"],
                        check_val_every_n_epoch=config["eval_frequency"],
                        logger=mlf_logger,
                        accelerator="auto",
                        callbacks=[
                            PeriodicTestCallback(config["eval_frequency"]),
                            checkpoint_callback,
                        ],

And with this, Im able to get my testing loop running, but at the end of testing phase I get:

lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 233, in metrics
    assert self.trainer._results is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

And when I looked in the code, it has something to do with the _results not being set properly on the active loop. But when I move .test() to outside of .fit() I don't run into this error at all. Any idea what could be happening? Or is it just not possible to run .test() inside of .fit() ?

[khari998]

I ended up just using my test data loader as my second validation loader by returning an array of dataloaders in the val_dataloader method for the DataModule. But very curious as to why using the test loop within the .fit method doesn't work