remove every_n_train_steps from ModelCheckpoint

[sichu2023]

Summary

Drop all val_check_interval to Trainer in favor of every_n_train_steps to ModelCheckpoint.

Details

In the current NeMo version, providing every_n_train_steps to ModelCheckpoint and val_check_interval to Trainer will save the first checkpoint before validation. This leads to incorrect val_loss logging in the first checkpoint name.

Also in favor of val_loss as monitored metric over reduced_train_loss.

[sichu2023]

Dropped since it is only compatible to newer NeMo version.

ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[LearningRateCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[GlobalStepStateCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[ConsumedSamplesCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[OptimizerStateCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[TrainInputCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[TrainOutputCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency[TrainLossCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_train_val_init_consumed_samples - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency_with_uneven_validation_sizes[ValidInputCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency_with_uneven_validation_sizes[ValidOutputCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.
ERROR sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_stop_and_go.py::TestESM2StopAndGo::test_stop_and_go_consistency_with_uneven_validation_sizes[ValidLossCallback] - nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/tmp/tmpuyrh4jvz/TestESM2StopAndGo/checkpoints. Cannot resume.