ModelCheckpoint in DDP #12598
Unanswered
thomas0809
asked this question in
DDP / multi-GPU / multi-node
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I don't understand how the
ModelCheckpoint
works in DDP. Does it run in all ranks? Can I only run it in rank zero?The reason I want to do this is because my evaluation is done on rank zero only.
ModelCheckpoint
's monitor does not get the metric on other ranks, thus raises an exception. I couldn't find a workaround.Beta Was this translation helpful? Give feedback.
All reactions