Gradient accumulation and total number of training steps

[konstantinjdobler]

When using accumulate_grad_batches=K with Lightning and training for a fixed number of steps n (instead of epochs), does Lightning either:

• execute K forward passes and then one optimization step per step, resulting in a total of n*K batches and n optimization steps
• or execute one forward pass per step for a total of n batches with an optimization step only every K steps (therefore only n/K optimization steps in total)?

[rohitgr7]

if you mean max_steps=n, then 1.