Regarding the question mentioned in the title, I found that if we predict the offset given a noisy anchor, the resulting trajectory may appear very zigzag, even for the top-1 trajectory. Direct trajectory prediction, on the other hand, could produce smoother and more coherent paths since it avoids compounding noise from intermediate steps.