Reproduce training process issue

[Yuuuuuuuuuuuuuuuuuummy]

Hi, thanks for the excellent work and open sourced.

Actually I have some issue about reproducing training process.

I tried to train drivoR on navsim and evaluate on Navhard using EPDMS

I found that using the default setting and trained 10 epochs was still lower than the score reported in paper, which is 48.3, I can only get the 44.62(bs=8x8).

I tried to extend the training epochs, it got 46.38 on the 11th epoch and dropped to 44 and 45 on the next several epochs.

Do you have any suggestions on this?

BTW, from the Simscale exp, you increased the training epochs from 10 to 30, it seems increased a lot, could you please provide some suggestions on parameter setting for larger scale data?

Thanks! Any help would be appreciated!

[yihongXU]

Hello,

Indeed, for Nav2, we did observe some variance during training only on Navtrain, depending on the devices, due to the PDM calculation with cpus. But the variance is greatly reduced when you scale with SimScale data.

It's common when the data increase, we need to increase the training epochs for convergence, and we observed that 30 epochs are enough for training a subset of SimScale data. The model performance is much less sensitive to hyperparameters when you scale the model with simScale data.

Btw, 48.3 is the Nav2 performance after the fix of bugs in human errors filtering: autonomousvision/navsim#151 (comment)

best regards,

[ucwxb]

Hi @yihongXU ,

Thanks for your previous response!

After further investigation, we have ruled out evaluation issues because evaluating your released checkpoint yields the exact 48.3 score. However, our training (fully aligned with your provided configurations) still plateaus around 47.6.

We suspect the gap might stem from library versioning and low-level optimizations. We noticed a few discrepancies:

1. Version Mismatch: The requirements.txt specifies pytorch-lightning==2.4.0, but the metadata in your released checkpoint suggests it was produced using version 2.5.4 (which also requires Python 3.9+).

2. Determinism: As you mentioned, the PDM calculations on CPU introduce some variance. We are wondering if the specific versions of PyTorch or Lightning affect the gradient updates or data loading synchronization during the Navtrain process.

Could you please provide some guidance on the following?

1. Did you use pytorch-lightning==2.5.4 for the final paper results? If so, are there other specific dependency versions (e.g., torch, torchvision) we should strictly align with?

2. Given that 47.6 is quite close but still underperforms, do you think this ~0.7 drop is within the "expected variance" you observed on different devices, or should we look into specific CUDA/cuDNN versions?

[yihongXU]

Hi,

I've checked:
pytorch-lightning 2.5.4 pypi_0 pypi

I believe that it's the issue of ray package and the data volume. I suggest to train with more data (from simscale), where we saw the training is more stable.

[ucwxb]

Hi,

I've checked: pytorch-lightning 2.5.4 pypi_0 pypi

I believe that it's the issue of ray package and the data volume. I suggest to train with more data (from simscale), where we saw the training is more stable.

Hi @yihongXU ,
Thanks for your reply.
But I test with the released checkpoint and get 48.3 score. So do you mean the problem stems from the ray package during traning?
And besides the version of pytorch-lightning, I wonder versions of other packages such as numpy and pytorch same as in the repo?

[yihongXU]

Hi,

We use A100 as GPU, and here are the numpy as torch version:

numpy.version
'1.23.4'>>> numpy.version
'1.23.4'

torch.version
'2.2.0+cu121'

[Yuuuuuuuuuuuuuuuuuummy]

Hi,

We use A100 as GPU, and here are the numpy as torch version:

numpy.version
'1.23.4'>>> numpy.version
'1.23.4'

torch.version
'2.2.0+cu121'

Hi,

The numpy version(1.23.4) has bug when generating training caching. From xiaomi-research/recogdrive#10, due to the provided environment using python 3.8, I changed my numpy version to 1.24.4

I would update some of my experiments.

I tried drivoR when setting seed to 2(somehow the provided script has some format issue lead to using default seed 0), then I got 46.9.

I also tried using simscale data and training 30 epochs, then model can achieve 54.6.

For your reference.

[yihongXU]

Hi,

Thanks for the information!