Cannot repreduce the results of the paper in the HumanML3D dataset

[Cocty]

Hello, author. The results of my test on the HumanML3D dataset are as follows:
latest.tar final result:
FID: 0.092, conf. 0.003
Diversity: 9.501, conf. 0.090
TOP1: 0.508, conf. 0.003, TOP2. 0.702, conf. 0.002, TOP3. 0.798, conf. 0.002
Matching: 3.016, conf. 0.009
Multimodality:1.287, conf.0.032
What could be the reason for this? Looking forward to your reply. Thank you.

[cr8br0ze]

@Cocty I have a similar problem in reproducing the results of the paper, do you mind sharing your reproduced RVQ results as well. Thank you.

[Murrol]

Hi, thanks for your interest.
Please provide more details, e.g. training and testing scripts, and the results of rvq and m-transformer.
You can refer to #27 for a similar issue.

[chuhaojin]

@Cocty I have a similar problem in reproducing the results of the paper, do you mind sharing your reproduced RVQ results as well. Thank you.

have same problem....

[Cocty]

@Cocty I have a similar problem in reproducing the results of the paper, do you mind sharing your reproduced RVQ results as well. Thank you.

my rvq results are as followed:
net_best_mm.tar final result, epoch 49
FID: 0.030, conf. 0.000
Diversity: 9.624, conf. 0.081
TOP1: 0.508, conf. 0.003, TOP2. 0.698, conf. 0.002, TOP3. 0.793, conf. 0.002
Matching: 3.026, conf. 0.007
MAE:0.039, conf.0.000
net_best_fid.tar final result, epoch 41
FID: 0.023, conf. 0.000
Diversity: 9.535, conf. 0.078
TOP1: 0.503, conf. 0.003, TOP2. 0.698, conf. 0.002, TOP3. 0.794, conf. 0.002
Matching: 3.027, conf. 0.006
MAE:0.041, conf.0.000
latest.tar final result, epoch 49
FID: 0.030, conf. 0.000
Diversity: 9.525, conf. 0.092
TOP1: 0.508, conf. 0.003, TOP2. 0.699, conf. 0.002, TOP3. 0.794, conf. 0.002
Matching: 3.030, conf. 0.007
MAE:0.039, conf.0.000

[Cocty]

Hi, thanks for your interest. Please provide more details, e.g. training and testing scripts, and the results of rvq and m-transformer. You can refer to #27 for a similar issue.

here is my training script for rvq
python train_vq.py --name rvq_first_time --gpu_id 0 --dataset_name t2m --batch_size 512 --num_quantizers 6 --max_epoch 50 --quantize_dropout_prob 0.2 --gamma 0.05
and here is my training script for mtrans
python train_t2m_transformer.py --name mtrans_first_time --gpu_id 0 --dataset_name t2m --batch_size 64 --vq_name rvq_first_time
and my testing results for mtrans are as followed:
latest.tar final result:
FID: 0.092, conf. 0.003
Diversity: 9.501, conf. 0.090
TOP1: 0.508, conf. 0.003, TOP2. 0.702, conf. 0.002, TOP3. 0.798, conf. 0.002
Matching: 3.016, conf. 0.009
Multimodality:1.287, conf.0.032
my testing results for rvq are as #86 (comment)

[Cocty]

Hi, thanks for your interest. Please provide more details, e.g. training and testing scripts, and the results of rvq and m-transformer. You can refer to #27 for a similar issue.

and i retrained the mtrans and rtrans with your pre-trained rvq and i get the test results as followed:
latest.tar final result:
FID: 0.053, conf. 0.002
Diversity: 9.602, conf. 0.082
TOP1: 0.512, conf. 0.003, TOP2. 0.705, conf. 0.002, TOP3. 0.802, conf. 0.002
Matching: 2.996, conf. 0.007
Multimodality:1.244, conf.0.027
more similar to the result in your paper，is any problem in training rvq?what's the parameter gamma

[Murrol]

Hi, thanks for your interest. Please provide more details, e.g. training and testing scripts, and the results of rvq and m-transformer. You can refer to #27 for a similar issue.

and i retrained the mtrans and rtrans with your pre-trained rvq and i get the test results as followed: latest.tar final result: FID: 0.053, conf. 0.002 Diversity: 9.602, conf. 0.082 TOP1: 0.512, conf. 0.003, TOP2. 0.705, conf. 0.002, TOP3. 0.802, conf. 0.002 Matching: 2.996, conf. 0.007 Multimodality:1.244, conf.0.027 more similar to the result in your paper，is any problem in training rvq?what's the parameter gamma

Hi, it might be because of the AMASS update or package version difference. For the processed dataset cloned from the original HumanML3D project, please send inquiry to [email protected] or [email protected]