About the generation results using all default configs of WoW

[Vonderland]

Thank you for your nice code! It is quite novel to consider sequantial relationships for knowledge selection.
I have used your code and pretrained models for WoW seen/unseen inference (under the default setting without changing anyting), and find the results are different from the samples shown in the paper. The metrics are shown as below:
seen
{'accuracy': 0.27355072463768115,
'kl_loss': 0.3048178,
'knowledge_loss': 1.7232132,
'perplexity': 53.37007,
'rouge1': 0.19250293278265435,
'rouge2': 0.06835883860623956,
'rougeL': 0.1739447536411383,
'total_loss': 6.0052814}
unseen
{'accuracy': 0.18561958184599694,
'kl_loss': 0.27510664,
'knowledge_loss': 2.3412037,
'perplexity': 82.69272,
'rouge1': 0.16044149682040593,
'rouge2': 0.04274795745293827,
'rougeL': 0.1446172269936499,
'total_loss': 7.0314417}
It is a little different from the results you provide in readme.

I have also compared the selection and generated results of the examples in the paper, but they are quite different:

31 (seen).
(context) truth ##fully , anything with cheese is the best
(knowledge_gt) [CLS] che ##dda ##r cheese _ _ knowledge _ _ it is the second - most popular cheese in the us ( behind mo ##zza ##rella ) , with an average annual consumption of
(knowledge_pred) [CLS] bread ##ed cut ##let _ _ knowledge _ _ chicken fingers or chicken tender ##s are an american dish prepared by bread ##ing and deep fry ##ing the pe ##ctor ##alis minor muscle
(gt) especially che ##dda ##r cheese ! it ' s the second most popular cheese in the us !
(pred) i love cheese fingers .

and the results from paper are:

knowledge_pred: Pizza is a traditional Italian dish consisting of a yeasted flatbread typically topped with tomato sauce and cheese and baked in an oven. (from “Pizza” Wikipedia page)

pred: i love pizza too ! it ’ s a traditional italian dish consisting of yeasted flatbread typically topped with tomato sauce and cheese

Should I change some config (like beam search size) to get the results closer to the paper?

Looking forward to your reply :)