why bm25 retriever's queires are different with other four retrievers?

[zhiyuanpeng]

Hi,

I find that mss, dpr, mss-dpr, contriever share the same test queries like their correspondingnq-test.jsons have the same queries, however, bm25's test dataset has different queries. Normally, all retrievers should share the same test queries for the same dataset. Could you please update the bm25 test dataset? Thanks.

Best

[DevSinghSachan]

Hi,
Thanks for reaching out. From what I remember is that the test queries for mss, dpr, mss-dpr, and contriever are preprocessed using BERT tokenizer. Some queries may also have [CLS] and [SEP] tokens.
If you can also preprocess BM25 queries with BERT tokenizer, then it will also be in the same format. Let me know if it works well.

[zhiyuanpeng]

Does this mean that even the bm25's text queries looks different with that of other retrievers, but, if I tokenize them into ids by BERT tokenzier, they should be same? Correct me if I am wrong. Thanks.

[DevSinghSachan]

Hi,
The test set queries are same for all the retrievers. As DPR, MSS-DPR etc are BERT -based their inputs need tokenization. In their data files, you will find the tokenized queries.

But for the case of BM25 no tokenization is needed. But for the purposes of matching queries, you can try to tokenize them with the BERT lowercase tokenizer.

[zhiyuanpeng]

Hi,

I run your script to download the retrievers' outputs which are json files. I extract the queries from the json files and find that for the same dataset, all the retrievers expect for the bm25 share the same queries. For nq, bm25 and other retrievers have about 300 different queries. Will you fix the bm25 dataset you shared? Thanks.