Skip to content

For the Musique dataset, how should the parameters be configured when using vector-based matrix retrieval? #26

Description

@QIQIQI1523

Thank you very much for open-sourcing such an excellent project! While currently attempting to reproduce the experiments, I encountered a question regarding the parameter configuration for the Musique dataset that I would like to ask you about. In the run.sh script, I noticed that the parameter settings for the Musique dataset are defined as follows:
musique
SPACY_MODEL="en_core_web_trf"
EMBEDDING_MODEL="model/all-mpnet-base-v2"
DATASET="musique"
LLM_MODEL="gpt-4o-mini"
MAX_WORKERS=16
MAX_ITERATION=5
PASSAGE_RATIO=2.0
THRESHOLD=0.1
TOP_K_SENTENCE=4
With the default parameter configuration listed above, using the BFS iterative method allows for the successful reproduction of the metrics reported in the original paper. However, when switching to the support matrix-based search method, the experimental results do not seem to reach the corresponding performance benchmarks, exhibiting a noticeable gap.
Could you please clarify whether the Musique dataset requires a specific combination of parameters when utilizing the matrix search method? Should the correct parameter configuration align with that of the 2WikiMultihop (HotpotQA) dataset, or are other customized adjustments necessary?

INFO - Initializing LinearRAG with config: LinearRAGConfig(dataset_name='musique', embedding_model=SentenceTransformer(
(0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
(2): Normalize()
), llm_model=<src.utils.LLM_Model object at 0x00000193C9272310>, chunk_token_size=1000, chunk_overlap_token_size=100, spacy_model='en_core_web_trf', working_dir='./import', batch_size=128, max_workers=8, retrieval_top_k=5, max_iterations=5, top_k_sentence=4, passage_ratio=2.0, passage_node_weight=0.05, damping=0.5, iteration_threshold=0.1, use_vectorized_retrieval=True, enable_hybrid_attribute_fallback=False, attribute_keyword_boost=0.25, attribute_query_keywords=['born', 'birth', 'where', 'when', 'located', 'location', 'founded', 'founder', 'died', 'death', 'nationality', 'capital', 'date', 'year'])
2026-06-02 17:31:27,434 - INFO - Using retrieval method: Vectorized Matrix-based
2026-06-02 17:31:27,435 - INFO - Using device: cuda for vectorized retrieval
[passage] Loaded 1354 records from ./import\musique\passage_embedding.parquet
[entity] Loaded 66328 records from ./import\musique\entity_embedding.parquet
[sentence] Loaded 39854 records from ./import\musique\sentence_embedding.parquet
2026-06-02 17:31:37,643 - INFO - Precomputing sparse adjacency matrices for vectorized retrieval...
2026-06-02 17:31:38,239 - INFO - Matrices built: Entity-Sentence torch.Size([66328, 39854]), Sentence-Entity torch.Size([39854, 66328])
2026-06-02 17:31:38,240 - INFO - E2S Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - S2E Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - Device: cuda
Retrieving: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [45:01<00:00, 5.40s/it]
QA Reading (Parallel): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [03:58<00:00, 2.10it/s]
Evaluating samples: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [02:03<00:00, 4.06sample/s, LLM_Acc=0.270, Contain_Acc=0.236]
2026-06-02 18:22:41,695 - INFO - Evaluation Results:
2026-06-02 18:22:41,695 - INFO - LLM Accuracy: 0.2700 (135.0/500)
2026-06-02 18:22:41,696 - INFO - Contain Accuracy: 0.2360 (118/500)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions