Thank you very much for open-sourcing such an excellent project! While currently attempting to reproduce the experiments, I encountered a question regarding the parameter configuration for the Musique dataset that I would like to ask you about. In the run.sh script, I noticed that the parameter settings for the Musique dataset are defined as follows:
musique
SPACY_MODEL="en_core_web_trf"
EMBEDDING_MODEL="model/all-mpnet-base-v2"
DATASET="musique"
LLM_MODEL="gpt-4o-mini"
MAX_WORKERS=16
MAX_ITERATION=5
PASSAGE_RATIO=2.0
THRESHOLD=0.1
TOP_K_SENTENCE=4
With the default parameter configuration listed above, using the BFS iterative method allows for the successful reproduction of the metrics reported in the original paper. However, when switching to the support matrix-based search method, the experimental results do not seem to reach the corresponding performance benchmarks, exhibiting a noticeable gap.
Could you please clarify whether the Musique dataset requires a specific combination of parameters when utilizing the matrix search method? Should the correct parameter configuration align with that of the 2WikiMultihop (HotpotQA) dataset, or are other customized adjustments necessary?
INFO - Initializing LinearRAG with config: LinearRAGConfig(dataset_name='musique', embedding_model=SentenceTransformer(
(0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
(2): Normalize()
), llm_model=<src.utils.LLM_Model object at 0x00000193C9272310>, chunk_token_size=1000, chunk_overlap_token_size=100, spacy_model='en_core_web_trf', working_dir='./import', batch_size=128, max_workers=8, retrieval_top_k=5, max_iterations=5, top_k_sentence=4, passage_ratio=2.0, passage_node_weight=0.05, damping=0.5, iteration_threshold=0.1, use_vectorized_retrieval=True, enable_hybrid_attribute_fallback=False, attribute_keyword_boost=0.25, attribute_query_keywords=['born', 'birth', 'where', 'when', 'located', 'location', 'founded', 'founder', 'died', 'death', 'nationality', 'capital', 'date', 'year'])
2026-06-02 17:31:27,434 - INFO - Using retrieval method: Vectorized Matrix-based
2026-06-02 17:31:27,435 - INFO - Using device: cuda for vectorized retrieval
[passage] Loaded 1354 records from ./import\musique\passage_embedding.parquet
[entity] Loaded 66328 records from ./import\musique\entity_embedding.parquet
[sentence] Loaded 39854 records from ./import\musique\sentence_embedding.parquet
2026-06-02 17:31:37,643 - INFO - Precomputing sparse adjacency matrices for vectorized retrieval...
2026-06-02 17:31:38,239 - INFO - Matrices built: Entity-Sentence torch.Size([66328, 39854]), Sentence-Entity torch.Size([39854, 66328])
2026-06-02 17:31:38,240 - INFO - E2S Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - S2E Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - Device: cuda
Retrieving: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [45:01<00:00, 5.40s/it]
QA Reading (Parallel): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [03:58<00:00, 2.10it/s]
Evaluating samples: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [02:03<00:00, 4.06sample/s, LLM_Acc=0.270, Contain_Acc=0.236]
2026-06-02 18:22:41,695 - INFO - Evaluation Results:
2026-06-02 18:22:41,695 - INFO - LLM Accuracy: 0.2700 (135.0/500)
2026-06-02 18:22:41,696 - INFO - Contain Accuracy: 0.2360 (118/500)
Thank you very much for open-sourcing such an excellent project! While currently attempting to reproduce the experiments, I encountered a question regarding the parameter configuration for the Musique dataset that I would like to ask you about. In the
run.shscript, I noticed that the parameter settings for the Musique dataset are defined as follows:musique
SPACY_MODEL="en_core_web_trf"
EMBEDDING_MODEL="model/all-mpnet-base-v2"
DATASET="musique"
LLM_MODEL="gpt-4o-mini"
MAX_WORKERS=16
MAX_ITERATION=5
PASSAGE_RATIO=2.0
THRESHOLD=0.1
TOP_K_SENTENCE=4
With the default parameter configuration listed above, using the BFS iterative method allows for the successful reproduction of the metrics reported in the original paper. However, when switching to the support matrix-based search method, the experimental results do not seem to reach the corresponding performance benchmarks, exhibiting a noticeable gap.
Could you please clarify whether the Musique dataset requires a specific combination of parameters when utilizing the matrix search method? Should the correct parameter configuration align with that of the 2WikiMultihop (HotpotQA) dataset, or are other customized adjustments necessary?
INFO - Initializing LinearRAG with config: LinearRAGConfig(dataset_name='musique', embedding_model=SentenceTransformer(
(0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
(2): Normalize()
), llm_model=<src.utils.LLM_Model object at 0x00000193C9272310>, chunk_token_size=1000, chunk_overlap_token_size=100, spacy_model='en_core_web_trf', working_dir='./import', batch_size=128, max_workers=8, retrieval_top_k=5, max_iterations=5, top_k_sentence=4, passage_ratio=2.0, passage_node_weight=0.05, damping=0.5, iteration_threshold=0.1, use_vectorized_retrieval=True, enable_hybrid_attribute_fallback=False, attribute_keyword_boost=0.25, attribute_query_keywords=['born', 'birth', 'where', 'when', 'located', 'location', 'founded', 'founder', 'died', 'death', 'nationality', 'capital', 'date', 'year'])
2026-06-02 17:31:27,434 - INFO - Using retrieval method: Vectorized Matrix-based
2026-06-02 17:31:27,435 - INFO - Using device: cuda for vectorized retrieval
[passage] Loaded 1354 records from ./import\musique\passage_embedding.parquet
[entity] Loaded 66328 records from ./import\musique\entity_embedding.parquet
[sentence] Loaded 39854 records from ./import\musique\sentence_embedding.parquet
2026-06-02 17:31:37,643 - INFO - Precomputing sparse adjacency matrices for vectorized retrieval...
2026-06-02 17:31:38,239 - INFO - Matrices built: Entity-Sentence torch.Size([66328, 39854]), Sentence-Entity torch.Size([39854, 66328])
2026-06-02 17:31:38,240 - INFO - E2S Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - S2E Sparsity: 99.99% (nnz=138728)
2026-06-02 17:31:38,240 - INFO - Device: cuda
Retrieving: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [45:01<00:00, 5.40s/it]
QA Reading (Parallel): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [03:58<00:00, 2.10it/s]
Evaluating samples: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [02:03<00:00, 4.06sample/s, LLM_Acc=0.270, Contain_Acc=0.236]
2026-06-02 18:22:41,695 - INFO - Evaluation Results:
2026-06-02 18:22:41,695 - INFO - LLM Accuracy: 0.2700 (135.0/500)
2026-06-02 18:22:41,696 - INFO - Contain Accuracy: 0.2360 (118/500)