feat: enhanced scientific RAG pipeline with cross-encoder reranking and metadata filtering#5
Open
hobyt-aluzar wants to merge 1 commit into
Open
Conversation
…nd metadata filtering
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhanced Scientific RAG Pipeline with Cross-Encoder Reranking
This PR introduces an enhanced scientific RAG pipeline that significantly improves retrieval quality and citation accuracy, enabling fully private, local semantic reranking.
Key Changes:
@xenova/transformerswithXenova/ms-marco-MiniLM-L-6-v2cross-encoder. It fetches an expanded pool of documents from ChromaDB, scores them semantically against the user's query, and returns the top results. This process runs 100% offline.inject-documents.tsto use targeted text separators for scientific sections (e.g., Abstract, Methods, Results). It now extracts the section, page, and dynamically generated fallback titles to build robust metadata.rag-chat.tsto pass the Rerank Score and precise citation keys (e.g.,[document-name:p1:c2]) directly into the LLM context. The prompt has been updated to enforce the usage of these citation keys, guaranteeing traceable and accurate responses.scientific-rag.test.tsto ensure stability of citation keys, title normalization, and section detection.This architecture scales cleanly and keeps all data on the user's local machine, while delivering near state-of-the-art semantic retrieval performance.