Enhanced RAG System with Hybrid Search and Re-ranking

[milkandvodka]

This PR introduces significant improvements to the retrieval-augmented generation (RAG) system by
implementing hybrid search capabilities and cross-encoder re-ranking. The changes enhance search
result relevance while maintaining full backward compatibility with existing code.

Key Changes

1. Enhanced Retrieval System (retrieval.py)

• New AdvancedRetriever class: Implements hybrid search combining vector similarity and keyword
  matching
• Cross-encoder re-ranking: Uses cross-encoder/ms-marco-MiniLM-L-6-v2 for improved result relevance
• Enhanced RetrievedItem dataclass: Added rerank_score and hybrid_score fields with default values
• Backward compatibility: Maintains legacy Retriever class as a wrapper for existing code

2. Improved Vector Search Agent (agents.py)

• Hybrid search integration: Vector search now uses both semantic and keyword matching
• Advanced re-ranking: Results are re-ranked using cross-encoder models for better relevance
• Enhanced result fusion: New scoring algorithm that considers multiple relevance factors
• Intelligent result scoring: Calculates advanced scores based on query term matching, phrase
  matching, and intent-specific scoring

3. Enhanced Assistant Capabilities (agents.py)

• Entity memory: Tracks entities mentioned in conversations for better context
• Research context tracking: Maintains research objectives and conversation threads
• Citation tracking: Records accessed datasets and references
• Pronoun resolution: Improves query understanding by resolving pronouns to relevant entities
• Next-step suggestions: Provides intelligent follow-up suggestions based on conversation context

Technical Improvements

Search Enhancement Features:

• Hybrid Search: Combines vector similarity with keyword matching using configurable alpha
  parameter
• Cross-Encoder Re-ranking: Uses transformer-based models to re-rank top results for relevance
• Advanced Scoring: Implements multi-factor scoring considering term matching, phrase matching, and
  intent alignment

Memory and Context Features:

• Session Memory: Enhanced tracking of conversation state and research context
• Entity Recognition: Automatic extraction of neuroscience-related entities from queries
• Research Thread Tracking: Maintains context across multiple conversation turns

Backward Compatibility

• All existing method signatures remain unchanged
• Legacy Retriever class continues to function as before
• No breaking changes to public APIs
• Existing import statements continue to work without modification

Dependencies

• sentence-transformers dependency is already present in pyproject.toml
• No additional dependencies required for enhanced functionality

Performance Considerations

• Re-ranking may add slight latency but only processes top candidates
• Hybrid search provides better results at minimal computational cost
• Enhanced features are optional and can be configured via environment variables

Testing

• Maintains all existing functionality
• Enhanced result quality through better relevance scoring
• Improved user experience through context awareness and intelligent suggestions

This PR significantly improves the quality of search results and user experience while maintaining
complete compatibility with existing code.