A production-ready Multi-Agent Retrieval-Augmented Generation (RAG) system built with FastAPI, LangChain, and local LLMs. This project implements a complete enterprise-grade RAG pipeline with 4 specialized agents, caching, streaming, memory, and comprehensive evaluation.
- Architecture
- Features
- Tech Stack
- Installation
- Configuration
- Usage
- API Endpoints
- Evaluation
- Project Structure
- Contributing
- License
┌─────────────────────────────┐
│ USER QUERY │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────────────┐
│ GATEWAY (Orchestrator) │
│ ┌─────────────────────────────┐ │
│ │ 🔍 Redis Cache (Check) │ │
│ └─────────────────────────────┘ │
│ ┌─────────────────────────────┐ │
│ │ 💾 Conversation Memory │ │
│ └─────────────────────────────┘ │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────┐
│ RETRIEVER AGENT │
│ (Semantic Search - BGE-M3) │
│ │
│ Vector Database (ChromaDB) │
│ ────────► Top-K Documents │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────┐
│ PLANNER AGENT │
│ (Nemotron 3 Ultra - FREE) │
│ │
│ Analyzes query & docs │
│ ────────► Decide N Docs │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────┐
│ REASONER AGENT │
│ (BGE-M3 + Cross-Encoder) │
│ │
│ ┌─────────────────────────────┐ │
│ │ 1. Filter (keep N docs) │ │
│ │ 2. Deduplicate (semantic) │ │
│ │ 3. Rerank (Cross-Encoder) │ │
│ │ 4. Merge (create context) │ │
│ └─────────────────────────────┘ │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────┐
│ ANSWER AGENT │
│ (Gemma 2 2B - Local) │
│ │
│ Generate response based on │
│ context + query │
└─────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────┐
│ RESPONSE │
│ ┌─────────────────────────────┐ │
│ │ 📄 Standard JSON Response │ │
│ │ 📡 SSE Streaming (Optional) │ │
│ │ ⚡ Cached for future use │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────────┘
| Agent | Model | Responsibility |
|---|---|---|
| Retriever | BAAI/bge-m3 | Semantic search over vector database |
| Planner | Nemotron 3 Ultra (FREE) | Decides optimal number of documents |
| Reasoner | BGE-M3 + Cross-Encoder | Deduplicates, reranks, merges context |
| Answer | Gemma 2 2B (Local) | Generates final Persian/English response |
- ⚡ Redis Cache: Stores query-response pairs for lightning-fast repeated queries (<1s vs 2+ minutes)
- 📡 SSE Streaming: Real-time token-by-token responses via Server-Sent Events
- 🧠 Conversation Memory: Maintains context across multiple turns per session
- 📊 Evaluation Framework: Precision@k, Recall@k, MRR, Hit Rate with 100+ test queries
- 🔍 Cross-Encoder Reranking: Improves retrieval precision with ms-marco-MiniLM-L-6-v2
- 💻 Fully Local: No API costs for Answer Agent (Gemma 2 runs locally)
| Component | Technology |
|---|---|
| Web Framework | FastAPI + Uvicorn |
| LLM Orchestration | LangChain |
| Embedding Model | BAAI/bge-m3 (Sentence-Transformers) |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 |
| Answer Model | google/gemma-2-2b-it (4-bit quantized) |
| Planner Model | nvidia/nemotron-3-ultra-550b-a55b:free (via OpenRouter) |
| Vector Database | ChromaDB |
| Cache | Redis |
| Evaluation | Custom metrics (Precision, Recall, MRR, Hit Rate) |
| Language | Python 3.10+ |
- Python 3.10 or higher
- Redis (for cache - optional, can be disabled)
- At least 8GB RAM (recommended: 16GB)
git clone https://github.com/alrezesi/Multi-Agent-RAG.git
cd Multi-Agent-RAG
Step 2: Create Virtual Environment
bash
# Windows
python -m venv venv
venv\Scripts\activate
# Linux/Mac
python3 -m venv venv
source venv/bin/activate
Step 3: Install Dependencies
bash
pip install -r requirements.txt
Step 4: Set Up Environment Variables
Create a .env file in the project root:
env
# OpenRouter (for Planner Agent - FREE)
OPENROUTER_API_KEY=your_openrouter_api_key
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
PLANNER_MODEL=nvidia/nemotron-3-ultra-550b-a55b:free
# Redis Cache
REDIS_URL=redis://localhost:6381/0
CACHE_TTL=3600
CACHE_ENABLED=True
# Answer Agent (Gemma 2)
ANSWER_MODEL=google/gemma-2-2b-it
DEVICE=cpu
TEMPERATURE=0.3
MAX_TOKENS=512
USE_4BIT=True
# Embedding Models
EMBEDDING_MODEL=BAAI/bge-m3
CROSS_ENCODER_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
# Memory
MAX_HISTORY=10
Step 5: Run Redis (Optional)
bash
# Using Docker
docker run -d --name redis-rag -p 6381:6379 redis:alpine
# Or use existing Redis instance
# Update REDIS_URL in .env accordingly
Step 6: Ingest Sample Documents
bash
python scripts/ingest_data.py
Expected output:
text
✅ Successfully ingested 6 documents.
📊 Total documents in DB: 6
🚀 Usage
Start the Server
bash
python main.py
API Documentation
Once running, access:
Swagger UI: http://localhost:8000/docs
Health Check: http://localhost:8000/health
Example Request
bash
curl -X POST "http://localhost:8000/v1/chat" \
-H "Content-Type: application/json" \
-d '{
"user_id": "test_user",
"session_id": "test_session_1",
"text": "What is Artificial Intelligence?",
"metadata": {}
}'
Streaming Example
bash
curl -X POST "http://localhost:8000/v1/chat/stream" \
-H "Content-Type: application/json" \
-d '{
"user_id": "test_user",
"session_id": "test_session_1",
"text": "Tell me about RAG systems",
"metadata": {}
}'
📡 API Endpoints
Method Endpoint Description
POST /v1/chat Standard chat response
POST /v1/chat/stream Streaming response (SSE)
GET /health System health check
GET / Welcome message
Request Schema
json
{
"user_id": "string", // Required: User identifier
"session_id": "string", // Required: Session identifier
"text": "string", // Required: User query
"metadata": {} // Optional: Additional parameters
}
Response Schema
json
{
"agent_name": "Answer_Gemma2_Local",
"success": true,
"data": "Response text...",
"error": null,
"metadata": {
"source": "gemma_2",
"model": "google/gemma-2-2b-it",
"response_length": 123,
"mode": "local_llm"
}
}
📊 Evaluation
Metrics
Metric Description Score
Recall@5 Proportion of relevant docs found 1.00
MRR Mean Reciprocal Rank 1.00
Hit Rate@5 At least one relevant doc in top-5 1.00
Precision@5 Proportion of relevant docs in top-5 0.80
Run Evaluation
bash
python scripts/evaluate_system.py
Results are saved to data/evaluation_results.json.
Generate Test Dataset
bash
python scripts/generate_test_dataset.py
Creates 100+ test queries in data/test_dataset_100.json.
📁 Project Structure
text
Multi-Agent-RAG/
├── src/
│ ├── agents/
│ │ ├── base.py # Abstract BaseAgent
│ │ ├── retriever_agent.py # Semantic search
│ │ ├── planner_agent.py # Document count decision
│ │ ├── reasoner_agent.py # Dedup, rerank, merge
│ │ └── answer_agent_local.py # Gemma 2 response generation
│ ├── core/
│ │ ├── orchestrator.py # Gateway orchestrator
│ │ └── vector_store.py # ChromaDB manager
│ ├── cache/
│ │ └── redis_client.py # Redis cache client
│ ├── memory/
│ │ └── conversation_memory.py # Conversation history
│ ├── api/
│ │ ├── routes.py # API endpoints
│ │ └── streaming.py # SSE utilities
│ ├── models/
│ │ └── schemas.py # Pydantic models
│ └── evaluation/
│ └── metrics.py # Precision, Recall, MRR, Hit Rate
├── data/
│ ├── chroma_db/ # Vector database
│ ├── test_dataset.json # Evaluation dataset
│ └── evaluation_results.json # Metrics results
├── scripts/
│ ├── ingest_data.py # Populate vector DB
│ ├── generate_test_dataset.py # Generate 100+ queries
│ ├── reduce_to_10.py # Reduce dataset size
│ └── evaluate_system.py # Run evaluation
├── tests/ # Unit tests
├── .env # Environment variables
├── .gitignore # Git ignore file
├── requirements.txt # Dependencies
└── main.py # Application entry point
🧪 Testing
bash
# Run all tests
pytest tests/
# Run with coverage
pytest --cov=src tests/
🤝 Contributing
Contributions are welcome! Please follow these steps:
Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request
📄 License
Distributed under the MIT License. See LICENSE for more information.
🙏 Acknowledgements
BAAI/bge-m3 - Embedding model
Google Gemma 2 - Answer model
NVIDIA Nemotron - Planner model
OpenRouter - Free LLM API
LangChain - LLM orchestration
ChromaDB - Vector database
📧 Contact
Alireza Eskandari - GitHub
Project Link: https://github.com/alrezesi/Multi-Agent-RAG
⭐ If you found this project useful, please consider giving it a star!