Scalable Semantic Term Retrieval with Transformers

An end-to-end semantic retrieval pipeline benchmarking lightweight bi-encoders against QLoRA-tuned large cross-encoders on the MWTR dataset.

This project evaluates the trade-off between accuracy, latency, and computational cost in modern transformer-based retrieval systems.

---

Problem

Search systems must balance:

• Semantic accuracy
• Latency
• Compute efficiency
• Scalability

Do larger LLMs justify their cost for semantic retrieval tasks?

---

System Architecture

Phase 1 — Fast Retrieval Layer (Bi-Encoders)

Models:

• MPNet-base
• MiniLM
• GTE-small

Training:

• Triplet Loss to reshape embedding space
• Vector similarity search for retrieval

Goal: High-speed semantic candidate generation.

---

Phase 2 — Precision Layer (Cross-Encoders)

Models:

• LLaMA-8B (QLoRA)
• DeepSeek-7B (QLoRA)
• Gemma-7B (QLoRA)

Training:

• 4-bit NF4 quantization
• LoRA adapters (parameter-efficient fine-tuning)
• Regression-based semantic scoring (0–1 scale)

Goal: Fine-grained semantic reranking.

---

Evaluation Metrics

• Spearman Correlation
• Pearson Correlation
• Retrieval Precision
• Embedding Space Analysis

---

Key Insights

• GTE-small delivered the best efficiency-to-performance ratio.
• Large cross-encoders significantly improved fine-grained semantic scoring.
• LLaMA-8B and DeepSeek-7B achieved highest correlation with human judgments.
• Large models function effectively as "semantic teacher models" in multi-stage retrieval systems.

---

Production Insight

A hybrid system is optimal:

• Small bi-encoders for fast retrieval
• Large cross-encoders for selective reranking
• Adaptive routing for latency-aware deployment

---

Future Improvements

• Knowledge distillation from large to small models
• Query-based routing strategy
• REST API deployment
• Interactive semantic search demo