Simple Retrieval Queries

[ustyuzhaninky]

Stage 8: Simple Retrieval Queries (Mother Issue)

Labels: stage-8, backend, retrieval, embeddings, rag, enhancement

---

Feature Overview

Introduce Simple Retrieval Queries to allow users and developers to search a small document collection using vector similarity.
This is the first step toward Retrieval-Augmented Generation (RAG) and enables semantic lookup of text chunks embedded in Stage 7.

This stage adds:

• a lightweight in-memory or file-based vector store
• vector similarity search
• document/chunk indexing
• retrieval endpoints

It sets the foundation for Stage 9 (Document Ingestion + Full RAG Pipeline).

---

Problem Statement

The service can now generate embeddings, but it cannot search embedded content or retrieve relevant text.
Without retrieval:

• No semantic search
• No context injection for long conversations
• No document lookup or reference answering
• No RAG pipeline
• No semantic dataset browsing for students

The system needs a minimal retrieval engine to make embeddings useful.

---

Proposed Solution

Add a Simple Retrieval Engine with:

1. Vector Store

A minimal, local storage mechanism for vectors:

• Option A: in-memory store persisted to JSON (/data/store/)
• Option B: SQLite table with vector columns
• Option C: Flat file of embeddings + metadata (like a simple FAISS replacement)

Start simple (JSON or SQLite), allow future upgrade.

2. Indexing

Allow adding documents or chunks to the index:

• each entry contains:
  • id
  • text or chunk
  • embedding vector
  • metadata (timestamp, tags, filename, etc.)

3. Retrieval Methods

Support k-nearest-neighbors (kNN) using cosine similarity or dot product:

• single query → top-k results
• optional thresholding (exclude low scores)

4. Retrieval API Endpoints

POST /api/v1/retrieval/query # returns top-k similar chunks
POST /api/v1/retrieval/add # add text to vector store
GET /api/v1/retrieval/index # list indexed items
DELETE /api/v1/retrieval/index/{id} # remove entry

5. Future Integration

Prepared for:

• document ingestion
• chunking pipelines
• contextual LLM prompting
• full RAG chains

---

User Personas

• Students building basic semantic search apps
• Developers experimenting with RAG-like workflows
• Admins indexing shared documents
• Future UI providing semantic lookup tools

---

Use Cases

• Search course notes or textbook excerpts
• Return semantically similar messages or documents
• Provide relevant context for LLM inference
• Compare model responses to known chunks
• Build a simple semantic FAQ system

---

Acceptance Criteria

Retrieval Engine

• Minimal VectorStore implemented (JSON or SQLite)
• Supports:
  • add text → embedding → index record
  • query embedding → similarity ranking → top-k
  • delete index entries
• Uses embeddings from EmbeddingEngine
• Cosine similarity implemented efficiently

API

• POST /retrieval/add adds text or chunk
• POST /retrieval/query returns semantic matches
• GET /retrieval/index lists stored items
• DELETE /retrieval/index/{id} removes an entry
• Input validation and limits (text length, k-max, etc.)

Security

• Add/delete requires admin privileges
• Query accessible to authenticated users
• Reject extremely long text additions without pre-chunking

Logging & Metrics

• Log indexing and query operations
• Expose metrics:
  • retrieval_queries_total
  • retrieval_latency_seconds
  • retrieval_index_size

Documentation & Tests

• README updated with retrieval example flows
• Document VectorStore format in docs/retrieval.md
• Unit tests for kNN search & indexing
• Integration tests for retrieval endpoints
• OpenAPI docs updated

---

Additional Context

This milestone enables:

• basic semantic search
• preliminary RAG workflows
• GPT-assisted information lookup
• student experimentation with embeddings & vectors

It prepares the system for:

• Stage 9 (Document ingestion + full RAG pipeline)
• Stage 10 (Hybrid search & advanced scoring)
• conversational agent with retrieval memory