ColPali-based RAG pipeline for PDF

[Sensiel]

This PR introduces an independent, end-to-end RAG pipeline for PDF document, built to showcase the ease-of-use of the ColPali model for embedding and retrieval.

The pipeline demonstrates:

• Easy Integration: A self-contained example of using ColPali for generating dense vector embeddings.
• Efficient Retrieval: Implements a retrieval system that uses ColPali's embeddings for context fetching.

[Copilot]

Pull Request Overview

This PR adds a ColPali-based document retrieval system with three main components: PDF processing, embedding indexing, and query retrieval. The implementation uses ColPali embeddings with Milvus for vector storage and supports API, batch, and single-query retrieval modes.

Key Changes

• Introduces PDF processing pipeline that converts PDFs to embeddings using ColPali models
• Implements Milvus-based indexing system for storing and searching ColPali embeddings
• Adds retrieval functionality with API server, batch processing, and single-query modes

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
src/mmore/colpali/run_retriever.py	Implements query retrieval with multiple modes (API/batch/single)
src/mmore/colpali/run_process.py	Processes PDFs to generate ColPali embeddings and stores them in Parquet
src/mmore/colpali/run_index.py	Indexes embeddings from Parquet into Milvus database
src/mmore/colpali/milvuscolpali.py	Core Milvus operations manager for ColPali embeddings
examples/colpali/config_retrieval.yml	Configuration for retrieval operations
examples/colpali/config_process.yml	Configuration for PDF processing
examples/colpali/config_index.yml	Configuration for indexing operations

Comments suppressed due to low confidence (1)

src/mmore/colpali/run_retriever.py:2

• Import of 'concurrent' is not used.

import concurrent.futures

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 18 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.