Cobb County Building & Fire Code Agentic RAG Assistant

Agentic RAG web app that answers Cobb County, Georgia building and fire code questions using local PDF documents first, with web search fallback when local evidence is insufficient or current-code verification is needed.

Core focus: Retrieval-Augmented Generation, agentic tool use, document indexing, source-grounded answers, and deployable ML application engineering.

---

Project Highlights

• Task: Agentic Retrieval-Augmented Generation (RAG)
• Domain: Local government building, permitting, and fire code information
• Objective: Help users query complex Cobb County code documents through a simple chat interface
• Retrieval: Chroma vector search plus local BM25 hybrid retrieval options over local PDFs
• Document processing: Original PyPDF pipeline plus optional Docling-enhanced parsing
• Agent behavior: Lightweight LLM router, local retrieval, evidence checks, and web fallback when current information is requested
• LLMs: OpenAI by default, optional Google Gemini
• App: Streamlit chat UI with source display, a Settings & Eval dashboard, and an explanatory "About the App" tab
• Deployment: Local Python, Docker Compose, and Streamlit Community Cloud compatible

---

What This Project Does

Building and fire code information is often spread across ordinances, county PDFs, checklists, permit forms, and web pages. This app gives users a single chat interface that:

• Loads local Cobb County building and fire code PDFs
• Builds two searchable Chroma collections from the same local PDF corpus
• Preserves the original PDF extraction pipeline as Option 1: PyPDF + Chromadb
• Adds layout-aware PDF parsing as Option 2: Docling + Chromadb
• Adds hybrid keyword + vector retrieval as Option 3: Docling + Chroma + BM25 Hybrid Search
• Adds LLM query expansion before hybrid retrieval as Option 4: Docling + Chroma + Query Expansion + BM25 Hybrid Search
• Displays persisted LangSmith evaluation metrics for each retrieval backend
• Retrieves relevant document excerpts for each user question
• Uses a lightweight LLM router to detect whether web search may be needed
• Checks whether local evidence is strong enough
• Uses web search when local retrieval is weak or when current code status needs verification
• Produces concise answers with source references

If the app cannot find reliable evidence, it responds clearly:

I could not find a reliable answer in the available documents or web sources.

---

Real-World Impact

This project demonstrates how RAG can reduce friction in document-heavy public-sector workflows.

Potential use cases include:

• Helping homeowners understand where to start with permit questions
• Supporting contractors who need to quickly locate relevant county guidance
• Assisting plan reviewers or administrative staff with document lookup
• Creating a searchable knowledge layer over local ordinances and PDF forms
• Demonstrating how AI systems can provide grounded answers instead of unsupported guesses

This is a portfolio demonstration, not an official Cobb County tool.

---

Dataset Description

The local corpus is designed for Cobb County, Georgia building and fire code research.

Expected document types:

• Cobb County Code of Ordinances PDFs
• Cobb County Fire Marshal forms and checklists
• Building permit and tenant build-out guidance
• Fire inspection, hydrant, sprinkler, and emergency equipment documents
• Georgia state fire safety and construction code reference materials
• Current adopted code references from county or state sources

Current local development corpus:

Dataset Item	Value
Local PDF files used during development	41
Approximate raw PDF size	60 MB
Loaded PDF pages	4,093+
Generated vector chunks	13,844
Public repo data policy	Raw PDFs excluded, vectorstore tracked with Git LFS

The data/README.md file explains where users should place their own PDFs before rebuilding the vector index. The generated Chroma vectorstore/ is tracked with Git LFS so Streamlit Community Cloud can load the demo index without private raw PDFs.

The app supports five retrieval backends:

Backend	UI Label	Retrieval Strategy
cobb_code_docs_original	Option 1: PyPDF + Chromadb	Original PyPDF/LangChain page extraction with Chroma vector search
cobb_code_docs_docling	Option 2: Docling + Chromadb	Docling layout-aware Markdown conversion with Chroma vector search
docling_chroma_bm25_hybrid	Option 3: Docling + Chroma + BM25 Hybrid Search	Docling chunks searched with Chroma vector retrieval and a persisted local BM25 keyword corpus
docling_chroma_bm25_expansion	Option 4: Docling + Chroma + Query Expansion + BM25 Hybrid Search	Reuses the Option 3 Chroma + BM25 indexes, expands the original question into five retrieval queries, and fuses deduplicated hybrid results
rag_anything_lightrag_option5	Option 5: RAG-Anything + LightRAG KG Search	Uses RAG-Anything/Docling to process document files into an isolated LightRAG knowledge graph with heading/table relations, rewrites the question into likely zoning/code terminology, then queries LightRAG in mixed mode

Option 1 preserves the original app behavior. Option 2 may improve retrieval quality for layout-heavy PDFs, tables, headings, sections, and regulatory documents. Option 3 adds local hybrid retrieval so keyword-heavy regulatory language and semantic similarity can both contribute to ranking without requiring a separate search server. Option 4 adds one query-expansion LLM call before local hybrid retrieval, which can improve recall for underspecified or vocabulary-sensitive code questions at the cost of additional latency. Option 5 is isolated from all existing Chroma and BM25 assets and is intended for document-aware knowledge graph retrieval over PDFs, HTML, markdown, text, Office documents, tables, checklists, headings, forms, and procedural material.

Retrieval depth is controlled by RETRIEVER_K in .env for Options 1-4 and by LIGHTRAG_TOP_K / LIGHTRAG_CHUNK_TOP_K for Option 5. The default value is RETRIEVER_K=10, so Options 1-4 pass the top 10 final chunks into answer generation by default. Option 3 retrieves a larger BM25 candidate pool internally with max(RETRIEVER_K * 4, 20), retrieves dense candidates from Chroma, then keeps the top RETRIEVER_K chunks after Reciprocal Rank Fusion. Option 4 expands the question into five total queries, retrieves up to RETRIEVER_K * 2 chunks per expanded query, deduplicates across those results, applies a second fusion pass, and keeps the final top RETRIEVER_K chunks.

Option 5: RAG-Anything + LightRAG KG Search

Option 5 builds a separate RAG-Anything/LightRAG knowledge graph index. It uses Docling as the parser, keeps table processing enabled, disables OCR/image/equation processing by default, and injects deterministic document-section/table relationships into the graph from Docling Markdown headings and tables. It does not read, delete, rebuild, or overwrite the existing Chroma vectorstore/, BM25 bm25_index/, or context sidecar folders used by Options 1-4.

Default storage:

vectorstore/rag_anything_lightrag_option5/
  lightrag/
  processed/
  option5_manifest.json
  option5_sources.jsonl

Build or update the Option 5 index:

python -m src.ingestion --pipeline rag_anything_lightrag_option5

The Option 5 ingestion command reads supported files from data/, processes them with RAG-Anything, stores LightRAG graph/vector/document-status artifacts under vectorstore/rag_anything_lightrag_option5/, and skips unchanged documents using the local source sidecar when possible. --rebuild is intentionally rejected for Option 5 to avoid destructive index deletion. To rebuild Option 5 from scratch without touching other indexes, manually remove only vectorstore/rag_anything_lightrag_option5/, then rerun the command above.

Enable and tune Option 5 in .env:

OPTION5_ENABLED=true
RAG_ANYTHING_OPTION5_STORAGE_DIR=vectorstore/rag_anything_lightrag_option5
RAG_ANYTHING_OPTION5_OUTPUT_DIR=vectorstore/rag_anything_lightrag_option5/processed
RAG_ANYTHING_PARSER=docling
RAG_ANYTHING_PARSE_METHOD=auto
RAG_ANYTHING_FILE_EXTENSIONS=.pdf,.html,.htm,.md,.txt,.docx,.xlsx
RAG_ANYTHING_ENABLE_IMAGE_PROCESSING=false
RAG_ANYTHING_ENABLE_TABLE_PROCESSING=true
RAG_ANYTHING_ENABLE_EQUATION_PROCESSING=false
RAG_ANYTHING_SECTION_RELATIONS=true
RAG_ANYTHING_MAX_CONCURRENT_FILES=3
RAG_ANYTHING_DOCLING_TABLES=true
RAG_ANYTHING_RETRY_WITHOUT_TABLES=true
RAG_ANYTHING_DOCLING_TABLE_MODE=fast
LIGHTRAG_WORKING_DIR=vectorstore/rag_anything_lightrag_option5/lightrag
LIGHTRAG_MODE=mix
LIGHTRAG_TOP_K=20
LIGHTRAG_CHUNK_TOP_K=30
LIGHTRAG_MAX_ENTITY_TOKENS=6000
LIGHTRAG_MAX_RELATION_TOKENS=8000
LIGHTRAG_MAX_TOTAL_TOKENS=30000
OPTION5_CONTEXT_MAX_CHARS=60000
OPTION5_QUERY_REWRITE_ENABLED=true
OPTION5_QUERY_REWRITE_ALTERNATES=3
LIGHTRAG_EMBEDDING_MODEL=text-embedding-3-small
LIGHTRAG_LLM_MODEL=gpt-4.1-mini
OPTION5_WEB_FALLBACK_ENABLED=true
ALLOWED_WEB_DOMAINS=cobbcounty.gov,www.cobbcounty.org,cobbcountyga.gov,dca.georgia.gov,oci.georgia.gov,rules.sos.ga.gov
ENABLE_PRE_EXTRACTION_CHUNKING=true
TARGET_EXTRACTION_CHUNK_TOKENS=4000
MAX_EXTRACTION_CHUNK_TOKENS=7000
SAVE_OVERSIZED_ITEM_AUDIT=true
OVERSIZED_ITEM_AUDIT_DIR=vectorstore/rag_anything_lightrag_option5/audit/oversized_items

Query Option 5 by selecting Option 5: RAG-Anything + LightRAG KG Search in the Settings & Eval tab. Retrieval uses LightRAG mix mode by default, which combines knowledge graph local/global retrieval with vector retrieval where supported.

Option 5 uses an LLM query rewrite before LightRAG retrieval because graph/entity routing can be sensitive to the vocabulary extracted from the user question. The rewrite preserves the original question and adds three likely zoning/code phrasings for retrieval only; it does not answer the question or add facts. OPTION5_QUERY_REWRITE_ENABLED=true controls this step, and OPTION5_QUERY_REWRITE_ALTERNATES=3 controls the number of alternate phrasings.

Option 5 also uses broader LightRAG retrieval than the other modes because mixed graph/vector retrieval can otherwise over-focus on extracted entities. LIGHTRAG_TOP_K=20 and LIGHTRAG_CHUNK_TOP_K=30 let more graph hits and vector chunks compete before LightRAG reranking. Option 5 keeps its own context budget because mixed-mode retrieval can return a large bundle of entities, relations, and reranked text chunks. LIGHTRAG_MAX_TOTAL_TOKENS=30000 gives LightRAG enough room to include table/code chunks after graph context, and OPTION5_CONTEXT_MAX_CHARS=60000 controls how much of that LightRAG context the app passes into the strict adequacy gate and final answer prompt.

Before LightRAG extraction, Option 5 estimates token counts for generated document/table chunks and splits oversized items into extraction-safe parts. Split chunks keep source metadata such as source file, content type, original item ID, part number, page/caption context, parent document ID, and parent content item ID. Full oversized originals and split parts are saved under the audit folder with a JSONL manifest so truncation events can be reviewed.

Docling table recognition can fail on unusually complex pages. Option 5 tries Docling table structure extraction in fast mode by default and, if a document throws during parsing, retries that document with tables=False so the rest of the corpus can continue processing. Set RAG_ANYTHING_DOCLING_TABLES=false to skip Docling table-structure recognition entirely for a more conservative build.

Web search remains supplemental. SerpAPI queries are scoped to the configured ALLOWED_WEB_DOMAINS, and returned URLs are filtered again before they are passed into answer generation. Keep the allowlist to official Cobb County and Georgia government domains. Web-derived sources are shown as URL sources separately from local Option 5 context.

Validate Option 5 after building:

python -m src.validate_option5 --query "What permits are required for residential construction?"

Implementation note: RAG-Anything is used as the primary ingestion/query wrapper and LightRAG is initialized underneath it in the same isolated working directory. The current adapter requests LightRAG context for the app's existing evidence gate. If an installed RAG-Anything version does not expose every LightRAG QueryParam knob through aquery, the adapter falls back to the supported aquery(question, mode=...) call and documents that behavior in code.

---

Feature Engineering Overview

This project does not train a traditional tabular ML model. Instead, it engineers retrieval features for semantic search.

Key steps:

• PDF parsing: PDFs are indexed through the original PyPDF pipeline and an optional Docling pipeline
• Docling conversion: Layout-aware PDF content is exported to Markdown before chunking
• Large Docling PDFs: PyMuPDF reads internal bookmarks/TOC first, then oversized sections are processed in overlapping page windows
• Text chunking: Pages are split into overlapping chunks to preserve context
• Metadata tracking: File name, source path, parser type, chunk index, and page information when available are retained
• Embeddings: Each chunk is converted into a dense vector representation
• Vector indexing: Chunks and metadata are stored in Chroma for Options 1 and 2; Options 3 and 4 reuse the Docling Chroma collection and a persisted local BM25 corpus
• Hybrid search: Options 3 and 4 combine BM25 keyword matching and Chroma vector search with Reciprocal Rank Fusion
• Query expansion: Option 4 generates four additional technical and step-back queries, retrieves with all five queries, deduplicates results, and fuses rankings before answer generation
• Query routing: A lightweight LLM classifier flags whether the query needs local retrieval, web search, or both
• Evaluation: LangSmith experiment scores persisted per vector store for faithfulness, answer relevance, context precision, and context recall
• Retrieval scoring: User questions are matched against indexed chunks
• Context expansion: Retrieved chunks are deterministically expanded with same-document neighboring chunks before evidence checks
• Reranking: Options 1-4 rerank expanded context blocks with cross-encoder/ms-marco-MiniLM-L6-v2 before the evidence gate; Option 5 reranks inside LightRAG
• Option 5 query rewrite: The original question is preserved and three zoning/code alternate phrasings are added before LightRAG graph/vector retrieval
• Evidence thresholding: Weak retrieval triggers fallback web search
• Strict evidence gate: Before generation, a JSON adequacy checker verifies that the supplied context contains an exact supporting quote for the requested fact

---

Modeling Approach

The app uses an agentic RAG workflow rather than a single prompt-only LLM call.

User question
    |
    v
Streamlit chat interface
    |
    v
Settings selector
    |
    +--> Option 1: cobb_code_docs_original
    |
    +--> Option 2: cobb_code_docs_docling
    |
    +--> Option 3: docling_chroma_bm25_hybrid
    |
    +--> Option 4: docling_chroma_bm25_expansion
    |
    +--> Option 5: rag_anything_lightrag_option5
    |
    v
Lightweight LLM query router
    |
    +--> Flags likely local retrieval, web search, or both
    |
    v
LangChain RAG controller
    |
    +--> Local retriever using selected backend
    |       |
    |       v
    |   Chroma vector DB, Chroma + BM25 hybrid retrieval,
    |   Chroma + BM25 query expansion retrieval,
    |   or RAG-Anything + LightRAG mixed-mode KG retrieval
    |       |
    |       v
    |   Options 1-4: deterministic neighbor expansion
    |   (retrieved chunk plus chunk_index -1 and +1)
    |       |
    |       v
    |   Options 1-4: CrossEncoder reranking
    |   before evidence adequacy checking
    |
    |   Option 5: LightRAG graph/vector context
    |   from the isolated Option 5 working directory
    |   after Option 5-only zoning/code query rewrite
    |   with LightRAG reranking
    |
    +--> Evidence quality check
            |
            +--> Strong local evidence: answer from documents
            |
            +--> Weak or current-code question: use web search
                    |
                    v
                Strict exact-evidence gate
                    |
                    +--> Exact support present: synthesize grounded answer with sources
                    |
                    +--> Exact support absent: conservative abstention response

Agent behavior:

• Uses a lightweight LLM router before retrieval
• Uses the selected Chroma collection or local hybrid retrieval strategy from the Settings & Eval tab
• Still retrieves local documents for Cobb County code questions
• Expands retrieved chunks before adequacy checking so nearby checklist items, table rows, and bullet values remain visible
• Reranks expanded Options 1-4 context with a local CrossEncoder before the adequacy gate
• Uses relevance scoring and a strict JSON evidence adequacy gate
• Requires an exact supporting quote before generation for numeric, code, inspection, permit, and procedural questions
• Keeps deterministic keyword/date routing as a backup
• Forces web verification for current, latest, adopted, or effective-date questions
• Keeps responses to 2-3 short paragraphs
• Shows whether the answer came from local documents, web search, or both
• Avoids legal, engineering, or permitting advice

---

Results and Validation

This project was validated through ingestion, retrieval, and fallback behavior checks.

Test Area	Result	Notes
PDF ingestion	Passed	Loaded Cobb County and Georgia code PDFs
Vector index build	Passed	Indexed 13,844 chunks into Chroma
Docling-enhanced indexing	Added	Builds cobb_code_docs_docling from Docling Markdown output
Retrieval backend switching	Added	Settings & Eval tab switches across PyPDF/Chroma, Docling/Chroma, local BM25 hybrid, and local query-expansion retrieval
LangSmith evaluation cache	Added	Saves per-backend metrics under eval_results/
Local retrieval smoke test	Passed	Retrieved relevant fire inspection sources
LLM query router	Passed	Flags current, dated, and fee-schedule questions for web verification
Web search fallback	Passed	SerpAPI Google Search works from the app environment
Current-date sanity check	Passed	Runtime date context answers simple date questions
Current-code verification	Passed	Forces web search for currently adopted/effective code questions
App syntax check	Passed	python -m compileall app src
Docker support	Included	Dockerfile and docker-compose.yml

Example retrieval test:

Query	Expected Behavior	Observed Behavior
When is a fire inspection required?	Local retrieval	Returns Cobb County fire-related sources
What is today's date?	Runtime/web fallback	Answers using runtime date context
What are the currently adopted construction codes for Cobb County building permits?	Local + web verification	Uses local documents and Cobb County/state web sources

Retrieval Method Performance Comparison

The final LangSmith evaluation results below come from the saved JSON files in eval_results/, using the fixed 50-question golden dataset. Quality metrics use the five-point evaluator scale aggregated across the dataset, where higher is better. Latency is measured in seconds, where lower is better.

Option	Method	Answer Relevance	Context Precision	Context Recall	Faithfulness	Latency: Avg (secs)	Latency: P50	Latency: P99
1	PyPDF + Chromadb	0.695	0.695	0.610	0.940	15.3	10.4	108.8
2	Docling + Chromadb	0.625	0.650	0.615	0.930	13.6	12.2	36.1
3	Docling + Chroma + BM25 Hybrid Search	0.725	0.765	0.720	0.955	15.4	11.4	69.0
4	Docling + Query Expansion + BM25 Hybrid Search	0.730	0.755	0.645	0.965	23.4	21.3	50.2

Major findings:

• Best overall retrieval quality: Option 3 is the strongest balanced configuration. Adding BM25 keyword search to Docling Chroma improves context precision and recall, which matters for code questions containing exact phrases, section references, thresholds, fees, and procedural language.
• Highest faithfulness: Option 4 has the highest faithfulness score at 0.965 and the highest answer relevance at 0.730, but the query expansion step increases median latency substantially and does not improve context recall versus Option 3.
• Best baseline: Option 1 remains a credible PyPDF/Chroma baseline with strong faithfulness at 0.940 and the fastest median latency at 10.4 seconds, but it trails hybrid retrieval on context recall.
• Docling alone is not enough: Option 2 improves document parsing quality but, by itself, does not outperform the PyPDF baseline on answer relevance or context precision. The biggest gain comes when Docling parsing is paired with BM25 hybrid retrieval.
• Recommended default for portfolio demos: Option 3 is the best practical default because it gives the strongest retrieval metrics without the extra query-expansion latency of Option 4.

---

Key Insights

• RAG is a strong fit for code and permitting documents because answers must be source-grounded.
• Local retrieval alone is not enough for "current" or "effective date" questions because codes change.
• Keeping file and page metadata is essential for user trust.
• Layout-aware parsing can improve retrieval when code PDFs contain headings, tables, or multi-column formatting.
• Hybrid retrieval outperformed either vector-only strategy in the final evaluation, especially on context precision and context recall.
• Deterministic context expansion reduces false refusals when a small retrieved chunk lands just before the relevant checklist item, bullet value, or table row.
• A conservative fallback response is safer than forcing an answer from weak or merely related evidence.
• Exact-evidence gating improves faithfulness by refusing numeric, code, inspection, permit, and procedural answers unless the supporting value or requirement appears in the supplied context.
• Streamlit is effective for quickly turning a RAG pipeline into a recruiter-friendly demo app.
• Docker support improves reproducibility for portfolio reviewers and hiring managers.

---

Tech Stack

• Language: Python 3.12
• App Framework: Streamlit
• RAG Framework: LangChain
• Vector Database: Chroma plus local BM25 corpus
• Document Processing: PyPDF/LangChain loaders and Docling
• Evaluation: LangSmith experiments with GPT-5.1 LLM-as-judge evaluators
• Embeddings: OpenAI by default, optional Gemini
• LLM: OpenAI by default, optional Google Gemini
• Web Search: SerpAPI Google Search
• PDF Loading: PyPDF / LangChain document loaders, Docling
• Large File Handling: Git LFS for prebuilt Chroma vectorstore files
• Deployment: Docker, Docker Compose, Streamlit Community Cloud
• Observability: Optional LangSmith tracing

---

Project Structure

.
├── app/
│   └── streamlit_app.py          # Streamlit chat UI and About tab
├── src/
│   ├── agent.py                  # Agentic RAG orchestration
│   ├── config.py                 # Environment and model configuration
│   ├── ingestion.py              # PDF loading, chunking, embedding, Chroma/BM25 indexing
│   ├── retriever.py              # Chroma/BM25 retrieval helpers and source formatting
│   ├── hybrid_store.py           # Local BM25 hybrid search and query expansion retrieval
│   ├── tools.py                  # Local retrieval and web search tools
│   └── check_web_search.py       # Web search diagnostic script
├── data/
│   └── README.md                 # Instructions for local PDF placement
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── .env.example
├── .gitignore
├── .dockerignore
└── README.md

Excluded from GitHub:

• .env
• secrets/
• notes/
• data/raw/
• .venv/
• .tiktoken_cache/

Tracked with Git LFS:

• vectorstore/**
• *.sqlite3
• *.parquet
• chroma*/**
• vectorstore*/**

---

How to Run Locally

1. Clone the repository

Install Git LFS before cloning or before pulling the prebuilt Chroma vectorstore and BM25 corpus files:

git lfs install

git clone https://github.com/helsharif/cobb-county-code-rag-assistant.git
cd cobb-county-code-rag-assistant
git lfs pull

2. Create a Python 3.12 virtual environment

Windows PowerShell:

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1

macOS/Linux:

python3.12 -m venv .venv
source .venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

For local NVIDIA GPU acceleration during Docling ingestion, install the optional CUDA PyTorch wheels after the base requirements:

pip install -r requirements-gpu.txt

Then set:

DOCLING_ACCELERATOR_DEVICE=cuda

Do not use requirements-gpu.txt on Streamlit Community Cloud. The hosted app should use the regular CPU-compatible requirements.txt.

4. Configure environment variables

cp .env.example .env

Fill in your selected provider keys. Keep .env local only and never commit it:

OPEN_API_KEY=<your-openai-key>
GEMINI_API_KEY=<optional-gemini-key>
SERPAPI_API_KEY=<your-serpapi-key>
LLM_PROVIDER=openai
EMBEDDING_PROVIDER=openai
OPENAI_MODEL=gpt-4.1-mini
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
EVAL_JUDGE_MODEL=gpt-5.1

OPENAI_MODEL controls runtime answer generation, routing, and query expansion. EVAL_JUDGE_MODEL controls LangSmith evaluator scoring. Changing either model does not require rebuilding Chroma or BM25 indexes as long as OPENAI_EMBEDDING_MODEL remains unchanged.

Environment handling best practices used in this repo:

• .env is ignored by Git and excluded from Docker build context.
• .env.example is the public template for required and optional settings.
• The Docker image does not copy .env or bake secrets into image layers.
• docker-compose.yml maps only the variables the app expects instead of passing every local environment variable into the container.
• For shared, hosted, or production deployments, use the platform's secret manager instead of committing or baking secrets into Docker images.

5. Add local PDF documents

Place PDFs under:

data/raw/

Example:

data/raw/cobb_county_fire/
data/raw/cobb_municode/
data/raw/applicable_codes/

6. Build the vector indexes

The original collection can still be rebuilt by itself:

python -m src.ingestion --rebuild

Build both Chroma-backed collections:

python -m src.ingestion --rebuild --pipeline both

Build all physical indexes needed by Options 1-4, including the Docling Chroma collection and local BM25 corpus used by Options 3 and 4:

python -m src.ingestion --rebuild --pipeline all

Build the Docling-enhanced collection with explicit CUDA acceleration:

python -m src.ingestion --rebuild --pipeline docling --docling-device cuda

For large PDFs, the Docling pipeline avoids arbitrary splitting when the PDF contains internal bookmarks or a table of contents. PyMuPDF reads the document TOC first, Docling converts logical sections, and only oversized sections are split into overlapping page_range windows. If a PDF has no usable TOC, the fallback is an overlapping fixed-page window so boundary content is less likely to be lost:

DOCLING_MAX_PAGES=250
DOCLING_PAGE_CHUNK_SIZE=30
DOCLING_PAGE_OVERLAP=5

Docling's DoclingDocument.concatenate API can reassemble converted ranges into one structured object. This project keeps each logical section or page window as its own Chroma source document instead, because that preserves clearer section, page_start, and page_end metadata for citations.

Build only the Docling-enhanced collection:

python -m src.ingestion --rebuild --pipeline docling

Build only the local BM25 hybrid backend:

python -m src.ingestion --rebuild --pipeline docling_chroma_bm25_hybrid

Option 4 does not create a separate index. It reuses the Option 3 Docling Chroma collection and BM25 corpus, then adds query expansion at retrieval time:

python -m src.ingestion --rebuild --pipeline docling_chroma_bm25_expansion

Ingestion accepts comma-separated slugs, so you can rebuild multiple selected backends:

python -m src.ingestion --rebuild --pipeline pypdf_chroma,docling_chroma_bm25_hybrid

After rebuilding, inspect deterministic context expansion for known checklist/guide cases:

python -m src.debug_context_expansion --collection-name cobb_code_docs_docling --verbose

Backend names:

• cobb_code_docs_original: original PDF extraction behavior
• cobb_code_docs_docling: Docling-enhanced layout-aware PDF parsing
• docling_chroma_bm25_hybrid: Docling-enhanced Chroma collection plus persisted local BM25 corpus
• docling_chroma_bm25_expansion: Option 4 retrieval mode that reuses the Chroma + BM25 hybrid indexes and adds LLM query expansion
• rag_anything_lightrag_option5: Option 5 retrieval mode that uses the isolated RAG-Anything/LightRAG knowledge graph

Ingestion also writes deterministic context-expansion sidecars under context_store/. These JSONL files store chunk text by doc_id, source, backend/parser, page metadata, and chunk index. At runtime, the app retrieves small chunks for search quality, then expands each retrieved hit with only same-document neighboring chunks: chunk_index - 1, the retrieved chunk, and chunk_index + 1. Expansion preserves raw retrieval priority globally, sorts only within each retrieved group by chunk index, deduplicates by stable metadata, and applies the context budget after retrieval-priority ordering. Options 1-4 then rerank the expanded context blocks with a local SentenceTransformers CrossEncoder before the adequacy gate. Docling modes use Docling-generated chunks only; they do not use PyPDF fallback content or full-page/page-level expansion. This reduces false refusals caused by truncated chunk boundaries without allowing long lower-ranked pages to push higher-ranked evidence out of the adequacy gate.

After pulling this feature, rebuild the desired ingestion pipelines so context_store/ is created alongside vectorstore/ and bm25_index/.

Docling acceleration:

• DOCLING_ACCELERATOR_DEVICE=auto: safe default for local CPU and Streamlit Cloud
• DOCLING_ACCELERATOR_DEVICE=cuda: local NVIDIA GPU acceleration when CUDA-enabled PyTorch is installed
• DOCLING_NUM_THREADS=4: CPU thread count used by Docling
• DOCLING_DO_OCR=false: safer default for born-digital regulatory PDFs
• DOCLING_BATCH_SIZE=1: conservative batch size to reduce memory pressure on large PDFs
• DOCLING_MAX_PAGES=250: PDFs above this size are processed as logical TOC/bookmark sections when possible
• DOCLING_PAGE_CHUNK_SIZE=30: maximum page-window size for oversized sections or PDFs with no usable TOC
• DOCLING_PAGE_OVERLAP=5: overlap between fallback page windows to protect tables and sections near boundaries

Context expansion:

• CONTEXT_EXPANSION_ENABLED=true: enable deterministic expansion after retrieval
• CONTEXT_EXPANSION_MODE=neighbors: neighbor-only expansion; off disables expansion
• CONTEXT_NEIGHBOR_WINDOW=1: immediate previous and next chunks are used; current code intentionally caps expansion to -1/+1
• CONTEXT_MAX_EXPANDED_DOCS=8: cap expanded context blocks
• CONTEXT_MAX_CHARS=18000: cap total local context passed to adequacy checking and generation

Reranking for Options 1-4:

• RERANKER_ENABLED=true: enable local CrossEncoder reranking after context expansion
• RERANKER_MODEL=cross-encoder/ms-marco-MiniLM-L6-v2: use the same model family configured for Option 5 LightRAG reranking
• RERANKER_TOP_N=8: keep the top reranked expanded context blocks
• RERANKER_BATCH_SIZE=16: CrossEncoder scoring batch size
• RERANKER_MIN_SCORE=0.0: optional raw CrossEncoder score floor; 0.0 disables filtering

7. Run the Streamlit app

streamlit run app/streamlit_app.py

Open:

http://localhost:8501

The committed Streamlit config does not pin a port so Streamlit Community Cloud can use its expected 8501 health check. If your local Windows machine reserves 8501, run the app on another port:

streamlit run app/streamlit_app.py --server.port=8502

Use the Settings & Eval tab to switch between Option 1: PyPDF + Chromadb, Option 2: Docling + Chromadb, Option 3: Docling + Chroma + BM25 Hybrid Search, Option 4: Docling + Chroma + Query Expansion + BM25 Hybrid Search, and Option 5: RAG-Anything + LightRAG KG Search. Switching affects new questions without requiring an app restart.

8. Run LangSmith evaluation

The Settings & Eval tab loads saved metrics immediately when available. Evaluations always use the fixed 50-row CSV test set:

eval_testset/cobb_county_testset.csv

The CSV must include:

• question
• ground_truth

When evaluation runs, the app creates or reuses a LangSmith dataset based on the CSV content hash, runs the selected RAG backend against those 50 questions, and retrieves the LangSmith experiment feedback scores for display. Cached dashboard results are stored per vector store:

eval_results/pypdf_chroma_results.json
eval_results/docling_chroma_results.json
eval_results/docling_chroma_bm25_hybrid_results.json
eval_results/docling_chroma_bm25_expansion_results.json
eval_results/rag_anything_lightrag_option5_results.json

The current naming convention is {slug}_results.json, where pypdf_chroma maps to Option 1, docling_chroma maps to Option 2, docling_chroma_bm25_hybrid maps to Option 3, docling_chroma_bm25_expansion maps to Option 4, and rag_anything_lightrag_option5 maps to Option 5. Background status files follow the same slug pattern under eval_status/ as {slug}_status.json.

Metrics shown:

• Faithfulness: Did the model invent facts? It measures if all claims in the answer are supported solely by the retrieved context. A high score means the model did not hallucinate.
• Answer relevance: Did the model answer the right question? It measures how relevant the generated answer is to the user's prompt, penalizing off-topic, incomplete, or redundant answers.
• Context precision: Did the retriever rank relevant items first? It measures the ratio of relevant documents within the top results, assessing the quality and ordering of retrieved information.
• Context recall: Did the retriever find all the necessary facts? It measures whether the retrieved context contains all the necessary information, compared to a "ground truth" answer.
• Latency: Mean, P50, and P99 execution time in seconds across the 50-question evaluation set. For Option 4, this includes the additional query-expansion LLM call.

The four quality metrics use a five-point LangSmith evaluator scale rather than binary pass/fail scoring:

Score	Interpretation
0.00	No meaningful support, irrelevant, or unusable
0.25	Minimal support with major missing or incorrect facts
0.50	Partially correct with important gaps or mixed evidence
0.75	Mostly correct with minor omissions, retrieval noise, or wording issues
1.00	Fully correct, well-supported, and technically precise

The evaluator prompts require step-by-step reasoning before assigning a score. They are intentionally strict with technical building and fire code details such as dimensions, dates, fire ratings, fees, code sections, and procedural requirements, while allowing equivalent semantic phrasing.

Evaluation is never triggered automatically on app launch. Use Run Evaluation Metrics or Re-run Evaluation from the dashboard. The app starts evaluation in a background Python process, writes a lightweight status file under eval_status/, and keeps the Streamlit chat UI responsive while LangSmith runs. On Windows, the app uses pythonw.exe when available so the evaluator does not open a blank console window. The dashboard shows the current phase, question progress, elapsed time, and automatically polls for updated status/results every 20 seconds while an evaluation is running. A Refresh now button is also available as a manual fallback.

---

Reliability & Evaluation

This project uses a fixed golden evaluation set at:

eval_testset/cobb_county_testset.csv

The set contains 50 Cobb County Fire Permit RAG questions with generated ground-truth responses. These examples are used for every evaluation run so all retrieval configurations are compared against the same target behavior while reducing LangSmith evaluation cost.

Golden dataset composition:

Type	Count	Why included
Simple	32	Direct factual recall with exact numbers, dates, thresholds, fees, code sections, and procedural requirements
Reasoning	10	Scenario and multi-rule questions that test whether the answer stays relevant and grounded
Multi-context	8	Cross-document or cross-section synthesis questions that test context recall

In this curated golden evaluation set, the Cobb County Code of Ordinances remains heavily represented because it is the most important corpus document, while the subset also preserves coverage across forms, fire inspection guidance, NFPA-derived requirements, fee schedules, hydrant requirements, high-rise requirements, and apartment requirements. Each retained question includes at least one stressor for the RAG pipeline, such as numeric thresholds, code section references, tiered fee amounts, multi-step logic, or cross-document synthesis.

Evaluation methodology:

• Model diversity: The golden test set was generated with Claude 4.6 Sonnet, while the RAG agent uses a different LLM at runtime. This decoupling reduces self-evaluation bias, where a model can favor its own phrasing, assumptions, or linguistic patterns.
• Information density: Ground-truth answers are intentionally dense, including details such as exact measurements, code section references, tiered fee amounts, and procedural conditions where applicable. This makes the evaluation stricter for faithfulness and context precision because vague or partially grounded answers are less likely to score well.
• Query distribution: The 50 questions are balanced across simple lookup, reasoning, multi-context, procedural, and scenario-based tasks. This reflects realistic fire permit workflows, from direct code lookups to questions that require synthesis across forms, ordinances, fee schedules, fire inspection guidance, and state or county code references.
• LangSmith scoring: The Settings & Eval dashboard runs the selected retrieval backend against the fixed dataset, records the experiment in LangSmith, and displays cached scores for faithfulness, answer relevance, context precision, context recall, and latency.
• Five-point evaluator scale: Quality scores are quantized to 0.00, 0.25, 0.50, 0.75, or 1.00 to reduce LLM judge jitter while still giving partial credit when an answer or retrieval result is mostly correct.

The evaluation is designed to test whether the app retrieves the right evidence and stays grounded. It is not a legal validation of Cobb County requirements.

---

Docker Run

Create a local .env from .env.example before running Docker Compose. Docker Compose reads this file for variable interpolation, but the image itself does not contain secrets.

Build and start the app:

docker compose up --build

Open:

http://localhost:8502

Rebuild both vector indexes inside Docker:

docker compose run --rm cobb-county-rag python -m src.ingestion --rebuild --pipeline both

The Docker Compose stack is now single-service and does not require a separate search database. To build every required physical index inside Docker:

docker compose run --rm cobb-county-rag python -m src.ingestion --rebuild --pipeline all

---

Example Questions

• What permits are required for residential construction in Cobb County?
• What are fire sprinkler requirements for commercial buildings?
• When is a fire inspection required?
• What are the currently adopted construction codes for Cobb County building permits, and when did they take effect?
• What is today's date?

---

Reproducibility and Best Practices

• Modular application code under src/
• Environment variables isolated in .env
• Public .env.example template
• No hardcoded secrets
• Raw PDFs excluded from Git
• Chroma vectorstore tracked with Git LFS for Streamlit Community Cloud deployment
• Rebuildable vector index
• Source metadata retained for citations
• Dockerized local deployment
• Streamlit Community Cloud compatible structure

---

Databricks Implementation

This repository also includes a Databricks implementation under databricks_implementation/. It adapts the local RAG design for a lakehouse workflow: source documents are loaded from a Unity Catalog volume, chunked into managed Delta tables, indexed through Databricks AI Search hybrid retrieval, and served through a Databricks Streamlit app. The Databricks version keeps the same core behavior as the local app where practical, including neighbor context expansion, optional CrossEncoder reranking, source-grounded answer generation, and a fixed 50-question evaluation set.

Key Databricks assets include:

• 01_build_fire_code_chunks.py: builds document chunks from files stored in Unity Catalog volumes.
• 03_databricks_rag_pipeline.py: connects to the Databricks AI Search hybrid index and runs the RAG pipeline.
• 05_evaluate_databricks_rag_updated.py: evaluates the Databricks app with the same 50-question golden test set.
• cobb-county-rag-app/: Streamlit app files for deployment as a Databricks app.

Databricks evaluation results are saved under databricks_implementation/eval_results/. The run below used 50 questions and was recorded in summary.json as databricks-rag-eval-20260627-055203.

Metric	Score
Faithfulness	0.920
Answer relevance	0.880
Context precision	0.875
Context recall	0.855
Average latency	9.69 sec
P50 latency	8.92 sec
P99 latency	24.67 sec

Overall, the Databricks version preserves strong answer grounding and retrieval quality while moving the document and search layers into managed Databricks services. The main tradeoff is latency: the managed app and model-serving path averaged just under 10 seconds per question in this evaluation, with higher tail latency on more complex retrieval or generation cases.

---

Disclaimer

This project is for portfolio demonstration and educational purposes only. It is not legal, engineering, building code, fire code, or permitting advice.

Users should verify all requirements directly with Cobb County, the Georgia Department of Community Affairs, the State Fire Marshal, and the relevant authority having jurisdiction.

---

Author

Husayn El Sharif
Senior Data Scientist / Machine Learning Engineer

---

Portfolio Relevance

This project highlights:

• Applied RAG system design
• Agentic tool orchestration
• Vector search over local documents
• Web fallback for current information
• Source-grounded LLM responses
• Production-minded Streamlit and Docker deployment
• Public-sector AI workflow design