docveil — Document Intelligence Framework

What it does: Analyses any technical document (BRD, RFC, proposal, specification) through a privacy-safe local AI pipeline. Sensitive names — company names, product names, system names — are masked before any AI model sees the document. You get a structured analysis and a full knowledge extract, with all real names restored in the final output.

Why privacy-safe: No text containing your sensitive terms ever leaves your machine. All AI inference runs locally. After the one-time model download, the pipeline runs entirely offline. No document data is ever sent to any cloud service.

Note: This guarantee applies only to the terms you explicitly list in your nebulization rules file. After a run, review the _neb.md output file to confirm all sensitive terms were masked before the AI processed the document.

Before your first run — checklist

Work through this list once before doing anything else.

Python 3.10 or newer installed (python3 --version to check)
Ollama installed and running (see Install Ollama below)
Required models downloaded (see Download models below)
Repository cloned or unzipped to a local folder
Python virtual environment created and activated (see Setup below)
Dependencies installed (pip install -r deploy/requirements.txt)

Prerequisites

Python

Python 3.10 or newer is required. To check your version:

python3 --version

Download Python from python.org if needed.

Install Ollama

Ollama is the local AI runtime that runs the language models on your machine. It exposes a standard API that the pipeline connects to.

Linux / macOS:

curl -fsSL https://ollama.com/install.sh | sh

Windows:

Download and run the installer from ollama.com/download.

After installation, start Ollama:

Linux/macOS: Ollama starts automatically as a background service. If it does not, run ollama serve in a terminal.
Windows: Ollama runs in the system tray after installation. Look for the Ollama icon in the taskbar notification area.

Verify Ollama is running:

curl http://localhost:11434/v1/models

You should see a JSON response. If you see a connection error, Ollama is not running.

Ollama's default port is 11434, which is the port the pipeline expects by default. No configuration change is needed unless you have a port conflict.

Download models

The pipeline uses two language models and one embedding model. Download them with these commands (run once, ~25 GB total free disk space required):

ollama pull qwen3:14b        # knowledge extraction — high accuracy required
ollama pull qwen3:8b         # analysis and reasoning — faster, lower memory
ollama pull nomic-embed-text # semantic indexing (required for --embed; preflight warns if missing)

Hardware requirements:

Model	Minimum RAM	Recommended
`qwen3:8b`	8 GB	16 GB
`qwen3:14b`	16 GB	24 GB
`nomic-embed-text`	2 GB	—

Ollama can run models entirely on CPU if no compatible GPU is available, but inference will be significantly slower (minutes per query instead of seconds). To check whether Ollama detected your GPU, run ollama ps while a model is loading — the PROCESSOR column shows GPU or CPU.

Cloud alternative — OVH AI Endpoints

If your machine does not have enough RAM to run the models locally, you can use OVH AI Endpoints as a cloud backend. OVH AI Endpoints is an OpenAI-compatible inference API — no code changes are needed, only environment variables.

Create an account at endpoints.ai.cloud.ovh.net
Generate an API key in your account settings
Note the endpoint URL and the model names available in your region
Set the environment variables before running the pipeline (see Environment variables):

Linux/macOS:

export OV_BASE_URL="https://<your-ovh-endpoint>/v1"
export OV_MODEL="<model-name-from-ovh>"
python scripts/run_pipeline.py ...

Windows (Command Prompt):

set OV_BASE_URL=https://<your-ovh-endpoint>/v1
set OV_MODEL=<model-name-from-ovh>
python scripts\run_pipeline.py ...

Replace <your-ovh-endpoint> with the endpoint URL from your OVH account dashboard. The URL must end with /v1 (e.g. https://llm.endpoints.kepler.ai.cloud.ovh.net/v1). Replace <model-name-from-ovh> with the exact model identifier shown in your OVH account (e.g. Meta-Llama-3.1-8B-Instruct). When using a cloud endpoint, your nebulized document (with placeholders, not original names) will be sent to OVH servers.

Setup

Create a virtual environment

A virtual environment keeps the pipeline's dependencies isolated from your system. Do this once, then activate it every time you open a new terminal.

Linux / macOS:

cd /path/to/SBSL-A          # navigate to the repository folder
python3 -m venv .venv
source .venv/bin/activate

Windows (Command Prompt):

cd C:\path\to\SBSL-A
python -m venv .venv
.venv\Scripts\activate

Windows (PowerShell):

cd C:\path\to\SBSL-A
python -m venv .venv
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned   # run once if activation fails
.venv\Scripts\Activate.ps1

When the virtual environment is active, your prompt will show (.venv) at the start. You must activate it again each time you open a new terminal.

Install dependencies

With the virtual environment active:

pip install --upgrade pip
pip install -r deploy/requirements.txt

For PDF input support (optional):

pip install "markitdown[pdf]"

All commands in this guide assume you are running from the repository root — the folder that contains scripts/ and deploy/. If you get "file not found" errors, check that you are in the right folder with pwd (Linux/macOS) or cd (Windows).

Quick start — run the example

Before running: confirm Ollama is running and models are downloaded. Quick check: curl http://localhost:11434/v1/models — you should see a JSON list.

The deploy/examples/ folder contains a ready-to-run example: a fictional Business Requirements Document (BRD) for a Customer Self-Service Portal.

Linux / macOS:

python scripts/run_pipeline.py \
  deploy/examples/customer_portal_brd.md \
  deploy/examples/nebulize_rules.yaml \
  deploy/examples/task_prompt.txt

Windows:

python scripts\run_pipeline.py deploy\examples\customer_portal_brd.md deploy\examples\nebulize_rules.yaml deploy\examples\task_prompt.txt

Run this command from the repository root (the folder containing scripts/ and deploy/). Output files will be created in deploy/examples/customer_portal_brd_output/. The full run takes roughly 3–10 minutes depending on hardware.

While running, the pipeline prints a label for each step as it starts, then a dot for every chunk of output received from the model. A long pause with dots appearing is normal — the model is generating. If the terminal is completely silent for more than 2 minutes, see Troubleshooting.

Output files

File	What it contains
`customer_portal_brd_final.md`	Your analysis — real names restored, ready to read
`customer_portal_brd_extract.yaml`	Structured knowledge extracted from the document
`customer_portal_brd_carved.md`	Formatted knowledge report
`customer_portal_brd_analysis.md`	Raw analysis before name restoration
`customer_portal_brd_neb.md`	Document with sensitive terms masked — review this to verify masking
`substitution_table.yaml`	Privacy map — keep this file; needed to restore names

Start with customer_portal_brd_final.md — that is the finished product.

Check document fit before a full run

For a new document, run a quick assessment first to verify the models handle it well. This is faster and cheaper than running the full pipeline on a document that produces poor results.

Linux / macOS:

python scripts/00_preflight.py \
  --doc deploy/examples/customer_portal_brd.md \
  --rules deploy/examples/nebulize_rules.yaml

Windows:

python scripts\00_preflight.py --doc deploy\examples\customer_portal_brd.md --rules deploy\examples\nebulize_rules.yaml

This extracts structure from your document and reports how many actors, requirements, integrations, and open questions were found, plus any quality warnings.

The pipeline also runs preflight automatically on every run. The manual command above is only for checking a document before committing to a full run.

Running on your own document

Step 1 — prepare your document as markdown

If your document is already a .md file, skip this step.

For Word, HTML, PDF, or PowerPoint files, convert first:

Linux / macOS:

python scripts/00_doc_converter.py your_document.docx your_document.md

Windows:

python scripts\00_doc_converter.py your_document.docx your_document.md

Supported input formats: .docx, .pptx, .xlsx, .html, .htm, .pdf, .csv, .xml.

PDF warning: PDF conversion is best-effort. Tables and multi-column layouts often need manual cleanup after conversion.

Step 2 — write your nebulization rules

Create a file named my_rules.yaml that lists every sensitive term to mask. Copy deploy/examples/nebulize_rules.yaml as your starting point.

A complete minimal example:

nebulize_urls: true   # replace all URLs with [URL_N] tokens

terms:
  - term: "Acme Corporation"
    placeholder: "[COMPANY_1]"
    category: company

  - term: "PaymentGateway"
    placeholder: "[SYSTEM_1]"
    category: system

  - term: "AuthService"
    placeholder: "[SYSTEM_2]"
    category: system

Supported categories: system, company, project, technology, tool, team, url

Rules for good nebulization:

Include every company name, product name, and internal system name in the document.
Multiple spellings of the same entity can share one placeholder (e.g. "Stripe" and "Stripe Payments" can both map to [SYSTEM_1]).
You do not need to worry about term order. The engine automatically processes longer terms before shorter ones, so "Stripe Payments" is always replaced before "Stripe" regardless of the order in your file.
Word boundaries are always respected by default. The term "Stripe" will not accidentally match inside a word like "StripeUser". To disable this, add word_boundary: false to a rule.
By default, matching is case-insensitive ("stripe", "Stripe", and "STRIPE" all match). Add case_insensitive: false to match the exact capitalisation only — useful for acronyms like CRM where you only want to mask the uppercase form.
nebulize_urls: true replaces every URL in the document with a [URL_N] token, independent of your term list.

Step 3 — write your task prompt

Create a file named my_task.txt that tells the AI what analysis to perform. You can copy deploy/examples/task_prompt.txt as inspiration, but do not use it as-is — it references requirement and integration IDs specific to the example BRD. Write your own prompt describing what you want the AI to find in your document.

Example prompts:

"List all requirements that have no corresponding integration defined."
"Identify the top 5 delivery risks and suggest mitigations for each."
"Summarise all open questions grouped by which team should own them."

The AI references requirement IDs (like FR-005) and integration IDs (like INT-003) exactly as they appear in the document. You can see what IDs were extracted by looking at the _extract.yaml output file after the first run.

Step 4 — run

Linux / macOS:

python scripts/run_pipeline.py \
  your_document.md \
  my_rules.yaml \
  my_task.txt

Windows:

python scripts\run_pipeline.py your_document.md my_rules.yaml my_task.txt

Output goes to your_document_output/ by default. To set a custom output folder:

Linux / macOS:

python scripts/run_pipeline.py your_document.md my_rules.yaml my_task.txt \
  --out-dir results/my_analysis/

Windows:

python scripts\run_pipeline.py your_document.md my_rules.yaml my_task.txt --out-dir results\my_analysis

Optional: semantic indexing

Add --embed to build a semantic index of the document (useful for later cross-document comparison with the nexus-weaver step):

python scripts/run_pipeline.py your_document.md my_rules.yaml my_task.txt --embed

Environment variables

Override defaults without editing any code. Set variables before the python command.

Variable	Default	Description
`OV_BASE_URL`	`http://localhost:11434/v1`	Inference API base URL
`OV_MODEL_STRIPPER`	`qwen3:14b`	Model for knowledge extraction (step 05)
`OV_MODEL_SWIRL`	`qwen3:8b`	Model for analysis (step 06)
`OV_MODEL_EMBED`	`nomic-embed-text`	Model for semantic indexing (step 03)
`OV_MODEL`	—	Fallback if step-specific variable is not set

Linux / macOS — set for one command only:

OV_MODEL_STRIPPER=qwen3:14b python scripts/run_pipeline.py ...

Linux / macOS — set for the whole terminal session:

export OV_MODEL_STRIPPER=qwen3:14b
python scripts/run_pipeline.py ...

Windows (Command Prompt) — set for the whole terminal session:

set OV_MODEL_STRIPPER=qwen3:14b
python scripts\run_pipeline.py ...

Windows (PowerShell) — set for the whole terminal session:

$env:OV_MODEL_STRIPPER = "qwen3:14b"
python scripts\run_pipeline.py ...

Pipeline steps explained

Step	Name	What it does
00a	Preflight	Checks the inference server is running and models are loaded. Runs automatically at the start of every pipeline run.
00b	Doc converter	(Run manually before the pipeline) Converts non-markdown files to markdown.
01	Standard mincer	Cleans the markdown: removes HTML noise, normalises heading levels, strips navigation links.
02	Meta nebulizer	Applies your nebulization rules — replaces sensitive terms with `[PLACEHOLDER]` tokens and saves the mapping.
03	Memory freezer	(Optional, requires `--embed`) Splits the document into sections and generates a mathematical fingerprint (embedding) for each — enables semantic search and cross-document comparison.
04	Nexus weaver	(Optional, requires `--embed`) Finds sections across documents that are semantically similar.
05	Mental stripper	Sends the masked document to an AI model and extracts structured information: actors, requirements, integrations, constraints, open questions.
06	Brain swirl	Runs your task prompt against the extracted information and produces the analysis.
07	Info carver	Formats the extracted information as a readable markdown report.
08	Meta replanter	Restores all `[PLACEHOLDER]` tokens to their original values in the final output.

Troubleshooting

Preflight fails: "server unreachable"

Ollama is not running. Start it:

Linux/macOS: ollama serve
Windows: start Ollama from the Start menu or system tray

Verify with: curl http://localhost:11434/v1/models

Preflight fails: "model not registered"

The model has not been downloaded yet. Run:

ollama pull qwen3:14b
ollama pull qwen3:8b

Extraction returns empty actors or zero requirements

Two common causes:

The nebulization rules do not cover the sensitive terms, so the model received ambiguous input. Check the _neb.md output — all sensitive names should appear as [PLACEHOLDER] tokens.
The document structure is unusual. Run the preflight assessment (see Check document fit) with both --doc and --rules to get specific hints.

PDF conversion produces garbled output

PDF is inherently lossy for structured text. Manually fix the converted .md file, or request the document in .docx or .html format from the author.

Pipeline is very slow

The models are running on CPU because no compatible GPU is available. This is normal and the pipeline will complete — it will just take longer (10–30 minutes for a large document). Alternatively, use OVH AI Endpoints as a cloud backend (see Cloud alternative).

Step 08 output still contains [PLACEHOLDER] tokens

The substitution_table.yaml file was not found. It must be in the same output directory as the analysis file. This happens if --out-dir was changed between runs. Find substitution_table.yaml in the output directory from your original run and copy it into the current output directory, then re-run step 08 or the full pipeline.

"externally managed environment" error when running pip

Your system Python is protected. Create a virtual environment first:

python3 -m venv .venv && source .venv/bin/activate   # Linux/macOS
python -m venv .venv && .venv\Scripts\activate        # Windows

Then re-run the pip install command.

Directory layout

The full user guide lives here at deploy/README.md. See the root README.md for a project overview.

scripts/
  run_pipeline.py          single-command pipeline runner
  00_preflight.py          check server and models
  00_doc_converter.py      convert non-markdown files to markdown
  01_standard_mincer.py    clean and normalise markdown
  02_meta_nebulizer.py     privacy masking
  03_memory_freezer.py     semantic indexing (optional)
  04_nexus_weaver.py       cross-document comparison (optional)
  05_mental_stripper.py    knowledge extraction
  06_brain_swirl.py        AI analysis
  07_info_carver.py        formatted knowledge report (optional)
  08_meta_replanter.py     restore real names

deploy/
  README.md                this file
  requirements.txt         Python dependencies
  examples/
    customer_portal_brd.md      example input document
    nebulize_rules.yaml         example nebulization rules
    task_prompt.txt             example analysis task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docveil — Document Intelligence Framework

Before your first run — checklist

Prerequisites

Python

Install Ollama

Download models

Cloud alternative — OVH AI Endpoints

Setup

Create a virtual environment

Install dependencies

Quick start — run the example

Output files

Check document fit before a full run

Running on your own document

Step 1 — prepare your document as markdown

Step 2 — write your nebulization rules

Step 3 — write your task prompt

Step 4 — run

Optional: semantic indexing

Environment variables

Pipeline steps explained

Troubleshooting

Directory layout

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

docveil — Document Intelligence Framework

Before your first run — checklist

Prerequisites

Python

Install Ollama

Download models

Cloud alternative — OVH AI Endpoints

Setup

Create a virtual environment

Install dependencies

Quick start — run the example

Output files

Check document fit before a full run

Running on your own document

Step 1 — prepare your document as markdown

Step 2 — write your nebulization rules

Step 3 — write your task prompt

Step 4 — run

Optional: semantic indexing

Environment variables

Pipeline steps explained

Troubleshooting

Directory layout