Skip to content

Commit 445ab6c

Browse files
temirofftemiroff
committed
Add NVIDIA visual RAG and full graph runs
1 parent 2a30161 commit 445ab6c

19 files changed

Lines changed: 2021 additions & 212 deletions

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ execution, and export the result as code.
5959
|---|---|---|
6060
| Visual workflow editor | Build and inspect typed node graphs with visible execution state. | [Beginner Walkthrough](docs/walkthrough.md) |
6161
| Agent control surface | MCP, HTTP, and WebSocket APIs for agents to create, connect, validate, run, save, inspect, and export workflows. | [Agent Guide](docs/agent-guide.md), [MCP Quickstart](docs/quickstart-mcp.md) |
62-
| NVIDIA workflow surface | Hosted NIM, local NIM launch planning, NIM benchmarks, AI-Q/NeMo Agent Toolkit integration, and streamable HTTP MCP. | [NVIDIA Mission Control](docs/nvidia-mission-control.md), [NVIDIA NIM Demo](docs/nvidia-nim-demo.md), [Blacknode and NVIDIA AI-Q](docs/aiq-integration.md) |
62+
| NVIDIA workflow surface | Hosted/local NIM, Nemotron query rewriting, NeMo Retriever embedding and reranking, visual RAG comparison, benchmarks, AI-Q/NeMo Agent Toolkit integration, and streamable HTTP MCP. | [NVIDIA Mission Control](docs/nvidia-mission-control.md), [NVIDIA Visual RAG](docs/nvidia-visual-rag.md), [Blacknode and NVIDIA AI-Q](docs/aiq-integration.md) |
6363
| GPU/CUDA blocks | Real CUDA kernels, custom NVRTC-compiled kernels, GPU image nodes, and capability/preflight nodes that run on your local NVIDIA GPU. | [NVIDIA GPU Blocks](docs/nvidia-gpu-blocks.md) |
6464
| Local chat agents | Control a local Blacknode workflow through Telegram long polling or Slack Socket Mode, with editor start/stop, live status, tools, images, and replay. | [Local Telegram Agent](docs/telegram-nim-demo.md), [Slack Agent](docs/slack-nim-demo.md) |
6565
| Typed ports and validation | Text, Int, Float, Bool, List, Dict, Embedding, Fn, Model, Number, Any, cycle checks, and MCP repair suggestions. | [Workflow Schema](docs/workflow-schema.md), [Agent Skill](.agents/skills/blacknode-workflow/SKILL.md) |
@@ -116,6 +116,7 @@ See [Blacknode and NVIDIA AI-Q](docs/aiq-integration.md) and
116116
|---|---|
117117
| [NVIDIA NIM Demo](docs/nvidia-nim-demo.md) | Hosted NVIDIA NIM demo path through MCP and the editor. |
118118
| [NVIDIA Mission Control](docs/nvidia-mission-control.md) | NVIDIA nodes, templates, local readiness, local NIM launch, and benchmark flow. |
119+
| [NVIDIA Visual RAG Comparator](docs/nvidia-visual-rag.md) | Compare original and Q2E retrieval with NVIDIA embeddings, reranking, cited answers, and run replay. |
119120
| [Blacknode and NVIDIA AI-Q](docs/aiq-integration.md) | Positioning Blacknode beside AI-Q and using streamable HTTP MCP. |
120121

121122
### Deployment

‎docs/nvidia-mission-control.md‎

Lines changed: 30 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,12 @@ advisory plan, wire the executing nodes (e.g. `NIMAgent`) into the graph — see
2323
| `NIMHealthCheck` | Checks a hosted or local OpenAI-compatible NIM `/models` endpoint. |
2424
| `NIMAgent` | Calls hosted or local NVIDIA NIM through an OpenAI-compatible endpoint. |
2525
| `NIMBenchmark` | Runs repeated NIM calls and returns text, average latency, min/max latency, and raw samples. |
26+
| `NIMQueryRewrite` | Extracts and expands retrieval queries with Nemotron, including Q2E. |
27+
| `NVIDIAEmbedding` | Calls hosted or local NeMo Retriever Embedding NIM in query or passage mode. |
28+
| `NVIDIAVectorSearch` | Runs inspectable cosine search over NVIDIA embeddings. |
29+
| `NVIDIARerank` | Reranks candidate passages with hosted or local NeMo Retriever Reranking NIM. |
30+
| `NIMCitationAnswer` | Generates an evidence-constrained NIM answer with numbered citations. |
31+
| `RetrievalCompare` | Compares original, rewritten, and reranked result orders. |
2632
| `NIMFineTune` / `NIMFineTuneStatus` | Submit and poll a NeMo Customizer fine-tuning job (dry-run by default). |
2733
| `VideoFolderInput` | Builds a local video-folder manifest for video-intelligence workflows. |
2834
| `CUDAKernelLab`, `CUDACustomKernel`, `CUDAImageFilter`, `TensorCoreGEMM`, `CUTLASS` | Run real GPU compute on the local NVIDIA GPU (NVRTC / CuPy / containerized CUTLASS). |
@@ -48,6 +54,7 @@ advisory plan, wire the executing nodes (e.g. `NIMAgent`) into the graph — see
4854
| `nvidia-nim-benchmark` | Live hosted/local NIM benchmark. Requires `NVIDIA_API_KEY` for hosted NIM or a running local endpoint. |
4955
| `nvidia-nim` | Simple prompt-to-NIM workflow through the general `LLMAgent`. |
5056
| `nvidia-nim-mcp-demo` | MCP-oriented demo that opens and cooks a visible NVIDIA NIM workflow. |
57+
| `nvidia-visual-rag-comparator` | Original-query versus Q2E semantic retrieval, NVIDIA reranking, cited generation, and visual comparison. |
5158

5259
## Quick Demo Path
5360

@@ -89,6 +96,12 @@ python -m blacknode.cli run templates\nvidia-nim-benchmark.json
8996
Expected result: the benchmark returns generated text, average latency, min/max
9097
latency, and raw latency samples.
9198

99+
### 4. Compare original and Q2E retrieval
100+
101+
Load `templates/nvidia-visual-rag-comparator.json` in the editor and cook the
102+
Q2E query, retrieval comparison, cited answer, and citations outputs. See
103+
[NVIDIA Visual RAG Comparator](nvidia-visual-rag.md).
104+
92105
## Hosted NIM vs Local NIM
93106

94107
Hosted NIM is the fastest demo path: set `NVIDIA_API_KEY`, keep the endpoint at
@@ -101,41 +114,32 @@ generate the container command, start it in a terminal, then point
101114

102115
## Implemented Surface
103116

104-
Blacknode exposes NVIDIA-oriented workflow nodes, hosted/local NIM calls, local
105-
readiness checks, benchmark metrics, local NIM launch commands, video
106-
intelligence planning, retriever planning, templates, run-history visibility,
107-
and Python export.
117+
Blacknode exposes NVIDIA-oriented workflow nodes, hosted/local NIM calls,
118+
embedding and reranking adapters, visual RAG comparison, local readiness
119+
checks, benchmark metrics, local NIM launch commands, video intelligence
120+
planning, retriever planning, templates, run-history visibility, and Python
121+
export.
108122

109-
## What we still need to build to actually run the pipeline
123+
## What remains for the video pipeline
110124

111-
The advisory nodes describe a real NVIDIA pipeline; to make the Video
112-
Intelligence Mission Control template *execute* instead of plan, each advisory
113-
stage needs a matching **executing** node. All of these call documented,
114-
OpenAI/HTTP-compatible NVIDIA services, so they fit the existing provider/HTTP
115-
pattern (`NIMAgent` is the reference). They are listed in dependency order.
125+
The generic text retrieval path now has executing embedding, vector search,
126+
reranking, query rewrite, and cited-answer nodes. To make the separate Video
127+
Intelligence Mission Control template execute instead of plan, the remaining
128+
video-specific stages need matching executing nodes.
116129

117130
| New executing node | Replaces (advisory) | What it must do | NVIDIA service / endpoint | Inputs → Outputs |
118131
|---|---|---|---|---|
119132
| `VideoFrameSampler` | (new, feeds the chain) | Decode each file from the `VideoFolderInput` manifest and sample frames/clips at an interval. Pure local (ffmpeg/PyAV) — no service. | local ffmpeg | `manifest:Dict` → `frames:List` (path + timestamp) |
120133
| `VLMCaptionNIM` | `NVIDIAVideoSummaryPlan` | Send sampled frames to a vision-language NIM and get a caption/events per timestamp. | VLM NIM (e.g. `nvidia/vila`, `microsoft/phi-3.5-vision`) on `integrate.api.nvidia.com/v1` (OpenAI vision messages) | `frames:List`, `model:Model` → `segments:List` (file, ts, caption) |
121134
| `CosmosEmbedVideo` (optional, alt path) | `NVIDIAVideoSummaryPlan` | Produce video/world-model embeddings for segments via Cosmos. | Cosmos embedding NIM | `frames:List` → `embeddings:List` |
122-
| `NeMoEmbedText` | part of `NVIDIARetrieverIndexPlan` | Embed segment captions with a NeMo Retriever embedding NIM. | embedding NIM (`nvidia/nv-embedqa-e5-v5`) `/embeddings` | `segments:List` → `vectors:List` |
123-
| `VectorIndexUpsert` / `VectorSearch` | part of `NVIDIARetrieverIndexPlan` | Store vectors + metadata (file, ts) and do top-k recall for a query. Start with a local index (FAISS/sqlite-vss); swap a managed store later. | local FAISS, or NeMo Retriever index | `vectors:List` / `query` → `hits:List` |
124-
| `NeMoRerankNIM` | part of `NVIDIARetrieverIndexPlan` | Rerank recalled segments against the query for answer quality. | reranking NIM (`nvidia/nv-rerankqa-mistral-4b-v3`) `/ranking` | `query`, `hits:List` → `ranked:List` |
125-
| `NIMVideoQA` | `NVIDIAQuestionAnswerPlan` | Take the reranked evidence + question and **actually call** NIM/Nemotron, citing file + timestamp. This is `NIMAgent` with an evidence-formatting wrapper — the smallest real win. | NIM/Nemotron on `integrate.api.nvidia.com/v1` | `question`, `ranked:List`, `model:Model` → `answer:Text`, `citations:List` |
135+
| `PersistentVectorIndex` | part of `NVIDIARetrieverIndexPlan` | Store vectors + metadata (file, timestamp) in a durable vector database. The current `NVIDIAVectorSearch` is in-process and intended for inspectable demos. | Milvus, LanceDB, or another NeMo Retriever-compatible store | `vectors:List` / `query` → `hits:List` |
136+
| `NIMVideoQA` | `NVIDIAQuestionAnswerPlan` | Adapt `NIMCitationAnswer` to cite file names and timestamps from real video segments. | NIM/Nemotron on `integrate.api.nvidia.com/v1` | `question`, `ranked:List`, `model:Model` → `answer:Text`, `citations:List` |
126137
| `NVIDIAExecReport` | `NVIDIAMissionReport` | Same report, but built from **executed** results (real captions, real hits, real answer) instead of plan text. | none | executed outputs → `report:Text` |
127138

128139
Notes:
129140

130-
- **Smallest first step:** `NIMVideoQA` (or just dropping a `NIMAgent` after a
131-
`RAGContext`/`KeywordSearch` over the captions) turns the demo from "a plan"
132-
into "a real answer" using infra that already exists. `VLMCaptionNIM` is the
133-
next-highest-value node because it makes the *video* real.
134-
- **Reuse what exists:** retrieval can start with the current `KeywordIndex` /
135-
`KeywordSearch` / `RAGContext` nodes (already real) before investing in
136-
`NeMoEmbedText` + `NeMoRerankNIM`. The advisory `NVIDIARetrieverIndexPlan`
137-
becomes a thin wrapper or is dropped.
138-
- **Credentials/decisions needed before building:** which VLM + embedding +
139-
rerank NIM models to target, hosted vs local for each, and whether to use a
140-
managed vector store or a bundled local index. These are product calls, not
141-
code calls — confirm them before implementing the table above.
141+
- **Reuse what exists:** `NVIDIAEmbedding`, `NVIDIAVectorSearch`,
142+
`NVIDIARerank`, and `NIMCitationAnswer` can operate on caption segments once
143+
`VLMCaptionNIM` produces them.
144+
- **Credentials/decisions still needed:** which VLM to target and whether the
145+
production index should use Milvus, LanceDB, or another managed store.

‎docs/nvidia-visual-rag.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# NVIDIA Visual RAG Comparator
2+
3+
Blacknode can compose NVIDIA's modular retrieval components as a visible,
4+
typed workflow. The included comparator runs the same semantic retrieval path
5+
twice: once with the user's original question and once after Nemotron applies
6+
Query-to-Expansion (Q2E).
7+
8+
```text
9+
original query -> query embedding -> vector search
10+
documents -> chunking -> passage embeddings
11+
Q2E query -> query embedding -> vector search
12+
|
13+
v
14+
NVIDIA reranking
15+
|
16+
v
17+
cited NIM answer
18+
```
19+
20+
The graph exposes the rewritten query, retrieval orders, reranked passages,
21+
citations, model calls, and node timings. This makes it possible to inspect
22+
whether query rewriting improved retrieval instead of treating RAG as one
23+
opaque call.
24+
25+
## Included Nodes
26+
27+
| Node | Purpose |
28+
|---|---|
29+
| `NIMQueryRewrite` | Uses a Nemotron chat NIM to extract a core query and optionally apply Q2E. |
30+
| `NVIDIAEmbedding` | Calls a hosted or local NVIDIA Embedding NIM in `query` or `passage` mode. |
31+
| `NVIDIAVectorSearch` | Runs deterministic cosine similarity over returned embeddings. |
32+
| `NVIDIARerank` | Sends retrieved candidates to a hosted or local NVIDIA Reranking NIM. |
33+
| `NIMCitationAnswer` | Generates an evidence-constrained answer with numbered citations. |
34+
| `RetrievalCompare` | Reports changes between original, rewritten, and reranked result orders. |
35+
36+
The default retrieval models are:
37+
38+
- `nvidia/llama-nemotron-embed-1b-v2`
39+
- `nvidia/llama-nemotron-rerank-1b-v2`
40+
41+
Query rewriting and cited generation default to
42+
`nvidia/nemotron-mini-4b-instruct` for responsive interactive demos. The model
43+
field remains configurable for larger hosted or local Nemotron models.
44+
45+
## Run The Template
46+
47+
You can set a hosted NVIDIA API key through the environment:
48+
49+
```powershell
50+
$env:NVIDIA_API_KEY="nvapi-..."
51+
```
52+
53+
Open the editor:
54+
55+
```powershell
56+
start.bat
57+
```
58+
59+
Alternatively, select any NVIDIA node and enter the key once in the
60+
**Shared Credential · NVIDIA NIM** card. Blacknode saves it in the editor key
61+
store and automatically reuses it for every NVIDIA service node. The canvas
62+
shows **NIM key shared** when the key is available and **NIM key missing** when
63+
it still needs to be configured. The `api_key` input port remains available
64+
for workflows that need an explicit per-node override.
65+
66+
Load **NVIDIA Visual RAG Comparator** from Templates and cook:
67+
68+
1. **Q2E Rewritten Query**
69+
2. **Retrieval Comparison**
70+
3. **Cited Answer**
71+
4. **Citations**
72+
73+
The default corpus is intentionally small. Replace the `corpus` Text node with
74+
a file, database, or extraction workflow for a real use case.
75+
76+
## Manual Credential Check
77+
78+
1. Open **NVIDIA Visual RAG Comparator** and select an NVIDIA node.
79+
2. Confirm that its inspector says **Key found** and the canvas badge says
80+
**NIM key shared**.
81+
3. If the key is missing, enter it once in the shared credential card.
82+
4. Select another NVIDIA node. It should immediately show the same shared-key
83+
status without asking for another key.
84+
5. Cook **Retrieval Comparison** or **Cited Answer** to verify the shared
85+
credential works in a live NVIDIA request.
86+
87+
## Hosted And Local Endpoints
88+
89+
Hosted defaults:
90+
91+
```text
92+
Embedding: https://integrate.api.nvidia.com/v1/embeddings
93+
Reranking: https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-nemotron-rerank-1b-v2/reranking
94+
Chat NIM: https://integrate.api.nvidia.com/v1
95+
```
96+
97+
For local NIM containers:
98+
99+
- Set `NVIDIAEmbedding.endpoint_url` to a service base such as
100+
`http://127.0.0.1:8000/v1`; Blacknode appends `/embeddings`.
101+
- Set `NVIDIARerank.endpoint_url` to a service base such as
102+
`http://127.0.0.1:8001/v1`; Blacknode appends `/ranking`.
103+
- Point `NIMQueryRewrite` and `NIMCitationAnswer` at the OpenAI-compatible chat
104+
NIM base URL.
105+
106+
Embedding and reranking commonly run as separate NIM services, so they can use
107+
different ports or hosts.
108+
109+
## Current Boundary
110+
111+
This first implementation keeps document embeddings in workflow values and
112+
performs cosine search in the Blacknode process. That makes the comparison
113+
portable and inspectable, but it is not a production vector database.
114+
115+
For a larger deployment, keep the same rewrite, rerank, citation, and comparison
116+
nodes while replacing `NVIDIAVectorSearch` with Milvus, LanceDB, or another
117+
vector store backed by the full NeMo Retriever ingestion pipeline.
118+
119+
Q2D, chain-of-thought rewrite comparison, automatic groundedness scoring,
120+
persistent indexes, and benchmark datasets are intentionally deferred until
121+
the basic Q2E experiment has real evaluation data.
122+
123+
## NVIDIA References
124+
125+
- [NeMo Retriever overview](https://docs.nvidia.com/nemo/retriever/latest/)
126+
- [Embedding NIM API](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/)
127+
- [Reranking NIM API](https://docs.nvidia.com/nim/nemo-retriever/text-reranking/latest/)
128+
- [Reasoning query rewriting with Nemotron](https://developer.nvidia.com/blog/how-to-enhance-rag-pipelines-with-reasoning-using-nvidia-llama-nemotron-models/)

0 commit comments

Comments
 (0)