Improve scientific RAG evidence and citation grounding

[victortran0904]

Candidate for Algora ISAAC-497 bounty: https://algora.io/isaac/bounties/clq18zr98000ejs0gt0nv7gwu

/claim #45

Summary:

• Add a server-side scientific evidence payload builder that normalizes Chroma results into bounded evidenceContext, stable SRC-* citation keys, citations, and a sourceManifest.
• Store citation-friendly primitive PDF metadata during ingestion, including title, source path, filename, page, chunk index, chunk id, and document id.
• Return the original Chroma query payload plus the evidence/citation contract from fetch-documents with bounded nResults and evidence size.
• Update rag-chat to use same-origin retrieval and instruct responses to cite exact source keys from the provided manifest.

Verification:

• rtk proxy npx vitest run tests/scientific-evidence.test.ts --reporter verbose
• rtk npx tsc --noEmit --pretty false
• rtk npm run lint
• rtk proxy npx prettier --check pages/api/fetch-documents.ts pages/api/inject-documents.ts pages/api/rag-chat.ts utils/server/scientific-evidence.ts tests/scientific-evidence.test.ts
• rtk git diff --check

Notes:

• npm run lint passes with existing React hook dependency warnings in unrelated files.
• This is a focused evidence/citation contract improvement for the existing upload, retrieval, and chat flow.

[Copilot]

Pull request overview

This PR introduces a server-side “scientific evidence” contract for RAG: it normalizes Chroma query results into a bounded evidenceContext, stable SRC-* citation keys, citations, and a sourceManifest, and updates the retrieval + rag-chat flow to use that evidence/citation payload.

Changes:

• Added buildScientificEvidencePayload to convert Chroma results into evidenceContext, citations, and sourceManifest with size bounds and deduplication.
• Updated ingestion to persist citation-friendly primitive metadata (title/source/page/chunk identifiers, plus optional PDF info fields).
• Updated fetch-documents to return bounded evidence/citation payload and rag-chat to consume it via same-origin retrieval.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ui/utils/server/scientific-evidence.ts	New evidence/citation payload builder for Chroma query results, including bounding/deduplication and manifest creation.
ui/pages/api/fetch-documents.ts	Adds request validation + bounded nResults/evidence size, returns evidence context + citations/manifest alongside raw Chroma results.
ui/pages/api/inject-documents.ts	Normalizes/stores primitive PDF/chunk metadata during ingestion to support citations/manifest.
ui/pages/api/rag-chat.ts	Switches retrieval to same-origin /api/fetch-documents and prompts the model to cite using provided SRC-* keys.
ui/tests/scientific-evidence.test.ts	Adds unit tests for dedupe, normalization, and evidence-size bounding behavior.

Comments suppressed due to low confidence (1)

ui/pages/api/rag-chat.ts:184

• temperatureToUse is computed earlier, but the OpenAI call hard-codes 0 for temperature. This makes the temperature request parameter and DEFAULT_TEMPERATURE ineffective for rag-chat. Pass temperatureToUse to OpenAIStream (as done in pages/api/chat.ts).

    const stream = await OpenAIStream(
      model,
      promptToSend,
      0,
      key,
      messagesToSend,
    );

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[victortran0904]

Follow-up pushed in a04f30f to address Copilot review feedback:

• safeTruncate now always respects the requested max length, including tiny limits shorter than the ellipsis.
• Citation keys and source IDs are fixed-width 8-character uppercase hex, matching SRC-XXXXXXXX / DOC-XXXXXXXX expectations.
• Evidence source extraction now prefers citation-friendly public fields before path-like metadata and normalizes path-like values down to a public basename.
• PDF ingestion stores a public filename/source label instead of the server temporary filepath; if no original filename is available it uses a generic public label rather than a temp path.

Validation after the follow-up:

• npx vitest run __tests__/scientific-evidence.test.ts --reporter verbose -> 5 passed
• npx tsc --noEmit --pretty false -> passed
• npm run lint -> passed with existing unrelated React hook dependency warnings
• npx prettier --check pages/api/fetch-documents.ts pages/api/inject-documents.ts pages/api/rag-chat.ts utils/server/scientific-evidence.ts __tests__/scientific-evidence.test.ts -> passed
• git diff --check -> passed

[victortran0904]

Additional small follow-up pushed in da0f3ed after checking the suppressed Copilot note:

• rag-chat now passes the computed temperatureToUse into OpenAIStream instead of hard-coding 0, matching the existing non-RAG chat path.

Validation after this follow-up:

• npx vitest run __tests__/scientific-evidence.test.ts --reporter verbose -> 5 passed
• npx tsc --noEmit --pretty false -> passed
• npm run lint -> passed with the same existing unrelated React hook dependency warnings
• npx prettier --check pages/api/fetch-documents.ts pages/api/inject-documents.ts pages/api/rag-chat.ts utils/server/scientific-evidence.ts __tests__/scientific-evidence.test.ts -> passed
• git diff --check -> passed

[victortran0904]

Verified current head da0f3ed already addresses the remaining fixed-width citation ID thread.

Evidence:

• utils/server/scientific-evidence.ts now returns uppercase hex padded to 8 chars via toString(16).toUpperCase().padStart(8, "0").
• __tests__/scientific-evidence.test.ts asserts ^SRC-[0-9A-F]{8}$ and ^DOC-[0-9A-F]{8}$.

Validation:

• rtk npm ci -> passed, with existing audit findings
• rtk npm test -- --run __tests__/scientific-evidence.test.ts -> 5 passed
• rtk npm run lint -> passed with existing unrelated React hook dependency warnings

No new code changes were needed for this pass.