Merge pull request #41 from namel3ss-Ai/feat/core-edition-retrieval-tuning

Feat/core edition retrieval tuning

Author: SsebowaDisan

docs/composition/includes.md

@@ -0,0 +1,47 @@
+# Deterministic Includes
+
+Evolution Edition adds deterministic multi-file composition through top-level include directives.
+
+## Capability Gate
+
+You must enable:
+
+```ai
+capabilities:
+  composition.includes
+```
+
+Without it, compile fails with:
+
+`Capability missing: composition.includes is required to use 'include' directives. Add 'capability is composition.includes' to the manifest.`
+
+## Syntax
+
+```ai
+include "modules/retrieval.ai"
+include 'modules/flows.ai'
+```
+
+Rules:
+
+- Include is top-level only.
+- Paths are relative, `.ai` only.
+- Absolute paths are rejected.
+- `..` traversal is rejected.
+- Include order is the merge order.
+
+## Merge and Validation
+
+- Root file remains authoritative for `ui` and `pages`.
+- Included files may contribute flows, records, and other declarations.
+- Duplicate declarations across files are compile errors.
+- Include cycles are compile errors with a stable path chain.
+- Duplicate include paths are collapsed with a deterministic warning:
+  - `Warning: Duplicate include ignored: "<path>"`
+
+## Determinism Guarantees
+
+- Paths are normalized to project-relative POSIX format.
+- Merge order is include order then in-file appearance order.
+- Source-map entries are stable and sorted by `decl_id`.
+- No absolute paths are emitted in composition metadata.

docs/determinism/regression_suite.md

@@ -0,0 +1,26 @@
+# Determinism Regression Suite
+
+Evolution Edition makes repeat-run stability a first-class CI signal.
+
+## Core Checks
+
+- Repeated compile of the same app must produce byte-identical manifest JSON.
+- Repeated retrieval trace generation must produce byte-identical trace JSON.
+- Include warnings and composition source-map output must be stable.
+- UI baselines must match checked-in goldens.
+
+## Local Commands
+
+```bash
+python tools/determinism_repeat_check.py app.ai
+python tools/ui_baseline_refresh.py --check
+python -m namel3ss.beta_lock.repo_clean
+```
+
+## CI Policy
+
+Determinism failures are blocking:
+
+- No timestamps or random IDs in compiler/runtime artifacts.
+- No absolute paths in manifests, diagnostics payloads, or emitted composition metadata.
+- Warning and error ordering remains stable across runs.

docs/diagnostics/retrieval_trace.md

@@ -0,0 +1,44 @@
+# Retrieval Trace and What-If
+
+Evolution Edition adds a deterministic retrieval trace contract for Studio diagnostics.
+
+## Capability Gate
+
+Enable:
+
+```ai
+capabilities:
+  diagnostics.trace
+```
+
+If missing, compiler emits:
+
+`Warning: Retrieval trace diagnostics are disabled (missing capability diagnostics.trace).`
+
+## Trace Contract
+
+Trace payload includes:
+
+- `query`
+- `params` (`semantic_weight`, `semantic_k`, `lexical_k`, `final_top_k`)
+- `filter_tags` (sorted)
+- `semantic`, `lexical`, `final` candidate lists
+- `tie_breaker` (static deterministic ordering description)
+
+## Deterministic Ordering
+
+Rows are sorted with:
+
+1. `final_score` descending
+2. `semantic_score` descending
+3. `lexical_score` descending
+4. `doc_id` ascending
+
+## What-If Simulation
+
+Studio computes deterministic what-if ranking from stored trace data only:
+
+- No external calls
+- No rerun of retrieval
+- Pure function of `(trace, params)`
+- Same input produces identical output

docs/manifest/composition_rules.md

@@ -0,0 +1,17 @@
+# Composition rules
+
+Namel3ss composition is deterministic and layer-scoped.
+
+1. Theme composition:
+   - App-level theme defaults are applied first.
+   - Page-level theme overrides apply next.
+   - Runtime theme state (when enabled) applies last.
+2. Page composition:
+   - Slot layout pages are evaluated in canonical slot order: `header`, `sidebar`, `main`, `footer`.
+   - Retrieval explain elements are injected ahead of non-error content in the active slot.
+3. Module composition:
+   - Imported module exports must be explicit.
+   - Flow and UI names must be unique at composition time.
+   - Unsupported exported actions are dropped by export guards instead of re-ordered or rewritten.
+
+Composition order is stable across runs, and manifests are serialized canonically.

docs/reports/code_audit.md

@@ -0,0 +1,39 @@
+# Retrieval Tuning
+
+Retrieval tuning controls adjust semantic/lexical blending and candidate limits while preserving deterministic ranking and preview output.
+
+## Tunable fields
+
+- `semantic_weight` (0..1)
+- `semantic_k` (int)
+- `lexical_k` (int)
+- `final_top_k` (int)
+
+Runtime state is stored under `state.retrieval.tuning` with canonical key ordering.
+
+## Deterministic preview ordering
+
+Preview rows are sorted by:
+1. `final_score` (desc)
+2. `semantic_score` (desc)
+3. `lexical_score` (desc)
+4. `doc_id` (asc)
+5. `chunk_id` (asc internal tie-break)
+
+## Tag filtering
+
+`filter_tags` can be provided explicitly or resolved from UI state scopes. Matching is deterministic:
+
+- normalized tags (trimmed, deduped, sorted)
+- per-result `matched_tags` sorted ascending
+- empty matches return deterministic empty `results` and `retrieval_preview`
+
+## Flows
+
+The platform wires retrieval tuning controls to flow actions when available:
+
+- `set_semantic_weight`
+- `set_semantic_k`
+- `set_lexical_k`
+- `set_final_top_k`
+

docs/retrieval/tuning.md

@@ -0,0 +1,34 @@
+# Retrieval tuning flows
+
+Namel3ss supports four built-in retrieval tuning flows:
+
+- `set_semantic_k(k: int)`
+- `set_lexical_k(k: int)`
+- `set_final_top_k(k: int)`
+- `set_semantic_weight(weight: float)`
+
+## Behavior
+
+- Calls are applied in declaration order within a flow.
+- `set_semantic_k` and `set_lexical_k` define candidate pool limits.
+- `set_final_top_k` limits final output size after semantic/lexical merge.
+- `set_semantic_weight` sets blend weight in `[0, 1]` and defaults to `0.5` when omitted.
+
+## Validation
+
+- Compile-time validation checks ordering and literal argument ranges when literals are provided.
+- Runtime raises `ValueError` for out-of-range dynamic values.
+- Unsupported tuning-like names raise `UnknownFlowError` at compile time.
+
+## State contract
+
+Runtime stores tuning state under:
+
+`state.retrieval.tuning`
+
+Canonical field order:
+
+1. `semantic_k`
+2. `lexical_k`
+3. `final_top_k`
+4. `semantic_weight`

docs/runtime/retrieval_flows.md

@@ -643,6 +643,17 @@ page "home":
   - `upload.missing_control` when uploads capability is enabled but no upload control is declared
   - `upload.unused_declaration` when an upload declaration is never referenced from `state.uploads.<name>`
 
+## 4.4) Retrieval tuning controls (Studio)
+- Retrieval tuning uses normal flow calls and does not add new UI DSL syntax.
+- Supported runtime tuning flows are:
+  - `set_semantic_k(<int>)`
+  - `set_lexical_k(<int>)`
+  - `set_final_top_k(<int>)`
+  - `set_semantic_weight(<number in [0, 1]>)`
+- Studio may render interim controls (dropdowns/radios) when these flows exist with valid one-field numeric contracts.
+- Production behavior is unchanged: if these flows are not defined, retrieval keeps legacy defaults.
+- Control metadata and any disabled reasons are deterministic and emitted in canonical manifest order.
+
 ## 5) Core UI primitives
 - `page "<title>":` container with optional `purpose is "<string>"` metadata (page-only; deterministic id generation). Duplicate page titles are rejected.
 - `purpose is "<string>"` may only appear at the page root; it is persisted to manifests and Studio payloads as metadata for runtime decisions.

docs/ui-dsl.md

@@ -0,0 +1,36 @@
+# Citations
+
+Enhanced citations provide stable inline chips plus deterministic snippet previews and drawer linking.
+
+## Capability
+
+`ui.citations_enhanced` enables enhanced chip/snippet behavior.
+
+If missing, the compiler emits:
+
+`Warning: Enhanced citations are disabled (missing capability ui.citations_enhanced). Falling back to legacy citations UI.`
+
+Fallback behavior is explicit and deterministic.
+
+## Citation contract
+
+Each citation entry includes:
+
+- `citation_id` (stable; generated when missing)
+- `index` (1-based, stable order)
+- `title`
+- `source_id` or `url`
+- optional deterministic `snippet` truncation
+
+## Sources drawer mapping
+
+- chips and drawer rows use `data-citation-id`
+- answer click focuses the matching drawer entry
+- drawer selection can resolve back to the matching citation ID
+
+## Snippet truncation
+
+- whitespace normalized
+- deterministic max length
+- truncated values end with `...`
+

docs/ui/citations.md

@@ -0,0 +1,39 @@
+# Diagnostics Panel
+
+The diagnostics panel supports deterministic filtering and sorting for retrieval-focused debugging.
+
+## Sort options
+
+- `severity`
+- `semantic score`
+- `lexical score`
+- `final score`
+- `doc id`
+
+Score sorts use stable tie-breaking and deterministic ordering.
+
+## Filters
+
+Toggles:
+
+- `semantic`
+- `lexical`
+- `final`
+
+Rows are shown only when at least one enabled toggle matches the row mode set.
+
+## Per-row metadata
+
+Each rendered row includes stable metadata:
+
+- `doc_id`
+- `semantic_score`
+- `lexical_score`
+- `final_score`
+
+## Determinism
+
+- repeated renders over the same diagnostics payload produce the same row order
+- score sorting has fixed numeric comparison and tie-break rules
+- empty filter state shows a stable empty-state row
+