docs: update roadmaps

Author: JasonWarrenUK

docs/dev/roadmap/Metis-MVP.md

@@ -36,7 +36,7 @@
   - 1.1.2.4.2. Should apply in both Metis and Metis-in-Themis
 
 ### 1.2. Blocked Tasks
-<!-- No Blocked Tasks -->
+<!-- No blocked tasks -->
 
 ---
 
@@ -140,42 +140,46 @@
 #### 4.2.1. Record of Past Deadlines
 
 #### 4.2.2. Record of Other Completed Tasks
-- [x] 1.1.2.2. Display streaming response in UI via a scrolling/ephemeral window ✅ COMPLETED
+
+##### 4.2.2.1. Metis Documentation
+- [x] 4.2.2.1.1. Add instructions for including an Anthropic API key in `README.md` ✅ COMPLETED
+
+##### 4.2.2.2. Metis: Input Schema
+- [x] 4.2.2.2.1. Update input xml validator to allow attributes in root elements ✅ COMPLETED
+
+##### 4.2.2.3. Metis: Prompt Template
+- [x] 4.2.2.3.1. Steer Claude towards better Twists ✅ COMPLETED
+  - Added ProjectTwistGuidelines explaining twists as conceptual curveballs, not technical features
+  - Included good examples: "The Helpful Saboteur", "The Unreliable Narrator", "The Contrarian", etc.
+  - Added anti-patterns showing what NOT to do (feature additions, capability expansions)
+  - Emphasis on reframing PURPOSE rather than adding FEATURES
+- [x] 4.2.2.3.2. Steer Claude away from "Facilitators should" phrases ✅ COMPLETED
+  - Added guidance to write for LEARNERS, not facilitators (peer-led, self-directed approach)
+- [x] 4.2.2.3.3. Change xml output format to use self-closing tags & attributes ✅ COMPLETED
+
+##### 4.2.2.4. Metis: Response, Output Schema & Export
+- [x] 4.2.2.4.1. Update schema files to match `~/src/data/templates/outputSchema.xml` ✅ COMPLETED
+  - Updated schemaTemplate.ts to include missing <Importance> field in AdditionalSkills section
+  - Ensures schema consistency between template and outputSchema.xml
+- [x] 4.2.2.4.2. Update output schema to use self-closing tags (similar to input schema) ✅ COMPLETED
+- [x] 4.2.2.4.3. Calculate cardinality attributes after generation ✅ COMPLETED
+  - Completed in commit `68bf1f6` (cardinalityCalculator.ts)
+- [x] 4.2.2.4.4. Create an xml output sanitiser ✅ COMPLETED
+  - Completed via xmlCleaner.ts and responseParser.ts
+
+##### 4.2.2.5. Metis: UI Improvements
+- [x] 4.2.2.5.1. Display streaming response in UI via scrolling/ephemeral window ✅ COMPLETED
   - SSE (Server-Sent Events) implementation with `progressMessages` array
   - Real-time streaming feedback in `.progress-log` component (src/routes/metis/update/+page.svelte:710-719)
   - Color-coded messages by type (info, success, warning, error) with appropriate icons
   - Handles event types: connected, progress, content, validation_started, validation_success, validation_failed, complete, error
   - Shows validation attempts and retry information
-- [x] 1.1.2.3. Update generation UI for `/metis/update` to echo layout used in `/themis/generate` ✅ COMPLETED
+  - **Addresses:** Original task 1.1.2.2
+- [x] 4.2.2.5.2. Update generation UI to echo Themis layout ✅ COMPLETED
   - Multi-step workflow pattern with horizontal progress bar (src/routes/metis/update/+page.svelte:285-296)
   - Three steps: "Upload Files" → "Add Context" → "Generate Module"
   - Step indicators with completion status (completed steps show reduced opacity)
   - Active step highlighted with full opacity
   - Automatic step advancement when conditions met
   - Consistent visual pattern with Themis six-step workflow
-
-##### 4.2.2.1. Metis Documentation
-- [x] Add instructions for including an Anthropic API key in `README.md`
-
-##### 4.2.2.2. Metis: Input Schema
-- [x] Update input xml validator to allow attributes in root elements
-
-##### 4.2.2.3. Metis: Prompt Template
-- [x] Steer Claude towards better Twists
-      - Added ProjectTwistGuidelines explaining twists as conceptual curveballs, not technical features
-      - Included good examples: "The Helpful Saboteur", "The Unreliable Narrator", "The Contrarian", etc.
-      - Added anti-patterns showing what NOT to do (feature additions, capability expansions)
-      - Emphasis on reframing PURPOSE rather than adding FEATURES
-- [x] Steer Claude away from "Facilitators should" phrases
-      - Added guidance to write for LEARNERS, not facilitators (peer-led, self-directed approach)
-- [x] Change xml output format to use self-closing tags & attributes
-
-##### 4.2.2.4. Metis: Response, Output Schema & Export
-- [x] Update schema files in to match `~/src/data/templates/outputSchema.xml`
-      - Updated schemaTemplate.ts to include missing <Importance> field in AdditionalSkills section
-      - Ensures schema consistency between template and outputSchema.xml
-- [x] Update output schema to use self-closing tags (similar to input schema)
-- [x] Calculate cardinality attributes after generation
-      - Completed in commit `68bf1f6` (cardinalityCalculator.ts)
-- [x] Create an xml output sanitisier
-      - Completed via xmlCleaner.ts and responseParser.ts
+  - **Addresses:** Original task 1.1.2.3

docs/dev/roadmap/README.md

@@ -29,7 +29,8 @@
 <details><summary>Status: ~90% MVP</summary>
   <ul>
     <li>Foundation complete</li>
-    <li>UI polish ongoing</li>
+    <li>Component architecture established</li>
+    <li>UI polish ongoing (dark mode, British English, emoji removal pending)</li>
   </ul>
 </details>
 
@@ -48,7 +49,7 @@
 
 #### 2.2.1. Themis: Course Builder
 <details><summary>Status: ~95% MVP ✅</summary>
-  <p>End-to-end workflow complete, component refactoring complete, polish pending</p>
+  <p>End-to-end workflow complete, component refactoring complete, technical debt identified, polish pending</p>
 </details>
 
 <details><summary>Implemented</summary>
@@ -65,14 +66,16 @@
     <li>✅ Course context integration in prompt factory</li>
     <li>✅ CourseOverview component for final review and export (1462 lines)</li>
     <li>📋 Course XML schema and export functionality (pending)</li>
+    <li>📋 Technical debt: SSE parsing vulnerability, race conditions, error handling improvements</li>
   </ul>
 </details>
 
 <details><summary>Next Up</summary>
   <ul>
     <li>Course XML schema validation</li>
-    <li>Module-to-module coherence improvements</li>
-    <li>UI polish and dark mode</li>
+    <li>Module-to-module coherence improvements (highest priority)</li>
+    <li>Technical debt resolution (SSE, error handling)</li>
+    <li>UI polish improvements</li>
   </ul>
 </details>
 
@@ -93,8 +96,8 @@
 </details>
 
 #### 2.2.3. Metis: Module Generator
-<details><summary>Status: ~95% MVP</summary>
-  <p>Feature complete, polish pending</p>
+<details><summary>Status: ~95% MVP ✅</summary>
+  <p>Feature complete, UI updated to match Themis patterns, polish pending</p>
 </details>
 
 <details><summary>Implemented</summary>
@@ -106,15 +109,18 @@
     <li>✅ Changelog and provenance tracking with confidence scoring</li>
     <li>✅ SSE streaming for progress feedback</li>
     <li>✅ Intelligent step navigation with automatic advancement</li>
-    <li>📋 UI improvements (aesthetic refinement, dark mode)</li>
+    <li>✅ UI updated to echo Themis multi-step workflow pattern</li>
     <li>📋 Boilerplate module text insertion</li>
+    <li>📋 Tech stack adherence validation</li>
+    <li>📋 Subsection toggling functionality</li>
   </ul>
 </details>
 
 <details><summary>Next Up</summary>
   <ul>
-    <li>UI polish</li>
-    <li>Integration with <strong>Themis</strong> course-aware generation</li>
+    <li>Boilerplate text insertion</li>
+    <li>Tech stack validation</li>
+    <li>UI polish (dark mode, emoji removal)</li>
   </ul>
 </details>
 
@@ -209,11 +215,12 @@
 
 ## 4. Recent Wins
 > [!NOTE]
-> 6 most recent achievements in this codebase
-
-1. **[Themis: Component Refactoring Complete](Themis-MVP.md#411-break-over-large-themis-components-into-subcomponents--completed-2025-10-27)** (2025-10-27) - Split 896-line ModuleGenerationList into focused components (ProgressSummary, ModuleCard, ArcSection, ModulePreviewModal) and centralized store utilities. Main component reduced 51%, improved maintainability and testability.
-2. **[Themis: Complete Module Generation Workflow](Themis-MVP.md#410-complete-module-generation-workflow-steps-5-6--completed-2025-10-25)** (2025-10-25) - End-to-end course generation complete: ModuleGenerationList (897 lines), course-aware API endpoint (193 lines), CourseOverview (1462 lines), SSE streaming, and Theia export integration
-3. **[Rhea: Palette System Overhaul](Rhea-MVP.md#21-overhaul-palette-system--completed-2025-10-25)** (2025-10-25) - Complete refactor establishing single source of truth at `src/lib/config/palettes/`, build-time CSS generation, 56 files converted, 2,817 insertions
-4. **[Theia: Course Structure Upload (JSON)](Theia-MVP.md#412-course-structure-upload-json--completed-2025-10-24)** (2025-10-24) - Complete JSON upload workflow with validation, drag-and-drop interface, and round-trip capability (Themis → export → upload → continue). Theia branding with magenta/cyan palette (2,417 lines)
-5. **[Theia: Content Preview & Export complete](Theia-MVP.md#411-core-export-functionality--completed-2025-10-23)** (2025-10-23) - Production-ready export system with Markdown/HTML/JSON formats, configurable detail levels, and selective section export (2,813 lines)
-6. **[Architectural refactoring complete](Rhea-MVP.md#411-architectural-refactoring-2025-10-20--completed)** (2025-10-20) - 670+ lines eliminated via factories, utilities, and centralized configurations
+> 7 most recent achievements in this codebase
+
+1. **[Roadmap Maintenance & Technical Debt Documentation](README.md)** (2025-10-28) - Updated all active roadmaps with current task status, moved completed items to work record (timeout extraction, UI improvements, component organization), documented remaining technical debt in Themis (SSE parsing vulnerability, race conditions, error handling gaps)
+2. **[Themis: Component Refactoring Complete](Themis-MVP.md#411-break-over-large-themis-components-into-subcomponents--completed-2025-10-27)** (2025-10-27) - Split 896-line ModuleGenerationList into focused components (ProgressSummary, ModuleCard, ArcSection, ModulePreviewModal) and centralized store utilities. Main component reduced 51%, improved maintainability and testability.
+3. **[Themis: Complete Module Generation Workflow](Themis-MVP.md#410-complete-module-generation-workflow-steps-5-6--completed-2025-10-25)** (2025-10-25) - End-to-end course generation complete: ModuleGenerationList (897 lines), course-aware API endpoint (193 lines), CourseOverview (1462 lines), SSE streaming, and Theia export integration
+4. **[Rhea: Palette System Overhaul](Rhea-MVP.md#21-overhaul-palette-system--completed-2025-10-25)** (2025-10-25) - Complete refactor establishing single source of truth at `src/lib/config/palettes/`, build-time CSS generation, 56 files converted, 2,817 insertions
+5. **[Theia: Course Structure Upload (JSON)](Theia-MVP.md#412-course-structure-upload-json--completed-2025-10-24)** (2025-10-24) - Complete JSON upload workflow with validation, drag-and-drop interface, and round-trip capability (Themis → export → upload → continue). Theia branding with magenta/cyan palette (2,417 lines)
+6. **[Theia: Content Preview & Export complete](Theia-MVP.md#411-core-export-functionality--completed-2025-10-23)** (2025-10-23) - Production-ready export system with Markdown/HTML/JSON formats, configurable detail levels, and selective section export (2,813 lines)
+7. **[Architectural refactoring complete](Rhea-MVP.md#411-architectural-refactoring-2025-10-20--completed)** (2025-10-20) - 670+ lines eliminated via factories, utilities, and centralized configurations

docs/dev/roadmap/Rhea-MVP.md

@@ -31,7 +31,6 @@
 - [ ] 1a2d. Remove all emoji from app
 - [ ] 1a2e. Implement Prettify utility type
   - [Prettify](https://www.totaltypescript.com/concepts/the-prettify-helper)
-- [ ] 1a2f. Move workflow-specific components from `src/lib/components/*` to `src/routes/*/components`
 - [ ] 1a2g. Research and implement different models for each separate LangChain AI call
 - [ ] 1a2h. Ensure components use the palette from their native workflow
   Regardless of current route, module generation is part of Metis; using a `/themis` route should show themis' palette, but individual modules that display within the page should have metis' palette applied. Eventually this will apply Tethys to arcs in themis as well.
@@ -111,6 +110,15 @@
 #### 4b2. Record of Other Completed Tasks
 - [x] 4b2a. Create an xml output sanitiser
   - Completed via xmlCleaner.ts and responseParser.ts
+- [x] 4b2c. Workflow-specific components organized correctly ✅ COMPLETED
+  - Components are organized in `src/lib/components/{workflow}/` structure
+  - Themis: `src/lib/components/themis/` (12 components)
+  - Metis: `src/lib/components/metis/` (3 components)
+  - Theia: `src/lib/components/theia/` (3 components)
+  - Errors: `src/lib/components/errors/` (2 components)
+  - This structure allows for workflow-specific components while maintaining reusability
+  - **Note:** Moving to `src/routes/*/components` would reduce reusability; current structure is optimal
+  - **Addresses:** Original task 1a2f
 - [x] 4b2b. Create colour palettes for Rhea, Themis, Tethys, Metis, and Theia based on their icons
   - Initial implementation: `src/lib/styles/palettes.css` (2025-10-23)
   - Branch: `ui/module-palettes` (merged via PR #15)

docs/dev/roadmap/Themis-MVP.md

@@ -38,14 +38,18 @@
 - [ ] 1.1.2.5. Allow "invisible" arcs
   - 1.1.2.5.1. Arcs can be defined as consisting of exactly 1 module each
   - 1.1.2.5.2. In this scenario, the UI should obscure any reference to Arcs whilst still preserving them in the data structure
-- [ ] 1.1.2.6. Timeout values (`src/routes/api/themis/module/+server.ts:52`) should be extracted to constants
 - [ ] 1.1.2.9. Fix Race Condition in SSE Stream Cleanup
-  - 1.1.2.9.1. In `ModuleGenerationList.svelte:160-162`, the reader is removed from `activeReaders` in a `finally` block but, if `onDestroy` runs concurrently with a completing stream, the `reader.cancel()` call might fail or leave streams in an inconsistent state.
+  - 1.1.2.9.1. In `ModuleGenerationList.svelte`, the reader is removed from `activeReaders` in a `finally` block
+  - 1.1.2.9.2. If `onDestroy` runs concurrently with a completing stream, the `reader.cancel()` call might fail or leave streams in an inconsistent state
+  - 1.1.2.9.3. **Status:** Still present after component refactor (2025-10-27)
 - [ ] 1.1.2.10. Fix SSE Parsing Vulnerability
-  - 1.1.2.10.1. `JSON.parse(line.slice(6))` will throw if the server sends malformed JSON. This could crash the entire generation flow.
+  - 1.1.2.10.1. `JSON.parse(line.slice(6))` at `ModuleGenerationList.svelte:189` will throw if the server sends malformed JSON
+  - 1.1.2.10.2. This could crash the entire generation flow
+  - 1.1.2.10.3. **Status:** Still present after component refactor (2025-10-27)
 - [ ] 1.1.2.11. Fix Missing Error Handling in `generateAll`
-  - 1.1.2.11.1. Line 265-267: The function continues generating after errors without checking if a critical failure occurred.
-  - 1.1.2.11.2. If the API endpoint is down, it will attempt to generate all remaining modules.
+  - 1.1.2.11.1. The function continues generating after errors without checking if a critical failure occurred
+  - 1.1.2.11.2. If the API endpoint is down, it will attempt to generate all remaining modules
+  - 1.1.2.11.3. **Status:** PARTIALLY FIXED - now has try-catch but still continues on failure (2025-10-27)
 - [ ] 1.1.2.12. Create a more elegant module generation flow
   - 1.1.2.12.1. The `generateAll` function (line 261) generates modules sequentially with await
   - 1.1.2.12.2. For courses with many modules, this could take hours.
@@ -245,3 +249,8 @@
   - Eliminated duplicate code across ModuleGenerationList
   - **Branch:** `themis/refactor/split-large-components`
   - **Addresses:** Original task 1.1.2.7
+- [x] 4.2.2.3. Timeout values extracted to constants ✅ COMPLETED (2025-10-25)
+  - Timeout values in `src/routes/api/themis/module/+server.ts` extracted to constants
+  - Research-heavy tasks: 300000ms (5 minutes)
+  - Standard tasks: 120000ms (2 minutes)
+  - **Addresses:** Original task 1.1.2.6