feat(census): add AI-powered census transcription with schema-based extraction

Major new feature: Census Transcription Tab with Gemini vision model integration

Key changes:
- Add census transcription UI tab with image selection and preview
- Implement YAML-based census schemas for years 1790-1950
- Create schema-driven prompt builder, response parser, and data validator
- Add LLM interaction logger for debugging (llm_interactions.log)
- Add HTML results display in browser tab with census-style formatting
- Add full-screen image viewing in browser for detailed inspection
- Optimize SQL queries from N+1 to batch pattern (3 queries vs 1400+)

New components:
- src/rmcitecraft/ui/tabs/census_transcription.py - Main UI tab
- src/rmcitecraft/schemas/census/*.yaml - Census year schemas
- src/rmcitecraft/services/census/ - Schema-based transcription services
- src/rmcitecraft/llm/llm_logger.py - Dedicated LLM logging
- src/rmcitecraft/models/census_schema.py - Schema dataclasses

Testing:
- Unit tests for all census service components
- Integration tests for transcription service
- Functional tests for schema loading

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: miams

GEMINI.md

@@ -0,0 +1,114 @@
+# GEMINI.md
+
+This document provides a comprehensive overview of the RMCitecraft project, its architecture, and development conventions to be used as a guide for future interactions.
+
+## Project Overview
+
+RMCitecraft is a Python-based desktop application designed to automate citation formatting and image management for the RootsMagic genealogy software. It primarily targets US Federal Census (1790-1950) and Find a Grave records. The application provides a user-friendly interface built with NiceGUI to streamline the process of creating *Evidence Explained* compliant citations and organizing associated images.
+
+The core value proposition is to save genealogists significant time and effort by automating tedious manual tasks, such as reformatting citations, renaming image files, and linking media to records within the RootsMagic database.
+
+## Architecture
+
+RMCitecraft is built on a modern Python stack and employs a service-oriented architecture.
+
+*   **Frontend:** The user interface is a web-based UI powered by **NiceGUI** (native mode). The UI is structured into tabs for different functionalities like batch processing and citation management.
+*   **Backend:** Python 3.11+ handles core logic, database interactions, and browser automation.
+*   **Database:** The application directly interacts with **SQLite** databases (RootsMagic 8/9 format).
+    *   **CRITICAL:** Uses the ICU extension for `RMNOCASE` collation.
+    *   **State Persistence:** Application state is saved to `~/.rmcitecraft/batch_state.db` for crash recovery.
+*   **Browser Automation:** **Playwright** is used to extract data from websites like FamilySearch and Find a Grave.
+*   **LLM Integration:** **LangChain** integrates LLMs (Claude, GPT, Ollama) for intelligent citation parsing.
+*   **Asynchronous Operations:** Extensive use of `asyncio` for non-blocking I/O and web automation.
+
+### Robustness Features
+*   **Adaptive Timeouts:** Dynamic adjustments for unreliable network connections.
+*   **Page Health Monitoring:** Detects and recovers from browser crashes.
+*   **Atomic Transactions:** Ensures database integrity during batch writes.
+
+## Critical Protocols
+
+### 1. Database Safety
+*   **Working Copy:** Operations must be performed on a working copy (e.g., `data/Iiams.rmtree`), NEVER on the production database.
+*   **RMNOCASE Collation:** You **must** load the ICU extension before querying the database.
+    ```python
+    from src.rmcitecraft.database.connection import connect_rmtree
+    conn = connect_rmtree('data/Iiams.rmtree')  # Loads ICU extension automatically
+    ```
+*   **Free-Form Citations:** For census citations, RootsMagic stores data in the **SourceTable.Fields BLOB**, not CitationTable TEXT fields.
+*   **Shared Events:** Census records are often shared via `WitnessTable`. Check both `EventTable` and `WitnessTable`.
+
+### 2. Citation Formatting
+*   **Standard:** Output must strictly adhere to *Evidence Explained* format.
+*   **Validation:** `FormattedCitationValidator` enforces rules (e.g., `footnote != short_footnote`).
+*   **Census Variations:**
+    *   1790-1840: Household head only.
+    *   1850-1870: Individual, no ED.
+    *   1880-1940: ED, sheet, family number.
+    *   1950: Uses "stamp" instead of "sheet".
+
+## Key Components
+
+```
+src/rmcitecraft/
+├── config/                # Settings (settings.py)
+├── database/              # Database access layer
+│   ├── connection.py                    # ICU extension, RMNOCASE
+│   ├── batch_state_repository.py        # Find a Grave state
+│   └── census_batch_state_repository.py # Census state
+├── services/              # Business logic
+│   ├── batch_processing.py              # Workflow controller
+│   ├── familysearch_automation.py       # FamilySearch browser automation
+│   ├── findagrave_automation.py         # Find a Grave automation
+│   └── citation_formatter.py            # Evidence Explained formatting
+├── ui/
+│   ├── tabs/              # Main UI tabs (BatchProcessing, CitationManager)
+│   └── components/        # Reusable components
+├── validation/
+│   └── data_quality.py    # FormattedCitationValidator
+└── main.py                # Application entry point
+```
+
+## Development Setup
+
+1.  **Install UV:** `curl -LsSf https://astral.sh/uv/install.sh | sh`
+2.  **Clone:** `git clone <repo_url>` and `cd RMCitecraft`
+3.  **Install:** `uv sync`
+4.  **Configure:** Copy `config/.env.example` to `.env` and set `RM_DATABASE_PATH`.
+
+## Building and Running
+
+| Command | Description |
+|---------|-------------|
+| `rmcitecraft start` | Start application (interactive) |
+| `rmcitecraft start -d` | Start in background |
+| `rmcitecraft status` | Check application status |
+| `rmcitecraft stop` | Stop application |
+| `uv run python sqlite-extension/python_example.py` | Verify DB connection |
+
+## Development Conventions
+
+### Before Modifying Code
+1.  **Read source:** Understand the specific module.
+2.  **Check tests:** Look for existing tests in `tests/unit/` and `tests/integration/`.
+3.  **Database Ops:** Review `docs/reference/DATABASE_PATTERNS.md` before writing SQL.
+4.  **Batch Processing:** Check state schema in `docs/reference/BATCH_STATE_DATABASE_SCHEMA.md`.
+
+### Testing Strategy
+*   **Unit Tests (`tests/unit/`):** Fast, isolated logic tests.
+*   **Integration Tests (`tests/integration/`):** Component interaction.
+*   **E2E Tests (`tests/e2e/`):** Browser automation (requires Chrome).
+*   **Database Integrity:** Use comparison-based testing to catch undocumented RootsMagic conventions.
+
+### Code Quality
+*   **Lint/Format:** `uv run ruff check . && uv run ruff format .`
+*   **Type Check:** `uv run mypy src/`
+*   **Test:** `uv run pytest`
+
+## Key Documentation
+*   **`CLAUDE.md`**: Primary developer guide and project status.
+*   **`AGENTS.md`**: Machine-readable instructions.
+*   **`docs/reference/schema-reference.md`**: RootsMagic database schema.
+*   **`docs/reference/DATABASE_PATTERNS.md`**: SQL patterns and examples.
+*   **`docs/BATCH_PROCESSING_PHASE1_IMPLEMENTATION.md`**: Details on the batch processing UI.
+*   **`docs/analysis/`**: In-depth analysis of Census eras and data structures.

README.md

@@ -64,7 +64,7 @@ Genealogists using RootsMagic spend significant time manually formatting citatio
 ### System
 - **Platform**: macOS (Apple Silicon optimized)
 - **Python**: 3.11+
-- **Database**: RootsMagic 8 or 9 (.rmtree SQLite database)
+- **Database**: RootsMagic 11 (tested), should work well for versions 9, 10, 11.
 
 ### Optional
 - **LLM API Key**: For citation parsing (Anthropic Claude, OpenAI, or local Ollama)
@@ -130,6 +130,13 @@ rmcitecraft help
 - **[PRD.md](PRD.md)** - Complete product requirements
 - **[docs/reference/schema-reference.md](docs/reference/schema-reference.md)** - RootsMagic database schema
 
+## Caveats and Known Issues
+1. The 1950 Census tested the household-based form in a few jurisdictions in Ohio and Michigan, instead of the traditional multi-family form. The household form is not as well-parsed by FamilySearch, so RMCiteCraft asks for some fields.  The ED can be identified using the Information button when displaying the image. These forms do not have sheet/page numbers, nor line numbers.  Instead, I substitute a sequential stamped number and the image number, and I note the family form.  For example:
+
+
+   1950 U.S. census, Genesee County, Michigan, Burton Township, enumeration district (ED) 25-11, stamp 366, image 372 of 441, Brady G Ijames and Charlotte Ijames; imaged, "United States Census, 1950, household form," *FamilySearch*, (https://www.familysearch.org/ark:/61903/1:1:6JJZ-JB42  : accessed 25 November 2025).
+
+
 ## License
 
 MIT License - See LICENSE file for details.

docs/architecture/ARCHITECTURE_DIAGRAM.md

@@ -0,0 +1,69 @@
+# RMCitecraft Architecture Diagram
+
+This diagram illustrates the high-level architecture of the RMCitecraft application, showing the relationships between the UI, controllers, services, data access layer, and external systems.
+
+```mermaid
+graph TD
+    %% Styling
+    classDef ui fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
+    classDef logic fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
+    classDef service fill:#fff3e0,stroke:#ef6c00,stroke-width:2px;
+    classDef db fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
+    classDef ext fill:#eeeeee,stroke:#616161,stroke-width:2px,stroke-dasharray: 5 5;
+
+    subgraph "Frontend (NiceGUI)"
+        A["Main Entry (main.py)"] --> B[Tab Manager]
+        B --> C[Census Batch Tab]
+        B --> D[Find a Grave Tab]
+        B --> E[Citation Manager Tab]
+    end
+    class A,B,C,D,E ui;
+
+    subgraph "Application Logic"
+        C --> F[Census Batch Controller]
+        D --> G[Find a Grave Controller]
+        E --> H[Citation Manager Controller]
+    end
+    class F,G,H logic;
+
+    subgraph "Core Services"
+        F --> I[FamilySearch Automation]
+        F --> J[LLM Extractor]
+        G --> K[Find a Grave Automation]
+        L[Image Processing Service]
+        M[File Watcher]
+    end
+    class I,J,K,L,M service;
+
+    subgraph "Data Access Layer"
+        N[Census State Repository]
+        O[Find a Grave State Repository]
+        P[RootsMagic Repository]
+        Q[Database Connection]
+    end
+    class N,O,P,Q db;
+
+    subgraph "External Systems"
+        R[FamilySearch.org]
+        S[FindAGrave.com]
+        T["LLM APIs (Claude/OpenAI)"]
+        U[(Local State DB)]
+        V[(RootsMagic DB + ICU)]
+        W[File System]
+    end
+    class R,S,T,U,V,W ext;
+
+    %% Relationships
+    I -- Playwright --> R
+    K -- Playwright --> S
+    J -- LangChain --> T
+    F -- Persist State --> N
+    N -- SQLite --> U
+    G -- Persist State --> O
+    O -- SQLite --> U
+    F -- Read/Write --> P
+    P -- Uses --> Q
+    Q -- SQL/ICU --> V
+    M -- Notify --> L
+    L -- Move/Rename --> W
+```

src/rmcitecraft/llm/base.py

@@ -303,6 +303,17 @@ def transcribe_census_image(
         """
         Transcribe and extract data from census image.
 
+        .. deprecated::
+            Use CensusTranscriber or CensusTranscriptionService instead.
+            These provide year-specific schemas loaded from YAML files with
+            much better accuracy and maintainability.
+
+            Example:
+                from rmcitecraft.services.census_transcriber import CensusTranscriber
+
+                transcriber = CensusTranscriber(provider=self)
+                result = transcriber.transcribe_census(image_path, census_year)
+
         Args:
             image_path: Path to census image
             census_year: Year of census (affects expected fields)
@@ -316,12 +327,48 @@ def transcribe_census_image(
             NotImplementedError: If not supported
             LLMError: For other errors
         """
+        import warnings
+        warnings.warn(
+            "transcribe_census_image is deprecated. Use CensusTranscriber or "
+            "CensusTranscriptionService for better accuracy with year-specific schemas.",
+            DeprecationWarning,
+            stacklevel=2
+        )
+
         if not self.supports(ModelCapability.VISION, model):
-            raise NotImplementedError(f"Census transcription requires vision support")
+            raise NotImplementedError("Census transcription requires vision support")
+
+        # Use the new service internally for better results
+        try:
+            from rmcitecraft.services.census.prompt_builder import CensusPromptBuilder
+            from rmcitecraft.services.census.response_parser import CensusResponseParser
+            from rmcitecraft.services.census.schema_registry import CensusSchemaRegistry
+
+            schema = CensusSchemaRegistry.get_schema(census_year)
+            builder = CensusPromptBuilder()
+            parser = CensusResponseParser()
+
+            prompt = builder.build_transcription_prompt(schema)
+            response = self.complete_with_image(prompt, image_path, model, **kwargs)
+
+            data = parser.parse_response(response.text)
+            persons = parser.extract_persons(data)
+            metadata = parser.extract_metadata(data)
+            confidence = data.pop("confidence", 0.7) if isinstance(data, dict) else 0.7
 
-        # Define schema based on census year
+            return ExtractionResponse(
+                data={"page_info": metadata, "records": persons},
+                confidence=confidence,
+                metadata={"census_year": census_year, "raw_response": response.text},
+            )
+        except (ImportError, FileNotFoundError):
+            # Fallback to legacy behavior if new service not available
+            pass
+
+        # Legacy implementation (kept for backward compatibility)
+        import json
         if census_year >= 1850 and census_year <= 1880:
-            schema = {
+            schema_dict = {
                 "dwelling_number": "string",
                 "family_number": "string",
                 "name": "string",
@@ -333,7 +380,7 @@ def transcribe_census_image(
                 "page": "string",
             }
         elif census_year >= 1900:
-            schema = {
+            schema_dict = {
                 "sheet": "string",
                 "enumeration_district": "string",
                 "family_number": "string",
@@ -346,7 +393,7 @@ def transcribe_census_image(
                 "birthplace": "string",
             }
         else:
-            schema = {
+            schema_dict = {
                 "head_of_household": "string",
                 "free_white_males": "object",
                 "free_white_females": "object",
@@ -355,20 +402,44 @@ def transcribe_census_image(
                 "page": "string",
             }
 
-        import json
         prompt = f"""Transcribe this {census_year} US Federal Census image.
 
 Extract the following information:
-{json.dumps(schema, indent=2)}
+{json.dumps(schema_dict, indent=2)}
 
 Provide the data in JSON format with a confidence score (0.0-1.0).
 Focus on accurately transcribing names, ages, and locations."""
 
         response = self.complete_with_image(prompt, image_path, model, **kwargs)
 
         try:
-            data = json.loads(response.text)
-            confidence = data.pop('confidence', 0.7)
+            import re
+            text = response.text.strip()
+
+            json_match = re.search(r'```(?:json)?\s*\n?([\s\S]*?)\n?```', text)
+            if json_match:
+                text = json_match.group(1).strip()
+            else:
+                obj_start = text.find('{')
+                arr_start = text.find('[')
+
+                if obj_start != -1 and (arr_start == -1 or obj_start < arr_start):
+                    text = text[obj_start:]
+                elif arr_start != -1:
+                    text = text[arr_start:]
+
+            data = json.loads(text)
+
+            if isinstance(data, list):
+                confidence = 0.7
+                for record in data:
+                    if isinstance(record, dict) and 'confidence' in record:
+                        confidence = record.pop('confidence')
+                        break
+                data = {'records': data}
+            else:
+                confidence = data.pop('confidence', 0.7)
+
             return ExtractionResponse(
                 data=data,
                 confidence=confidence,