ClaimCheck – Structured evaluation of AI vendor claims. (MVP)

Originally developed by Michael Scott / Composed Software + ChatGPT & Codex

ClaimCheck applies an explicit, rule-based rubric to AI vendor claims to estimate evidence strength, business impact, and risk signals.

Important

The GitHub Pages site is a frontend demo only. It does not run the backend API or database, so end-to-end claim analysis is not functional there.

Acknowledgements

This project was developed conversationally using AI assistance.
Initial design, specification, and guidance were provided by the project author, Michael Scott.
Coding support and implementation were assisted using ChatGPT and Codex during development.

While AI tools were used to help generate and iterate on code and design elements, all decisions regarding architecture, scoring logic, and product direction were guided by the author.

Monorepo layout

• apps/api: FastAPI backend (/analyze_claim, /evidence/search, /health)
• apps/web: Vite + React + TypeScript frontend (/, /app, /resources)
• docker-compose.yml: Postgres + pgvector, plus optional API service

What the app does

Input a claim and receive:

• category: Mature & Proven | Scaling & Strategic | Frontier & Experimental | Mirage Zone
• maturity_score (0-10, shown in UI as Evidence strength), impact_score (0-10)
• red_flags, cautions, questions
• evidence (keyword-matched DB evidence + user links + optional named-customer signals)
• plot coordinates (maturity, impact)

API contract

POST /analyze_claim

Request:

{
  "text": "string",
  "industry": "string|null",
  "claimed_outcomes": ["string"],
  "links": ["string"],
  "named_customers": ["string"]
}

Response:

{
  "category": "string",
  "maturity_score": 0,
  "impact_score": 0,
  "evidence": [
    {
      "title": "string",
      "url": "string",
      "published_at": "YYYY-MM-DD|null",
      "quote": "string",
      "matched_keywords": ["string"],
      "match_reason": "keyword:title|keyword:snippet|user_link",
      "independent": true,
      "named_customer": "string|null",
      "production": false,
      "source_type": "independent_study|customer_case_study|testimonial|vendor_blog|vendor_website|user_supplied|placeholder",
      "source_weight": 0.0
    }
  ],
  "red_flags": ["string"],
  "cautions": ["string"],
  "questions": ["string"],
  "plot": { "maturity": 0, "impact": 0 }
}

POST /evidence/search

Request:

{
  "query": "string",
  "top_k": 5
}

GET /health

Returns:

{ "status": "ok" }

Scoring rubric (implemented, rules-based)

Maturity score

• independent_refs: weighted independent evidence, rounded, cap 2
• named_customers: weighted named-customer evidence, rounded, cap 2
• production_mentions: weighted production evidence, rounded, cap 2
• recency_bonus: +1 if any evidence in last 180 days
• red_flag_penalty: full autonomy/no human review = 2, other flags = 1, total cap 3
• structural skepticism penalties apply for weak evidence structure and vendor-controlled-only evidence

Formula:

maturity = 3*independent_refs + 2*named_customers + 2*production_mentions + recency_bonus - penalties
clamped to 0..10.

Category by maturity:

• >=7: Mature & Proven
• 5-6: Scaling & Strategic
• 3-4: Frontier & Experimental
• <=2: Mirage Zone

Impact score

Base by industry:

• marketing 6, ops 7, finance 8, healthcare 9, legal 8, manufacturing 8, supply_chain 9, default 7

Adjustments:

• +1 if claimed_outcomes is not empty
• -2 governance penalty for healthcare/legal claims implying no human review

Result clamped to 0..10.

Local run

Startup shortcuts (PowerShell, from repo root)

pnpm start:docker   # Docker DB + API
pnpm start:db       # Docker DB only
pnpm start:api      # Local API (creates venv/install deps on first run)
pnpm start:web      # Local web app
pnpm start:local    # DB + local API + local web (opens new terminals)
pnpm stop:docker    # Stop docker services

Option A: Docker for DB + API, local web

1. Start DB and API:

docker compose up --build

2. In another terminal, run web:

pnpm install
pnpm dev

Web: http://localhost:5173
API: http://localhost:8080

Option B: Docker DB + local API + local web

1. Start only DB:

docker compose up db

2. Start API:

cd apps/api
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
python -m uvicorn app.main:app --host 0.0.0.0 --port 8080 --reload

3. Start web from repo root:

pnpm install
pnpm dev

CI/CD and deployment

This repo now includes GitHub Actions workflows:

• .github/workflows/ci.yml: runs web tests/build plus API tests on pull requests and pushes to main.
• .github/workflows/deploy-pages.yml: builds apps/web and deploys it to GitHub Pages on pushes to main.

GitHub Pages limitations

GitHub Pages hosts static assets only, so this setup deploys the React frontend. The backend API (apps/api) still needs to run elsewhere (e.g., Hetzner, Fly.io, Render, Railway, etc.). If you open the GitHub Pages URL, treat it as a UI demo, not a fully working deployed system.

For production, set the frontend API endpoint with:

• VITE_API_BASE_URL in your GitHub Action/repository variables or secrets (or hardcode if preferred).

Enable Pages in GitHub

1. Push to main so the workflow runs.
2. In GitHub repo settings, open Pages.
3. Set Source to GitHub Actions.
4. After deployment, your site URL will be shown in the workflow output.

Using a custom domain (optional)

If you want your own domain/subdomain:

1. Create DNS record to GitHub Pages target.
2. Add a CNAME file in apps/web/public (or configure custom domain in Pages settings).
3. Enable HTTPS in Pages settings.

Tests

API tests:

cd apps/api
pytest

Web smoke test:

cd apps/web
pnpm test

Verdict JSON migration

When cached verdicts need refreshing, you can either normalize schema shape only, or fully recompute using current scoring/evidence logic:

cd apps/api
python -m scripts.migrate_verdict_json --dry-run
python -m scripts.migrate_verdict_json
python -m scripts.migrate_verdict_json --recompute --dry-run
python -m scripts.migrate_verdict_json --recompute

Notes

• CORS is enabled for localhost/127.0.0.1 on any local port.
• No demo seed evidence is included by default in this public version.
• Schema includes embedding vector(1536) as a placeholder for future embedding search.
• Claim.parsed_json and verdict_json are placeholders for future LLM parser and RAG pipeline integration.

Attribution

Created by Michael Scott / Composed Software.