feat: cut over metadata history to snapshot store (#158)

* feat: add llmfit sidecar enrichment for hf metadata

* feat: cut over metadata history to snapshot store

* fix: satisfy dialyzer for snapshot tasks

* fix: clarify initial history seed failures

Author: mikehostetler

.github/PULL_REQUEST_TEMPLATE.md

@@ -26,7 +26,7 @@ Brief description of changes.
 - [ ] All new and existing tests pass
 - [ ] My commits follow conventional commit format
 - [ ] I have **NOT** edited `CHANGELOG.md` (it is auto-generated by git_ops)
-- [ ] I have **NOT** directly edited files in `priv/llm_db/providers/` (they are generated by `mix llm_db.build`)
+- [ ] I have **NOT** directly edited generated snapshot/history artifacts unless this PR intentionally regenerates them (`priv/llm_db/snapshot.json`, `priv/llm_db/history/**`)
 
 ## Related Issues
 

.github/workflows/README.md

@@ -24,43 +24,38 @@ Runs on every push and pull request to ensure code quality.
 - OTP versions: 25, 26
 - Excludes: Elixir 1.14 with OTP 26 (compatibility)
 
-### 2. Refresh Upstream Metadata (`build-metadata.yml`)
+### 2. Publish Snapshot Catalog (`build-metadata.yml`)
 
-Automatically pulls latest LLM model metadata from upstream sources and creates PRs for review.
+Automatically pulls latest upstream metadata, publishes a content-addressed snapshot,
+and rebuilds the published history bundle from the snapshot store.
 
 **Triggers:**
 - **Schedule**: Every Monday at 00:00 UTC
 - **Manual**: Via workflow_dispatch in GitHub Actions UI
 
 **Jobs:**
 1. Pull latest metadata using `mix llm_db.pull`
-2. Regenerate metadata using `mix llm_db.build`
-3. Reset `metadata-update` from `origin/main`
-4. If non-history metadata changes are detected:
-   - Commit metadata artifacts first
-   - Run `mix llm_db.history.sync --to HEAD`
-   - Commit history artifacts second
-   - Create or update the PR with `gh pr create` / `gh pr edit`
+2. Publish the current canonical snapshot using `mix llm_db.snapshot.publish`
+3. Rebuild and publish `history.tar.gz` from the published snapshot chain using `mix llm_db.history.rebuild --publish`
+4. Validate the packaged snapshot with `mix llm_db.build --check --install`
+5. Run the test suite against the resulting packaged snapshot
 
 **Output:**
-- Pull request with metadata changes for human review
-- Summary includes provider/model statistics and changed files
-- Metadata update PRs now contain two commits
-- Metadata update PRs must be merged with a merge commit, not squash-merged or rebase-merged
+- Updated GitHub Releases snapshot assets
+- Updated `catalog-index` assets: `latest.json`, `snapshot-index.json`, `history.tar.gz`, and `history-meta.json`
 
-### 3. Publish Release (`publish-release.yml`)
+### 3. Publish Release (`release.yml`)
 
-Automatically publishes new Hex.pm releases when metadata updates are merged.
+Automatically publishes new Hex.pm releases from the latest published snapshot.
 
 **Triggers:**
 - Push to `main` branch
-- Only when `priv/llm_db/snapshot.json` changes
-- Only from metadata update merges
+- Release workflow fetches the latest published snapshot and packages it into `priv/llm_db/snapshot.json`
 
 **Jobs:**
-1. Verify trigger is from metadata update merge
+1. Fetch the latest published snapshot into `priv/llm_db/snapshot.json`
 2. Prepare release using `mix llm_db.release prepare`
-   - Determines version from snapshot timestamp (YYYY.MM.DD format)
+   - Determines version from the packaged snapshot timestamp (YYYY.MM.DD format)
    - Updates `mix.exs` version
 3. Run tests to ensure quality
 4. Build Hex package
@@ -123,40 +118,26 @@ Cron examples:
 - `'0 0 * * 0'` - Weekly on Sunday
 - `'0 0 1 * *'` - Monthly on the 1st
 
-#### PR Reviewers
-
-To auto-assign reviewers to metadata update PRs, modify the PR creation step in `build-metadata.yml`:
-
-```bash
-gh pr create \
-  --base main \
-  --head "metadata-update" \
-  --title "Update model metadata - $(date +%Y-%m-%d)" \
-  --body-file "$SUMMARY_FILE" \
-  --label "metadata-update" \
-  --label "automated" \
-  --reviewer "username1,username2"  # Add this line
-```
-
 ## Manual Operations
 
-### Manually Trigger Metadata Update
+### Manually Trigger Snapshot Publish
 
 1. Go to Actions tab in GitHub
-2. Select "Refresh Upstream Metadata" workflow
+2. Select "Publish Snapshot Catalog" workflow
 3. Click "Run workflow"
 4. Select branch (usually `main`)
 5. Click "Run workflow"
 
 ### Manually Create a Release
 
-Releases are automatically triggered when metadata updates merge to main. To manually release:
+Releases package the latest published snapshot. To manually release:
 
-1. Ensure snapshot is updated: `mix llm_db.pull`
-2. Prepare release: `mix llm_db.release prepare`
-3. Review version in `mix.exs`
-4. Commit and push to main
-5. Workflow will detect snapshot change and publish
+1. Ensure the latest snapshot has been published: `mix llm_db.snapshot.publish`
+2. Rebuild the published history bundle if needed: `mix llm_db.history.rebuild --publish`
+3. Prepare release: `mix llm_db.release prepare`
+4. Review version in `mix.exs`
+5. Commit and push to main
+6. Workflow will fetch the latest published snapshot and publish the package
 
 ## Workflow Scripts
 
@@ -166,8 +147,8 @@ Helper scripts in `.github/workflows/scripts/`:
 
 Generates PR description for metadata updates with:
 - Provider and model counts
+- Snapshot publication details
 - Generated timestamp
-- File diff statistics
 - Review checklist
 
 ### `generate_release_notes.sh`

.github/workflows/build-metadata.yml

@@ -1,4 +1,4 @@
-name: Refresh Upstream Metadata
+name: Publish Snapshot Catalog
 
 on:
   schedule:
@@ -9,13 +9,15 @@ permissions:
   contents: write
 
 jobs:
-  update-metadata:
-    name: Pull and update LLM model metadata
+  publish-snapshot:
+    name: Pull upstream metadata and publish snapshot
     runs-on: ubuntu-latest
     timeout-minutes: 30
     concurrency:
       group: llm-metadata
       cancel-in-progress: false
+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
     steps:
       - name: Checkout main
@@ -64,62 +66,26 @@ jobs:
           set -euo pipefail
           echo "Pulling latest LLM model metadata..."
           mix llm_db.pull
-          mix llm_db.build
+          mix llm_db.snapshot.publish
 
-      - name: Commit metadata artifacts
-        id: metadata_commit
+      - name: Install packaged snapshot for validation
         run: |
           set -euo pipefail
+          mix llm_db.build --install
 
-          git add -A -- priv/llm_db lib/llm_db/generated
-          git restore --staged -- priv/llm_db/history
-
-          if git diff --cached --quiet; then
-            echo "created=false" >> "$GITHUB_OUTPUT"
-            echo "No non-history metadata changes detected."
-            exit 0
-          fi
-
-          git commit -m "chore: refresh model metadata"
-          echo "created=true" >> "$GITHUB_OUTPUT"
-
-      - name: Sync history to committed metadata
-        run: |
-          set -euo pipefail
-          mix llm_db.history.sync --to HEAD
-
-      - name: Commit synced history
-        id: history_commit
+      - name: Rebuild and publish history bundle
         run: |
           set -euo pipefail
+          mix llm_db.history.rebuild --publish
 
-          git add -A -- priv/llm_db/history
-
-          if git diff --cached --quiet; then
-            echo "created=false" >> "$GITHUB_OUTPUT"
-            echo "No history changes detected."
-            exit 0
-          fi
-
-          git commit -m "chore: sync model history"
-          echo "created=true" >> "$GITHUB_OUTPUT"
-
-      - name: Validate refreshed catalog
-        if: steps.metadata_commit.outputs.created == 'true' || steps.history_commit.outputs.created == 'true'
+      - name: Validate published catalog
         run: |
           set -euo pipefail
-          mix llm_db.build --check
-          mix llm_db.history.check
+          mix llm_db.build --check --install
+          mix llm_db.history.check --allow-missing
           mix test
 
-      - name: Push refreshed catalog to main
-        if: steps.metadata_commit.outputs.created == 'true' || steps.history_commit.outputs.created == 'true'
-        run: |
-          set -euo pipefail
-          git push origin HEAD:main
-
-      - name: Report no changes
-        if: steps.metadata_commit.outputs.created != 'true' && steps.history_commit.outputs.created != 'true'
+      - name: Report success
         run: |
-          echo "No metadata or history changes detected."
-          echo "### No Changes Detected" >> "$GITHUB_STEP_SUMMARY"
+          echo "### Snapshot Publish Complete" >> "$GITHUB_STEP_SUMMARY"
+          echo "- Snapshot release and catalog index updated" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/ci.yml

@@ -95,6 +95,6 @@ jobs:
           otp-version: "27"
           elixir-version: "1.18"
       - run: mix deps.get
-      - run: mix llm_db.build --check
+      - run: mix llm_db.build --check --install
       - name: Check history drift
-        run: mix llm_db.history.check
+        run: mix llm_db.history.check --allow-missing

.github/workflows/release.yml

@@ -66,6 +66,11 @@ jobs:
           mix deps.get
           MIX_ENV=dev mix deps.compile git_ops
 
+      - name: Fetch latest published snapshot
+        run: |
+          set -euo pipefail
+          mix llm_db.snapshot.fetch --ref latest --install
+
       - name: Run tests
         if: ${{ inputs.skip_tests != true }}
         run: |
@@ -125,6 +130,7 @@ jobs:
           fi
           echo "- Package: llm_db" >> "$GITHUB_STEP_SUMMARY"
           echo "- Version: ${{ steps.version.outputs.version }}" >> "$GITHUB_STEP_SUMMARY"
+          echo "- Snapshot: priv/llm_db/snapshot.json" >> "$GITHUB_STEP_SUMMARY"
 
       - name: Create GitHub release
         if: ${{ inputs.dry_run != true }}

AGENTS.md

@@ -10,10 +10,14 @@
 - **Format code**: `mix format`
 - **Compile**: `mix compile`
 - **Quality check**: `mix quality` (format, compile warnings, dialyzer, credo)
-- **Update model data**: `mix llm_db.pull` (fetches from configured remote sources and regenerates snapshot)
-- **Backfill model history (one-time)**: `mix llm_db.history.backfill --force` (generates `priv/llm_db/history/events/*.ndjson` from git snapshot history)
-- **Sync model history (incremental)**: `mix llm_db.history.sync`
-- **Check history drift**: `mix llm_db.history.check`
+- **Build local snapshot artifact**: `mix llm_db.build --install`
+- **Pull upstream metadata**: `mix llm_db.pull`
+- **Publish snapshot to GitHub Releases**: `mix llm_db.snapshot.publish`
+- **Fetch published snapshot locally**: `mix llm_db.snapshot.fetch --ref latest --install`
+- **Migrate reachable Git history (one-time)**: `mix llm_db.history.migrate_git --force`
+- **Rebuild/publish history bundle**: `mix llm_db.history.rebuild --publish`
+- **Sync published history bundle**: `mix llm_db.history.sync`
+- **Check published history drift**: `mix llm_db.history.check --allow-missing`
 - **Dependencies**: `mix deps.get`
 - **Release**: `mix llm_db.version && mix git_ops.release && git push && git push --tags` (bumps to date-based version, updates CHANGELOG, tags, and pushes)
 
@@ -61,7 +65,7 @@ config :llm_db,
 - **Type**: Elixir library providing fast, persistent_term-backed LLM model metadata catalog
 - **Core modules**: `LLMDB` (main API), `LLMDB.Engine` (ETL pipeline), `LLMDB.Store` (persistent_term storage)
 - **Data structures**: `LLMDB.Provider`, `LLMDB.Model` with Zoi validation schemas in `lib/llm_db/schema/`
-- **Storage**: O(1) lock-free queries via `:persistent_term`, snapshot in `priv/llm_db/snapshot.json`
+- **Storage**: O(1) lock-free queries via `:persistent_term`, packaged catalog snapshot in `priv/llm_db/snapshot.json`
 - **ETL Pipeline**: Ingest → Normalize → Validate → Merge → Enrich → Filter → Index + Publish (7 stages)
 - **Startup**: Catalog automatically loads on application start via `LLMDB.Application` (no manual `load()` needed in IEx or runtime)
 

README.md

@@ -253,23 +253,27 @@ Snapshot is shipped with the library. To rebuild with fresh data:
 # Fetch upstream data (optional)
 mix llm_db.pull
 
-# Run ETL and write snapshot.json
+# Build canonical snapshot artifacts
 mix llm_db.build
+
+# Install the packaged snapshot for local runtime/package validation
+mix llm_db.build --install
 ```
 
-To generate historical change events from committed snapshot history (initial setup):
+To migrate legacy Git-tracked metadata history into the snapshot store (one-time maintainer task):
 
 ```bash
-mix llm_db.history.backfill --force
+mix llm_db.history.migrate_git
 ```
 
-This writes append-only NDJSON history artifacts under `priv/llm_db/history/`:
-`priv/llm_db/history/events/YYYY.ndjson`,
-`priv/llm_db/history/snapshots.ndjson`, and `priv/llm_db/history/meta.json`.
+This writes snapshot-based history artifacts under `priv/llm_db/history/` and
+materializes immutable historical snapshots under `_build/llm_db/snapshot_store/snapshots/`.
 
-For daily/incremental maintenance:
+For daily publication and local history maintenance:
 
 ```bash
+mix llm_db.snapshot.publish
+mix llm_db.history.rebuild --publish
 mix llm_db.history.sync
 mix llm_db.history.check
 ```
@@ -286,8 +290,8 @@ add optional lineage overrides in `priv/llm_db/history/lineage_overrides.json`:
 }
 ```
 
-History artifacts are intended for Git/path dependencies and local repo usage.
-Hex packages do not guarantee inclusion of `priv/llm_db/history/**`.
+History artifacts remain optional local/published data.
+Hex packages still only ship `priv/llm_db/snapshot.json`.
 
 See the [Sources & Engine](guides/sources-and-engine.md) guide for details.
 

config/config.exs

@@ -17,12 +17,14 @@ config :llm_db,
   # Cache directory for remote sources
   models_dev_cache_dir: "priv/llm_db/upstream",
   openrouter_cache_dir: "priv/llm_db/upstream",
+  llmfit_cache_dir: "priv/llm_db/upstream",
   upstream_cache_dir: "priv/llm_db/upstream",
   openai_cache_dir: "priv/llm_db/remote",
   anthropic_cache_dir: "priv/llm_db/remote",
   google_cache_dir: "priv/llm_db/remote",
   xai_cache_dir: "priv/llm_db/remote",
   zenmux_cache_dir: "priv/llm_db/remote",
+  llmfit_enrichment: true,
   azure_foundry_cache_dir: "priv/llm_db/remote"
 
 if Mix.env() == :dev do
@@ -46,7 +48,7 @@ if Mix.env() == :dev do
       pre_commit: [
         tasks: [
           {:mix_task, :format, ["--check-formatted"]},
-          {:cmd, "mix llm_db.build --check"}
+          {:cmd, "mix llm_db.build --check --install"}
         ]
       ],
       pre_push: [

config/test.exs

@@ -4,7 +4,11 @@ import Config
 config :llm_db,
   compile_embed: false,
   integrity_policy: :warn,
+  skip_packaged_load: true,
+  snapshot_path: "priv/llm_db/snapshot.json",
   # Use test-specific cache directory to avoid polluting production cache
   models_dev_cache_dir: "tmp/test/upstream",
   openrouter_cache_dir: "tmp/test/upstream",
+  llmfit_cache_dir: "tmp/test/upstream",
+  llmfit_enrichment: true,
   upstream_cache_dir: "tmp/test/upstream"