Skip to content
This repository was archived by the owner on Jul 4, 2026. It is now read-only.

CHANGELOG.md

Latest commit

 

History

History
256 lines (206 loc) · 18.4 KB

CHANGELOG.md

File metadata and controls

256 lines (206 loc) · 18.4 KB
Raw

Changelog

All notable changes to skillscan-security are documented here.

Format follows Keep a Changelog. Versioning follows Semantic Versioning.

[1.0.0] — 2026-04-20

Added

  • skillscan delta — security-focused skill version comparison with suppression support
  • skillscan alert — regression detection comparing scan reports with known-regression filtering
  • skillscan watch — real-time file monitoring with automatic re-scan
  • --summary flag for multi-skill directory scan roll-ups
  • --exclude patterns with sensible defaults (tests/, node_modules/, etc.)
  • --badge-out, --badge-dir, --coverage-badge-out for shields.io badge generation
  • Frontmatter includes/excludes for skill boundary declaration
  • 5 detection integrations: YARA (--yara-rules), VirusTotal (--virustotal), Semgrep (--semgrep-rules), OSV.dev live (--live-vuln-check), Snyk/Dependabot (--vuln-report)
  • --require-model flag for CI gates mandating ML detection
  • Interactive model download prompt in TTY mode
  • -h help shorthand and --version flag
  • Integration and boundary info in scan report provenance
  • skillscan badge combine for compound scan+lint badges

Changed

  • JSON output: finding id field renamed to rule_id
  • JSON output: mitigation field removed from findings
  • Intel lookup uses suffix matching (not substring)
  • Rules decoupled to separate kurtpayne/skillscan-rules repo
  • Detection section: 3 pillars (rule-based, ML classifier, integrations)
  • Build backend migrated from setuptools to hatchling
  • ML model: Qwen2.5-1.5B GGUF replaces DeBERTa-v3 ONNX

Removed

  • [ml-onnx] extra (deprecated; use [ml] with llama-cpp-python)
  • --rulepack-channel flag (non-functional, removed)
  • Showcase examples directory (migrated to inline test_input in rules)

Fixed

  • 26 bugs from comprehensive QA pass
  • 99+ documentation mismatches corrected
  • All phantom CLI flags and commands removed from docs

[Unreleased]

Added

  • C-2 decontamination (2026-03-29): removed 204 training files that were duplicated in held_out_eval/. Post-audit v11 true Macro F1 on 259 clean examples: 0.555 (injection recall 19.6%). v11 reported F1 of 0.926 was inflated by training/eval leakage introduced during v11 prep (commit cf855ab8). Full disclosure in EVAL_RESULTS.md.
  • eval-integrity.yml CI gate: fails any push to main if any held_out_eval/ file hash appears in training_corpus/. Prevents future training/eval leakage.
  • 86 targeted injection training examples across 8 zero-recall archetypes: organic real-world malicious (22 FN), enterprise auth/redirect (10 FN), organic PI (8 FN), MCP poisoning (8 FN), supply chain (6 FN), social engineering (8 FN), jailbreaks (18 FN), evasion (7 FN). Added to training_corpus/malicious/ with zero eval overlap.
  • v12 training launched (2026-03-29): decontaminated corpus (19,407 files, 0 eval overlap), injection class weight cap raised to 12×, warmup ratio 0.08, label smoothing 0.05, LR 1.5e-5, 6 epochs, early stopping patience 3. Training on Modal A10G.
  • 4 new eval archetypes in held_out_eval/: ClawHub ranking exfil supply chain (sup023), NemoClaw/AMOS README dropper (mal055), calendar event indirect injection (pi86), hallucination squatting (sup024).
  • PINJ-ML-LARGE-FILE (13e): LOW advisory finding emitted when --ml-detect is used on a file exceeding 200 lines or 8,000 characters. The model was trained on short skill files; very large files are chunked and per-chunk scores may miss distributed intent.
  • PINJ-ML-NO-MODEL (M10.5 belt-and-suspenders): programmatic API path now emits a structured finding when ML extras are installed but the LoRA adapter weights have not been downloaded, rather than falling through to base-model inference.
  • holdout-eval.yml — weekly CI workflow (M10.6) that runs all 451 held-out eval files against the current scanner on every push to main, Sundays at 07:00 UTC, and on demand. Computes macro F1 + FPR; fails if F1 < 0.92 (configurable via workflow_dispatch input). Posts per-file FN/FP lists and a metrics table to step summary. Closes the organic eval regression gate loop.
  • .github/dependabot.yml — weekly pip + GitHub Actions dependency updates
  • .github/PULL_REQUEST_TEMPLATE.md — contributor checklist covering ruff, mypy, pytest, rule tests, showcase, website sync, and no-debug-code requirements
  • CHANGELOG.md and RELEASING.md — release process documentation
  • docs/AUDIT_2026-03.md — comprehensive codebase audit report (correctness, security, quality, CI, docs)
  • timeout-minutes: 20 on CI test job to prevent hung runners

Changed

  • strict policy: added prompt_injection: 2 and supply_chain: 2 category weights. HIGH-severity rules in these categories now score 70 (was 35), correctly reaching the BLOCK threshold without co-occurring findings. Fixes FN regressions PINJ-015 and SUP-022.
  • MAL-001 pattern tightened to require a pipe or redirect ([|>]) between the download command and the shell invocation. Eliminates false positives where curl and a shell name appear in the same documentation sentence (e.g., zsh).
  • CHN-002 window_lines reduced from 40 to 5. API integration skills that read an auth token and call their own endpoint on the same line no longer trigger the secret-access + network-exfil chain.
  • SKILLSCAN_NO_USER_RULES=1 now set in CI env so user-local ~/.skillscan/rules/ never influences test results
  • load_builtin_rulepack() emits INFO-level provenance log (version, rule counts, channel) on every cold load
  • User-local rulepack files older than their bundled counterpart are now skipped with a WARNING (version-gate)
  • tests/test_custom_rules.py custom_rules_dir fixture clears SKILLSCAN_NO_USER_RULES so custom-rules tests pass in CI
  • Pattern-update skill step 8 now has a mandatory pytest -q gate that must exit 0 before gh pr create

Fixed

  • test_se_sem_001_semantic_classifier_fires: fixture text updated from "access token" to text containing password and credentials. The previous fixture did not stem to any credential_roots so the semantic classifier correctly returned None and the static SE-001 rule fired instead of SE-SEM-001.

[0.7.0] — 2026-03-26

This release represents the largest single jump in SkillScan's history — ten milestones, 237 commits, and a complete rearchitecture of the ML detection layer, CLI surface, and threat intel pipeline since v0.3.1.

ML Model (M7 / M7.5)

  • Production ML model: DeBERTa-v3-base + LoRA adapter retrained from 111 examples (v0.3.1) to 18,161 examples across 5 training runs. Final metrics: F1 0.9752, FPR 1.89%, precision 0.9738, recall 0.9766 on 1,817-example held-out eval set.
  • Sliding-window chunking: SKILL.md files longer than 512 tokens are chunked with 128-token overlap; worst-chunk verdict is reported. Eliminates blind spots on large files.
  • ONNX FP32 export: model exported to FP32 ONNX (was INT8). Eliminates quantization-induced false negatives on borderline examples.
  • Attack-type hints (M10.8): PINJ-ML-* findings now include attack_type field with human-readable label (e.g., prompt_override, data_exfil, tool_poisoning) derived from the top softmax class.
  • Enterprise benign corpus (M7.5): 2,400 enterprise SKILL.md files added to training corpus (Salesforce, SAP, ServiceNow, Workday, Jira, Confluence, GitHub, GitLab, Slack, Teams, Zoom, Notion, Asana, Linear, Figma, Stripe, Twilio, SendGrid, HubSpot, Zendesk, Datadog, PagerDuty, Splunk, Elastic). Reduces enterprise false-positive rate.
  • MODEL_CARD.md: published to HuggingFace Hub (kurtpayne/skillscan-deberta-adapter) with full architecture, training data, eval history, known FN archetypes, and usage instructions.

CLI Consolidation (M10.7)

  • skillscan update [--no-model]: single command to refresh rules, intel feeds, and ML model. Replaces rule sync, intel sync, and model sync.
  • model install [--repo] [--force]: replaces model sync. Accepts --repo for custom HuggingFace repos.
  • intel add --url <url>: URL-based intel feed registration. Replaces file-based intel add <path>.
  • intel lookup <indicator>: query the local IOC DB for a domain, IP, or CIDR.
  • policy list / policy show-default: list and inspect built-in policy profiles.
  • rule test <file>: test a custom rule file against the bundled showcase corpus.
  • benchmark --verbose: show per-file verdicts during benchmark runs.
  • scan --baseline <file>: compare current scan against a saved baseline; report new/resolved findings.
  • scan --no-provenance: suppress the provenance meta block in JSON output.
  • scan --no-suppress: ignore all suppression files for this run.
  • Provenance meta block: every --format json scan now includes meta with scanner version, rules SHA, model version, policy profile, and scanned_at timestamp.
  • Staleness warning: skillscan scan emits a warning if rules or intel DB are older than 7 days.
  • Removed commands (8): diff, skill-diff, rule sync, intel sync, intel rebuild, intel add <file>, model sync, suppress add.

Policy Profiles (M10.7)

  • 5 built-in profiles: strict (default), ci, balanced, permissive, enterprise, observe. Stored as inspectable YAML in src/skillscan/data/policies/.
  • observe profile: all findings downgraded to WARN; exit code always 0. For initial rollout without blocking CI.

Intel & Vuln DB (M5)

  • IOC DB: expanded to 5,500 entries (3,951 domains, 8 IPs, 1,538 CIDRs, 3 URLs) via Hagezi DoH bypass domains (fourth bundled feed).
  • Vuln DB: covers 23 Python packages and 4 npm packages with 111 vulnerable versions from OSV.dev.
  • CI gate tests: 5 tests prevent silent regression below 5,000-entry and 20-package thresholds.

Rule Expansion (M6 / M9 / M10.9)

  • 158 total rules (was 77 at v0.3.1): +81 rules across MAL, SUP, PINJ, EXF, SE, INJ, CHN, CAP categories.
  • New rules include: PINJ-011/013/014, EXF-018/019, SE-002, SUP-018 (M6 P2); MAL-042 through MAL-049; SUP-015 through SUP-020; PINJ-015 (pattern updates).
  • SE category: social engineering rules (SE-001, SE-002, SE-SEM-001) for credential harvest and authority impersonation patterns.
  • Lint rule expansion (M10.9): skillscan-lint extended with additional QL-* and GR-* lint rules.

Skill Graph Analysis (M8)

  • PSV-004: unknown frontmatter keys detector — flags undocumented keys in SKILL.md frontmatter.
  • GR-007: cycle detection in skill invocation graphs — flags circular dependencies.
  • PSV rule wiring: all PSV rules now correctly wired to the skill graph analysis layer.

Missing-Model UX (M10.5)

  • When --ml-detect is passed and the model is not installed, SkillScan now emits a guided download prompt with the exact skillscan model install command instead of a silent skip or cryptic error.

Bug Fixes

  • Chain rule proximity window (_CHAIN_WINDOW_LINES = 40): chain rules now only fire when all required actions appear within 40 lines of each other. Eliminates false positives on large SKILL.md files.
  • Multilang language filter: JS/TS/Ruby/Go/Rust rules now only apply to files with matching extensions. Eliminates cross-language false positives.
  • AV-ADVISORY (LOW): emitted when script files are present but --clamav was not requested.

Documentation

  • docs/CLI_REFERENCE.md: complete M10.7 CLI reference.
  • docs/custom-rules-format.md, docs/custom-intel-format.md, docs/custom-policy-format.md: authoring guides.
  • docs/benchmark-guide.md: benchmark workflow and interpretation guide.
  • docs/suppression-format.md: suppression file format reference.
  • CONTRIBUTING.md, SECURITY.md: contribution guide and responsible disclosure policy.
  • .github/ISSUE_TEMPLATE/: 4 issue templates (false-positive, false-negative, bug-report, feature-request).

Infrastructure

  • Docker image: kurtpayne/skillscan:latest and kurtpayne/skillscan:v2026.03.26 published to Docker Hub. Includes ClamAV.
  • Skill Fuzzer (tools/skill-fuzzer/): LLM-powered adversarial SKILL.md variant generator. Installable as pip install skillscan-security[fuzzer].

0.3.2 — 2026-03-18

Added

  • SE-001: social engineering credential harvest static rule (HIGH) — detects instructions that solicit API tokens, passwords, or session cookies from users.
  • SE-SEM-001: offline semantic social engineering classifier — stem-and-score classifier for credential solicitation patterns.
  • Intel DB seeding: IOC DB expanded from 11 to 1,475 entries (163 domains, 1,310 IPs, 2 CIDRs) from URLhaus, Feodo Tracker, and Spamhaus DROP.
  • Vuln DB seeding: vuln DB expanded from 4 to 27 packages with 111 versions from OSV.dev.
  • docs/DETECTION_MODEL.md: comprehensive detection architecture documentation covering all 8 layers.
  • Showcase example 20: renamed from 20_ai_semantic_risk to 20_social_engineering_credential_harvest.

Removed

  • AI assist layer (--ai-assist, --ai-provider, --ai-model, --ai-base-url, --ai-required): removed entirely. SkillScan is repositioned as a free, offline, privacy-first pre-filter. For nuanced semantic analysis, use Invariant Analyzer or Lakera Guard after SkillScan passes.

0.3.1 — 2026-03-17

Added

  • Skill graph analysis (skillscan scan --graph): structural analysis of skill invocation graphs. Three new rules:
    • PINJ-GRAPH-001 — skill loads a remote .md file at runtime (dead-drop instruction injection, HIGH)
    • PINJ-GRAPH-002 — skill grants Bash/Computer/Shell tool without a declared purpose section (MEDIUM)
    • PINJ-GRAPH-003 — skill instructs agent to write memory files (SOUL.md, MEMORY.md, AGENTS.md, .claude/settings.json) (CRITICAL)
  • ML-based prompt injection detection (skillscan scan --ml-detect): ONNX INT8 DeBERTa adapter trained on 111-example corpus. Adds PINJ-ML-* findings.
  • skillscan model commands: model sync (download/update ONNX model), model status (show age, staleness). Emits PINJ-ML-STALE at 30 days, PINJ-ML-UNAVAIL when not installed.
  • skillscan rule commands: rule sync (pull latest rulepack), rule status (show version, age).
  • Reusable GitHub Actions workflow (.github/workflows/skillscan-reusable.yml): call from any repo with uses: kurtpayne/skillscan-security/.github/workflows/skillscan-reusable.yml@main. Outputs SARIF to GitHub Security tab.
  • Pre-commit hook (.pre-commit-hooks.yaml): skillscan-security and skillscan-lint hooks for local dev gates.
  • VS Code extension scaffold (editors/vscode/): TypeScript extension with status bar, inline diagnostics, and marketplace publish workflow.
  • Homebrew formula (packaging/homebrew/skillscan-security.rb).
  • Signature-as-data architecture: rules, IOCs, and ML model are versioned data artifacts separate from the scanner binary. Auto-updated via corpus-sync workflow.
  • MAL-029: Solana RPC blockchain C2 resolution (GlassWorm Wave 5 — getSignaturesForAddress dead-drop technique).
  • CORPUS_CARD.md: reproducibility metadata for the ML training corpus (SHA, example counts, fine-tune history).

Changed

  • PINJ-ML-UNAVAIL now emits a finding (was a silent skip) when ML model is not installed and --ml-detect is passed.
  • pyproject.toml: added PyPI classifiers, keywords, project URLs, and Docker Hub / GitHub links.
  • README: corrected badge URLs, added PyPI + Docker Hub badges, added skillscan.sh and Related section.

Fixed

  • eval_strategy parameter name for transformers>=4.46 compatibility in fine-tune script.
  • Corpus manifest sha256_index race condition between corpus-sync and fine-tune jobs (added push_options: --force-with-lease).

0.3.0 — 2026-03-10

Added

  • Signature-as-data foundation: src/skillscan/data/ directory for versioned rules, IOCs, and vulnerability DB.
  • Corpus pipeline: corpus/ directory with 111 labeled examples (54 benign / 57 injection), corpus/manifest.json, automated scraping via corpus-sync workflow.
  • skillscan corpus commands: corpus status, corpus add, corpus sync.
  • MAL-025 through MAL-028: GlassWorm Wave 4 patterns (npm typosquatting, MCP tool poisoning, cross-server namespace collision, rug-pull mutation).
  • CVE-2026-4270: @awslabs/aws-api-mcp-server path traversal (CVSS 6.8, fixed in 1.3.9).
  • IOC additions: 6 GlassWorm/PylangGhost IPs, 3 domains.
  • Chain rules: 14 multi-step attack chain detectors.
  • Action patterns: 19 behavioral action pattern detectors.
  • Capability patterns: 3 capability escalation detectors.

Changed

  • Rule count: 64 static + 14 chain + 19 action + 3 capability = 100 total rules.
  • Rules version: 2026.03.17.3.

0.2.3 — 2026-02-28

Added

  • Docker Hub release pipeline (release-docker.yml).
  • PyPI trusted publisher configuration (release-pypi.yml) with SBOM and cosign signing.
  • SARIF output format for GitHub Security tab integration.
  • Offline semantic prompt injection classifier (PINJ-SEM-001).
  • Policy profiles: strict, balanced, permissive.
  • URL scanning: follow links up to configurable depth.

Fixed

  • Archive extraction handling for nested zip/tar.gz artifacts.
  • Unicode normalization edge cases in instruction hardening pipeline.

0.2.0 — 2026-02-10

Added

  • Binary artifact classification (executables, libraries, bytecode, blobs).
  • IOC extraction with local intel matching.
  • Dependency vulnerability and unpinned-version checks.
  • skillscan intel command group for managing local intel sources.
  • Pretty terminal output with color-coded verdicts.
  • JSON report format with structured findings.

0.1.0 — 2026-01-20

Added

  • Initial release.
  • Static analysis of SKILL.md files: malware patterns, instruction abuse, Unicode attacks.
  • Archive-safe extraction pipeline.
  • Verdicts: allow, warn, block.
  • Policy enforcement via YAML policy files.
  • skillscan scan, skillscan explain, skillscan policy commands.
  • 29 initial detection rules.