Skip to content

Latest commit

 

History

History
605 lines (464 loc) · 23 KB

CLAUDE.md

File metadata and controls

605 lines (464 loc) · 23 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

rtk (Rust Token Killer) is a high-performance CLI proxy that minimizes LLM token consumption by filtering and compressing command outputs. It achieves 60-90% token savings on common development operations through smart filtering, grouping, truncation, and deduplication.

This is a fork with critical fixes for git argument parsing and modern JavaScript stack support (pnpm, vitest, Next.js, TypeScript, Playwright, Prisma).

⚠️ Name Collision Warning

Two different "rtk" projects exist:

  • ✅ This project: Rust Token Killer (rtk-ai/rtk)
  • ❌ reachingforthejack/rtk: Rust Type Kit (DIFFERENT - generates Rust types)

Verify correct installation:

rtk --version  # Should show "rtk 0.28.0" (or newer)
rtk gain       # Should show token savings stats (NOT "command not found")

If rtk gain fails, you have the wrong package installed.

Development Commands

Note: If rtk is installed, prefer rtk <cmd> over raw commands for token-optimized output. All commands work with passthrough support even for subcommands rtk doesn't specifically handle.

Build & Run

# Development build
cargo build                   # raw
rtk cargo build               # preferred (token-optimized)

# Release build (optimized)
cargo build --release
rtk cargo build --release

# Run directly
cargo run -- <command>

# Install locally
cargo install --path .

Testing

# Run all tests
cargo test                    # raw
rtk cargo test                # preferred (token-optimized)

# Run specific test
cargo test <test_name>
rtk cargo test <test_name>

# Run tests with output
cargo test -- --nocapture
rtk cargo test -- --nocapture

# Run tests in specific module
cargo test <module_name>::
rtk cargo test <module_name>::

Linting & Quality

# Check without building
cargo check                   # raw
rtk cargo check               # preferred (token-optimized)

# Format code
cargo fmt                     # passthrough (0% savings, but works)

# Run clippy lints
cargo clippy                  # raw
rtk cargo clippy              # preferred (token-optimized)

# Check all targets
cargo clippy --all-targets
rtk cargo clippy --all-targets

Package Building

# Build DEB package (Linux)
cargo install cargo-deb
cargo deb

# Build RPM package (Fedora/RHEL)
cargo install cargo-generate-rpm
cargo build --release
cargo generate-rpm

Architecture

Core Design Pattern

rtk uses a command proxy architecture with specialized modules for each output type:

main.rs (CLI entry)
  → Clap command parsing
  → Route to specialized modules
  → tracking.rs (SQLite) records token savings

Key Architectural Components

1. Command Modules (src/*_cmd.rs, src/git.rs, src/container.rs)

  • Each module handles a specific command type (git, grep, etc.)
  • Responsible for executing underlying commands and transforming output
  • Implement token-optimized formatting strategies

2. Core Filtering (src/filter.rs)

  • Language-aware code filtering (Rust, Python, JavaScript, etc.)
  • Filter levels: none, minimal, aggressive
  • Strips comments, whitespace, and function bodies (aggressive mode)
  • Used by read and smart commands

3. Token Tracking (src/tracking.rs)

  • SQLite-based persistent storage (~/.local/share/rtk/tracking.db)
  • Records: original_cmd, rtk_cmd, input_tokens, output_tokens, savings_pct
  • 90-day retention policy with automatic cleanup
  • Powers the rtk gain analytics command
  • Configurable database path: Via RTK_DB_PATH env var or config.toml
    • Priority: env var > config file > default location

4. Configuration System (src/config.rs, src/init.rs)

  • Manages CLAUDE.md initialization (global vs local)
  • Reads ~/.config/rtk/config.toml for user preferences
  • rtk init command bootstraps LLM integration
  • New: tracking.database_path field for custom DB location

5. Tee Output Recovery (src/tee.rs)

  • Saves raw unfiltered output to ~/.local/share/rtk/tee/ on command failure
  • Prints one-line hint [full output: ~/.local/share/rtk/tee/...] so LLMs can read instead of re-run
  • Configurable via [tee] section in config.toml or env vars (RTK_TEE, RTK_TEE_DIR)
  • Default mode: failures only, skip outputs < 500 chars, 20 file rotation, 1MB cap
  • Silent error handling: tee failure never affects command output or exit code

6. Shared Utilities (src/utils.rs)

  • Common functions for command modules: truncate, strip_ansi, execute_command
  • Package manager auto-detection (pnpm/yarn/npm/npx)
  • Consistent error handling and output formatting
  • Used by all modern JavaScript/TypeScript tooling commands

Command Routing Flow

All commands follow this pattern:

main.rs:Commands enum
  → match statement routes to module
  → module::run() executes logic
  → tracking::track_command() records metrics
  → Result<()> propagates errors

Proxy Mode

Purpose: Execute commands without filtering but track usage for metrics.

Usage: rtk proxy <command> [args...]

Benefits:

  • Bypass RTK filtering: Workaround bugs or get full unfiltered output
  • Track usage metrics: Measure which commands Claude uses most (visible in rtk gain --history)
  • Guaranteed compatibility: Always works even if RTK doesn't implement the command
  • Prototyping: Test new commands before implementing optimized filtering

Examples:

# Full git log output (no truncation)
rtk proxy git log --oneline -20

# Raw npm output (no filtering)
rtk proxy npm install express

# Any command works
rtk proxy curl https://api.example.com/data

# Tracking shows 0% savings (expected)
rtk gain --history | grep proxy

Tracking: All proxy commands appear in rtk gain --history with 0% savings (input = output) but preserve usage statistics.

Critical Implementation Details

Git Argument Handling (src/git.rs)

  • Uses trailing_var_arg = true + allow_hyphen_values = true to properly handle git flags
  • Auto-detects --merges flag to avoid conflicting with --no-merges injection
  • Propagates git exit codes for CI/CD reliability (PR #5 fix)

Output Filtering Strategy

  • Compact mode: Show only summary/failures
  • Full mode: Available with -v verbosity flags
  • Test output: Show only failures (90% token reduction)
  • Git operations: Ultra-compressed confirmations ("ok ✓")

Language Detection (src/filter.rs)

  • File extension-based with fallback heuristics
  • Supports Rust, Python, JS/TS, Java, Go, C/C++, etc.
  • Tokenization rules vary by language (comments, strings, blocks)

Module Responsibilities

Module Purpose Token Strategy
git.rs Git operations Stat summaries + compact diffs
grep_cmd.rs Code search Group by file, truncate lines
ls.rs Directory listing Tree format, aggregate counts
read.rs File reading Filter-level based stripping
runner.rs Command execution Stderr only (err), failures only (test)
log_cmd.rs Log parsing Deduplication with counts
json_cmd.rs JSON inspection Structure without values
lint_cmd.rs ESLint/Biome linting Group by rule, file summary (84% reduction)
tsc_cmd.rs TypeScript compiler Group by file/error code (83% reduction)
next_cmd.rs Next.js build/dev Route metrics, bundle stats only (87% reduction)
prettier_cmd.rs Format checking Files needing changes only (70% reduction)
playwright_cmd.rs E2E test results Failures only, grouped by suite (94% reduction)
prisma_cmd.rs Prisma CLI Strip ASCII art and verbose output (88% reduction)
gh_cmd.rs GitHub CLI Compact PR/issue/run views (26-87% reduction)
vitest_cmd.rs Vitest test runner Failures only with ANSI stripping (99.5% reduction)
pnpm_cmd.rs pnpm package manager Compact dependency trees (70-90% reduction)
ruff_cmd.rs Ruff linter/formatter JSON for check, text for format (80%+ reduction)
pytest_cmd.rs Pytest test runner State machine text parser (90%+ reduction)
mypy_cmd.rs Mypy type checker Group by file/error code (80% reduction)
pip_cmd.rs pip/uv package manager JSON parsing, auto-detect uv (70-85% reduction)
go_cmd.rs Go commands NDJSON for test, text for build/vet (80-90% reduction)
golangci_cmd.rs golangci-lint JSON parsing, group by rule (85% reduction)
tee.rs Full output recovery Save raw output to file on failure, print hint for LLM re-read
utils.rs Shared utilities Package manager detection, common formatting
discover/ Claude Code history analysis Scan JSONL sessions, classify commands, report missed savings

Performance Constraints

RTK has strict performance targets to maintain zero-overhead CLI experience:

Metric Target Verification Method
Startup time <10ms hyperfine 'rtk git status' 'git status'
Memory overhead <5MB resident /usr/bin/time -l rtk git status (macOS)
Token savings 60-90% Verify in tests with count_tokens() assertions
Binary size <5MB stripped ls -lh target/release/rtk

Performance regressions are release blockers - always benchmark before/after changes:

# Before changes
hyperfine 'rtk git log -10' --warmup 3 > /tmp/before.txt

# After changes
cargo build --release
hyperfine 'target/release/rtk git log -10' --warmup 3 > /tmp/after.txt

# Compare (should be <10ms)
diff /tmp/before.txt /tmp/after.txt

Why <10ms matters: Claude Code users expect CLI tools to be instant. Any perceptible delay (>10ms) breaks the developer flow. RTK achieves this through:

  • Zero async overhead: Single-threaded, no tokio runtime
  • Lazy regex compilation: Compile once with lazy_static!, reuse forever
  • Minimal allocations: Borrow over clone, in-place filtering
  • No user config: Zero file I/O on startup (config loaded on-demand)

Error Handling

RTK follows Rust best practices for error handling:

Rules:

  • anyhow::Result for CLI binary (RTK is an application, not a library)
  • ALWAYS use .context("description") with ? operator
  • NO unwrap() in production code (tests only - use expect("explanation") if needed)
  • Graceful degradation: If filter fails, fallback to raw command execution

Example:

use anyhow::{Context, Result};

pub fn filter_git_log(input: &str) -> Result<String> {
    let lines: Vec<_> = input
        .lines()
        .filter(|line| !line.is_empty())
        .collect();

    // ✅ RIGHT: Context on error
    let hash = extract_hash(lines[0])
        .context("Failed to extract commit hash from git log")?;

    // ❌ WRONG: No context
    let hash = extract_hash(lines[0])?;

    // ❌ WRONG: Panic in production
    let hash = extract_hash(lines[0]).unwrap();

    Ok(format!("Commit: {}", hash))
}

Fallback pattern (critical for all filters):

// ✅ RIGHT: Fallback to raw command if filter fails
pub fn execute_with_filter(cmd: &str, args: &[&str]) -> Result<()> {
    match get_filter(cmd) {
        Some(filter) => match filter.apply(cmd, args) {
            Ok(output) => println!("{}", output),
            Err(e) => {
                eprintln!("Filter failed: {}, falling back to raw", e);
                execute_raw(cmd, args)?;
            }
        },
        None => execute_raw(cmd, args)?,
    }
    Ok(())
}

// ❌ WRONG: Panic if no filter
pub fn execute_with_filter(cmd: &str, args: &[&str]) -> Result<()> {
    let filter = get_filter(cmd).expect("Filter must exist");
    filter.apply(cmd, args)?;
    Ok(())
}

Common Pitfalls

Don't add async dependencies (kills startup time)

  • RTK is single-threaded by design
  • Adding tokio/async-std adds ~5-10ms startup overhead
  • Use blocking I/O with fallback to raw command

Don't recompile regex at runtime (kills performance)

  • ❌ WRONG: let re = Regex::new(r"pattern").unwrap(); inside function
  • ✅ RIGHT: lazy_static! { static ref RE: Regex = Regex::new(r"pattern").unwrap(); }

Don't panic on filter failure (breaks user workflow)

  • Always fallback to raw command execution
  • Log error to stderr, execute original command unchanged

Don't assume command output format (breaks across versions)

  • Test with real fixtures from multiple versions
  • Use flexible regex patterns that tolerate format changes

Don't skip cross-platform testing (macOS ≠ Linux ≠ Windows)

  • Shell escaping differs: bash/zsh vs PowerShell
  • Path separators differ: / vs \
  • Line endings differ: LF vs CRLF

Don't break pipe compatibility (users expect Unix behavior)

  • rtk git status | grep modified must work
  • Preserve stdout/stderr separation
  • Respect exit codes (0 = success, non-zero = failure)

Fork-Specific Features

PR #5: Git Argument Parsing Fix (CRITICAL)

  • Problem: Git flags like --oneline, --cached were rejected
  • Solution: Fixed Clap parsing with proper trailing_var_arg configuration
  • Impact: All git commands now accept native git flags

PR #6: pnpm Support

  • New Commands: rtk pnpm list, rtk pnpm outdated, rtk pnpm install
  • Token Savings: 70-90% reduction on package manager operations
  • Security: Package name validation prevents command injection

PR #9: Modern JavaScript/TypeScript Tooling (2026-01-29)

  • New Commands: 6 commands for T3 Stack workflows
    • rtk lint: ESLint/Biome with grouped rule violations (84% reduction)
    • rtk tsc: TypeScript compiler errors grouped by file/code (83% reduction)
    • rtk next: Next.js build with route/bundle metrics (87% reduction)
    • rtk prettier: Format checker showing files needing changes (70% reduction)
    • rtk playwright: E2E test results showing failures only (94% reduction)
    • rtk prisma: Prisma CLI without ASCII art (88% reduction)
  • Shared Infrastructure: utils.rs module for package manager auto-detection
  • Features: Exit code preservation, error grouping, consistent formatting
  • Testing: Validated on a production T3 Stack project

Python & Go Support (2026-02-12)

  • Python Commands: 3 commands for Python development workflows
    • rtk ruff check/format: Ruff linter/formatter with JSON (check) and text (format) parsing (80%+ reduction)
    • rtk pytest: Pytest test runner with state machine text parser (90%+ reduction)
    • rtk pip list/outdated/install: pip package manager with auto-detect uv (70-85% reduction)
  • Go Commands: 4 commands via sub-enum for Go ecosystem
    • rtk go test: NDJSON line-by-line parser for interleaved events (90%+ reduction)
    • rtk go build: Text filter showing errors only (80% reduction)
    • rtk go vet: Text filter for issues (75% reduction)
    • rtk golangci-lint: JSON parsing grouped by rule (85% reduction)
  • Architecture: Standalone Python commands (mirror lint/prettier), Go sub-enum (mirror git/cargo)
  • Patterns: JSON for structured output (ruff check, golangci-lint, pip), NDJSON streaming (go test), text state machine (pytest), text filters (go build/vet, ruff format)

Testing Strategy

TDD Workflow (mandatory)

All code follows Red-Green-Refactor. See .claude/skills/rtk-tdd/ for the full workflow and Rust-idiomatic patterns. See .claude/skills/rtk-tdd/references/testing-patterns.md for RTK-specific patterns and untested module backlog.

Test Architecture

  • Unit tests: Embedded #[cfg(test)] mod tests in each module (105+ tests, 25+ files)
  • Smoke tests: scripts/test-all.sh (69 assertions on all commands)
  • Dominant pattern: raw string input -> filter function -> assert output contains/excludes

Pre-commit gate

cargo fmt --all --check && rtk cargo clippy --all-targets && rtk cargo test

Test commands

cargo test                    # All tests
cargo test filter::tests::    # Module-specific
cargo test -- --nocapture     # With stdout
bash scripts/test-all.sh      # Smoke tests (installed binary required)

Dependencies

Core dependencies (see Cargo.toml):

  • clap: CLI parsing with derive macros
  • anyhow: Error handling
  • rusqlite: SQLite for tracking database
  • regex: Pattern matching for filtering
  • ignore: gitignore-aware file traversal
  • colored: Terminal output formatting
  • serde/serde_json: Configuration and JSON parsing

Build Optimizations

Release profile (Cargo.toml:31-36):

  • opt-level = 3: Maximum optimization
  • lto = true: Link-time optimization
  • codegen-units = 1: Single codegen for better optimization
  • strip = true: Remove debug symbols
  • panic = "abort": Smaller binary size

CI/CD

GitHub Actions workflow (.github/workflows/release.yml):

  • Multi-platform builds (macOS, Linux x86_64/ARM64, Windows)
  • DEB/RPM package generation
  • Automated releases on version tags (v*)
  • Checksums for binary verification

Build Verification (Mandatory)

CRITICAL: After ANY Rust file edits, ALWAYS run the full quality check pipeline before committing:

cargo fmt --all && cargo clippy --all-targets && cargo test --all

Rules:

  • Never commit code that hasn't passed all 3 checks
  • Fix ALL clippy warnings before moving on (zero tolerance)
  • If build fails, fix it immediately before continuing to next task
  • Pre-commit hook will auto-enforce this (see .claude/hooks/bash/pre-commit-format.sh)

Why: RTK is a production CLI tool used by developers in their workflows. Bugs break developer productivity. Quality gates prevent regressions and maintain user trust.

Performance verification (for filter changes):

# Benchmark before/after
hyperfine 'rtk git log -10' --warmup 3
cargo build --release
hyperfine 'target/release/rtk git log -10' --warmup 3

# Memory profiling
/usr/bin/time -l target/release/rtk git status  # macOS
/usr/bin/time -v target/release/rtk git status  # Linux

Testing Policy

Manual testing is REQUIRED for filter changes and new commands:

  • For new filters: Test with real command (rtk <cmd>), verify output matches expectations

    • Example: rtk git log -10 → inspect output, verify condensed correctly
    • Example: rtk cargo test → verify only failures shown, not full output

  • For hook changes: Test in real Claude Code session, verify command rewriting works

    • Create test Claude Code session
    • Type raw command (e.g., git status)
    • Verify hook rewrites to rtk git status

  • For performance: Run hyperfine comparison (before/after), verify <10ms startup

    • Benchmark baseline: hyperfine 'rtk git status' --warmup 3
    • Make changes, rebuild
    • Benchmark again: hyperfine 'target/release/rtk git status' --warmup 3
    • Compare results: startup time should be <10ms

  • For cross-platform: Test on macOS + Linux (Docker) + Windows (CI), verify shell escaping

    • macOS (zsh): Test locally
    • Linux (bash): Use Docker docker run --rm -v $(pwd):/rtk -w /rtk rust:latest cargo test
    • Windows (PowerShell): Trust CI/CD pipeline or test manually if available

Anti-pattern: Running only automated tests (cargo test, cargo clippy) without actually executing rtk <cmd> and inspecting output.

Example: If fixing the git log filter, run rtk git log -10 and verify:

  1. Output is condensed (shorter than raw git log -10)
  2. Critical info preserved (commit hashes, messages)
  3. Format is readable and consistent
  4. Exit code matches git's exit code (0 for success)

Working Directory Confirmation

ALWAYS confirm working directory before starting any work:

pwd  # Verify you're in the rtk project root
git branch  # Verify correct branch (main, feature/*, etc.)

Never assume which project to work in. Always verify before file operations.

Avoiding Rabbit Holes

Stay focused on the task. Do not make excessive operations to verify external APIs, documentation, or edge cases unless explicitly asked.

Rule: If verification requires more than 3-4 exploratory commands, STOP and ask the user whether to continue or trust available info.

Examples of rabbit holes to avoid:

  • Excessive regex pattern testing (trust snapshot tests, don't manually verify 20 edge cases)
  • Deep diving into external command documentation (use fixtures, don't research git/cargo internals)
  • Over-testing cross-platform behavior (test macOS + Linux, trust CI for Windows)
  • Verifying API signatures across multiple crate versions (use docs.rs if needed, don't clone repos)

When to stop and ask:

  • "Should I research X external API behavior?" → ASK if it requires >3 commands
  • "Should I test Y edge case?" → ASK if not mentioned in requirements
  • "Should I verify Z across N platforms?" → ASK if N > 2

Plan Execution Protocol

When user provides a numbered plan (QW1-QW4, Phase 1-5, sprint tasks, etc.):

  1. Execute sequentially: Follow plan order unless explicitly told otherwise
  2. Commit after each logical step: One commit per completed phase/task
  3. Never skip or reorder: If a step is blocked, report it and ask before proceeding
  4. Track progress: Use task list (TaskCreate/TaskUpdate) for plans with 3+ steps
  5. Validate assumptions: Before starting, verify all referenced file paths exist and working directory is correct

Why: Plan-driven execution produces better outcomes than ad-hoc implementation. Structured plans help maintain focus and prevent scope creep.

Filter Development Checklist

When adding a new filter (e.g., rtk newcmd):

Implementation

  • Create filter module in src/<cmd>_cmd.rs (or extend existing)
  • Add lazy_static! regex patterns for parsing (compile once, reuse)
  • Implement fallback to raw command on error (graceful degradation)
  • Preserve exit codes (std::process::exit(code) if non-zero)

Testing

  • Write snapshot test with real command output fixture (tests/fixtures/<cmd>_raw.txt)
  • Verify token savings ≥60% with count_tokens() assertion
  • Test cross-platform shell escaping (macOS, Linux, Windows)
  • Write unit tests for edge cases (empty output, errors, unicode, ANSI codes)

Integration

  • Register filter in main.rs Commands enum
  • Update README.md with new command support and token savings %
  • Update CHANGELOG.md with feature description

Quality Gates

  • Run cargo fmt --all && cargo clippy --all-targets && cargo test
  • Benchmark startup time with hyperfine (verify <10ms)
  • Test manually: rtk <cmd> and inspect output for correctness
  • Verify fallback: Break filter intentionally, confirm raw command executes

Documentation

  • Add command to this CLAUDE.md Module Responsibilities table
  • Document token savings % (from tests)
  • Add usage examples to README.md

Example workflow (adding rtk newcmd):

# 1. Create module
touch src/newcmd_cmd.rs

# 2. Write test first (TDD)
echo 'raw command output fixture' > tests/fixtures/newcmd_raw.txt
# Add test in src/newcmd_cmd.rs

# 3. Implement filter
# Add lazy_static regex, implement logic, add fallback

# 4. Quality checks
cargo fmt --all && cargo clippy --all-targets && cargo test

# 5. Benchmark
hyperfine 'rtk newcmd args'

# 6. Manual test
rtk newcmd args
# Inspect output, verify condensed

# 7. Document
# Update README.md, CHANGELOG.md, this file