Read CONTRIBUTING.md first. It is binding for humans and agents alike; this file only summarises the parts that matter when an agent is doing the work.
AI-assisted code is welcome. Submitting code the contributor does not understand is not. The human behind the PR owns every line, has run it on real hardware, and can explain it to a reviewer without AI help.
Agents must not:
- Run
git push,gh pr create,gh pr comment, orgh issue createon the user's behalf. - Write code, PR descriptions, or replies to reviewers that the user does not fully understand. The user must be able to explain and defend every line without AI help.
- Report tests or benchmarks as run when they were not.
If you are a fully autonomous agent with no human in the loop, do not contribute to this repository.
The main subsystems:
python/freetoken/ the engine, installed as the `freetoken` package with the `ft` CLI
server/ OpenAI / Anthropic / Responses HTTP APIs, streaming, tool-call parsers
scheduler/ chunked prefill, batching, cache manager
kvcache/ paged KV pools and the radix prefix caches
moe/ expert offload cache, CPU / GPU / hybrid MoE backends, quantized experts
models/ model registry and per-architecture loaders
kernel/ CUDA / Triton kernels, JIT cache, C++ extensions (`csrc/`)
layers/, attention/ fused ops and attention backends
engine/ cache budget planning and config resolution
checkpoint/ HF -> FTW fast-load conversion
tests/ mirrors python/freetoken/ by subsystem, see tests/README.md
benchmarks/ end-to-end and micro benchmarks, see benchmarks/README.md
docs/ install, quickstart, CLI and model docs
freetoken-kernel-cache/ companion wheel of prebuilt kernels, see its README
scripts/ wheel build and release scripts
checkpoint_diagnose/ why an HF safetensors checkpoint does not load, and fixes for non-standard ones
gguf_check/ which GGUF files of a family load
Linux x86_64 with an NVIDIA GPU. Use uv, not bare pip:
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
uv run pytest tests/ -m "not slow"CUDA kernels are JIT-compiled with nvcc on first use unless the prebuilt freetoken-kernel-cache wheel is installed. The C++ extensions under python/freetoken/kernel/csrc/ are built by setup.py; after changing them run python setup.py build_ext --inplace.
The ggml kernels under python/freetoken/kernel/csrc/ggml/ are copied from llama.cpp and edited in place. See docs/ggml-kernels.md before changing or updating them.
Performance changes come with A/B numbers against main.
A test is code someone keeps green and updates on every refactor. The suite should grow with the engine's silent failure modes, not with the number of PRs. The rules, for fixes, features and new models alike, are in tests/README.md; read them before writing a test. The parts agents most often get wrong:
- Verify a fix by re-running what failed (the command, request, or checkpoint from the issue) and report the result. If you cannot run it (missing hardware or checkpoint), say so and say what you ran instead. Verification alone does not call for a new test.
- Before adding a test, name the future change that would turn it red. If the only answer is "editing the test", do not write it. A crash or error on a path anyone runs shows itself and needs no test.
- Extend an existing
parametrize, table, or test file before creating anything. Ask the user before adding a test file, directory, fixture, or helper, and before deleting a test. - A new model's own tests go in
tests/models/<model>/without asking, and meet the same bar: parity against its reference, not copies of the config, registry, and smoke tests every model shares.
- Search existing issues and PRs before starting. Items on the Roadmap are discussed with maintainers before implementation; features not on it start as an issue.
- When helping the user draft an issue, follow the matching template in
.github/ISSUE_TEMPLATE/(engine bug, model checkpoint, feature request) and fill in every required field: hardware, driver, FreeToken version, checkpoint link, exact command, and the full log. - When a checkpoint fails to load, follow scripts/checkpoint_diagnose/README.md before drafting a model checkpoint issue: it tells a FreeToken gap from a non-standard checkpoint, and repairs the latter.
- One change per PR, linked to its issue, with the hardware, checkpoint ID and exact command it was tested with.
Comments explain a non-obvious "why", never restate the code. Write the code first, then add a comment only where a reader would otherwise be confused. Keep them to one or two lines. Configuration files get no comments. Use ASCII: - not em-dash, -> not arrows.
Conventional Commits, one line, imperative, lowercase, no trailing period:
fix(kvcache): size the SWA radix pool for chunked prefill
PRs are squash-merged, so the PR title follows the same format. The subject line is usually enough; add a body only when the change needs a why that the diff does not show, and keep it to a few lines. Only commit when the user asks. If the user wants attribution, use Assisted-by: <agent name>, not Co-authored-by.