Development Guide

Last Updated: 2026-01-19

Quick Reference

# Install
bun install

# Build all packages
bun run build

# Lint
bun run lint
bun run lint:fix

# Test (package-specific)
cd packages/<package> && bun run test

# Run UI dev server
cd packages/ui && bun run dev

# Create changeset
bun run changeset

# Version packages
bun run version

# Publish all (automated CI)
bun run release

# Manual release (when CI fails)
export NPM_TOKEN=<token>
export GITHUB_TOKEN=<token>
bun scripts/release-all.ts

Development Workflow

1. Setup

git clone <repo>
cd root
bun install

Requirements:

Bun v1.3.4+ (bun.sh)
Git
Modern browser (Chrome 86+, Edge 86+)

2. Making Changes

Critical Rules (from AGENTS.md):

⚠️ NEVER auto-commit - always ask user approval first
⚠️ ALWAYS fetch/pull main before creating new branch
⚠️ NEVER suppress issues - ask user what to do

Workflow:

Fetch latest: git fetch origin && git pull origin main
Create branch: git checkout -b <branch-name>
Make changes
Test locally
Create changeset: bun run changeset
Commit with approval
Push: git push -u origin <branch-name>
Create PR via gh pr create

3. Testing Strategy

Default: NO testing during development

Test AFTER feature completion
Only run tests if you modified source files
Test files in tests/ directories, never in src/

Per-Package:

# kql-lezer (currently failing - dep issues)
cd packages/kql-lezer && bun run test

# kql-to-duckdb (113 tests)
cd packages/kql-to-duckdb && bun run test tests

# ui (manual testing)
cd packages/ui && bun run dev

4. Documentation Updates

After ANY feature:

Update feature checklists (mark [x] for completed)
Add patterns/gotchas to dev guides
Update status docs

Why: AI agents have no memory between sessions. Without updates, knowledge is lost.

Package-Specific Workflows

kql-lezer Development

When to Modify:

Adding new KQL operator support
Fixing parsing edge cases
Improving AST structure

Process:

Update src/kql.grammar (Lezer grammar)
Run bun run build:grammar (regenerates parser.ts)
Update src/parser/cst-to-ast/ mappers if needed
Update types in src/types.ts
Run tests (when dep issue fixed)
Update docs

Key Files:

src/kql.grammar - Hand-written grammar (THIS is the source of truth)
src/parser.ts - Generated (DO NOT EDIT manually)
src/parser/cst-to-ast/index.ts - Main mapper

Grammar Debugging: See packages/kql-lezer/docs/howto-grammar-debug.md

kql-to-duckdb Development

When to Modify:

Adding support for new operators
Adding function mappings
Fixing SQL generation bugs

Process:

Import types from kql-lezer if needed
Update src/translator.ts
Add function mapping if needed
Add test cases in tests/
Run bun run test tests
Update status docs

Key Patterns:

// Adding new operator
function translateOperator(op: OperatorType): string {
  const column = translateExpression(op.column)
  return `SELECT ${column} FROM ...`
}

// Adding new function
const functionMap = {
  kqlName: 'SQL_NAME',
  // ...
}

ui Development

When to Modify:

UI/UX improvements
Adding query features
Fixing layout issues

Process:

Start dev server: bun run dev
Make changes in src/
Test in browser
Update styles in src/styles/theme.css
Document gotchas in docs/ui-dev.md

Hot Module Replacement: Vite auto-reloads on changes

Common Tasks

Adding a New KQL Operator

kql-lezer: Add grammar rule in src/kql.grammar
kql-lezer: Add CST→AST mapper in src/parser/cst-to-ast/operators/
kql-lezer: Add type definition
kql-to-duckdb: Add translator in src/translator.ts
kql-to-duckdb: Add tests
Both: Update status docs

Fixing a Build Issue

Check TypeScript errors: bun run build
Check lint errors: bun run lint
Fix issues
Verify: bun run build && bun run lint

Creating a Pull Request

Required by AGENTS.md:

Run git status, git diff, git log in parallel
Analyze all changes (NOT just latest commit)
Draft PR summary covering ALL commits
Push with -u flag if new branch
Use gh pr create with HEREDOC for body:

gh pr create --title "Title" --body "$(cat <<'EOF'
## Summary
- Bullet points of changes

## Test plan
- Checklist for testing
EOF
)"

MUST include disclaimer: _This PR was created by an AI agent on behalf of @<username>._

Gotchas & Lessons Learned

General

Use Bun, not npm: All scripts use bun run, not npm run
Workspace dependencies: Use workspace:* for internal packages
ESM only: All packages use "type": "module"
Read before edit: Always read files before modifying
Minimize output: Use head, tail, grep to limit bash output

kql-lezer

Generated files: parser.ts is auto-generated, don't edit directly
Grammar syntax: Lezer uses { } for alternation, not tree-sitter's choice()
CST nodes: Lezer CST uses .node property, not .children
Test deps: @lezer/lr resolution issue - tests currently failing

kql-to-duckdb

CTE naming: Use cte_0, cte_1, etc. for deterministic output
Expression parentheses: Always parenthesize sub-expressions to maintain precedence
Function mapping: Case-sensitive - count() ≠ COUNT()
Let variables: Substitute before translation, not during

ui (SolidJS)

No hooks: SolidJS uses createSignal, not useState
Reactivity: Wrap derived data in createMemo, not regular functions
Grid truncation: MUST use min-width: 0 on grid children
Theme switching: Update DOM classes, not just CSS
DuckDB files: Must be in public/, not bundled by Vite
File handles: Lost on page reload without user permission

Troubleshooting

Tests Failing

kql-lezer:

Issue: Cannot find @lezer/lr
Workaround: None currently - known issue

kql-to-duckdb:

Issue: "0 test files matching pattern"
Fix: Ensure test files have .test.ts or .spec.ts suffix
Run: bun test tests (not just bun test)

Build Errors

TypeScript errors:

Check imports from workspace packages
Verify tsconfig.json references
Run bun run build from root to build in order

Lezer grammar errors:

Check grammar syntax in kql.grammar
Run bun run build:grammar to see parser generation errors
See docs/howto-grammar-debug.md

UI Not Loading

DuckDB errors:

Check public/duckdb-eh.wasm exists
Check browser console for WASM errors
Verify DuckDB WASM version compatibility

CodeMirror errors:

Check kql-lezer build output
Verify @codemirror packages installed
Check browser console for module errors

Git Workflow

Branch Naming

Feature: feat/<description>
Fix: fix/<description>
Agent work: claude/<description>-<session-id>

Commit Messages

feat(package): short description

Longer description if needed.
Explains why, not what.

Examples:

feat(kql-lezer): add support for between operator
fix(ui): prevent grid column overflow with long text
docs: update kql-to-duckdb status with test count

GitHub CLI Usage

Always use gh for GitHub operations:

# Create issue
gh issue create --title "Title" --body "Body" --label "bug,agent"

# Create PR
gh pr create --title "Title" --body "$(cat <<'EOF'
Body here
EOF
)"

# View workflow
gh run view <run-id>

# Check logs
gh run view --log-failed --job=<job-id>

MUST include disclaimer on ALL GitHub content:

---
_This {issue|PR|comment} was created by an AI agent on behalf of @<username>._

Monorepo Management

Turborepo

Build order: Automatically handled by turbo Caching: Turbo caches build outputs Filtering: Use --filter to target specific packages

# Build only ui and its dependencies
bun run build --filter=./packages/ui

Changesets

Creating a changeset:

bun run changeset
# Follow prompts:
# 1. Select packages that changed
# 2. Choose version bump (patch/minor/major)
# 3. Write summary

Versioning:

# Update package versions
bun run version

# Publish to npm
bun run release

Fixed mode: All packages version together (currently all at 1.2.0)

Manual Release Process

The automated changesets workflow may not always work. Use these manual scripts when needed:

Prerequisites:

# Set required tokens
export NPM_TOKEN=<your-npm-token>
export GITHUB_TOKEN=<your-github-token>

# Version packages first
bun run version

Individual steps:

# 1. Publish to npm only
bun scripts/publish-npm.ts

# 2. Publish to GitHub Package Registry only
bun scripts/publish-github.ts

# 3. Create GitHub release only
bun scripts/create-release.ts

# 4. Deploy UI only
bun scripts/deploy-ui.ts

Complete release (all steps):

# Runs: build → npm publish → GitHub publish → create release → deploy UI
bun scripts/release-all.ts

Scripts location: /scripts/

publish-npm.ts - Publish all packages to npm
publish-github.ts - Publish all packages to GitHub registry
create-release.ts - Create GitHub release from version
deploy-ui.ts - Deploy UI to fossiq.github.io
release-all.ts - Run complete release workflow

Notes:

UI deployment requires either SSH key or GITHUB_TOKEN
All packages must be built before publishing
Versions should be bumped via changesets before running scripts

Code Style

TypeScript

Functions: Small, focused, single responsibility
Naming: Descriptive (no single letters except loop indices)
Files: Under ~100-150 lines
Errors: Handle with descriptive messages
Pure functions: Prefer over classes

Imports

// External first
import { something } from 'external-package'

// Internal workspace
import { Type } from '@fossiq/kql-ast'

// Relative
import { helper } from './helpers'

Comments

Only where logic isn't self-evident
Explain WHY, not WHAT
Update when code changes

CI/CD

GitHub Actions

Workflows:

ci.yml - Main workflow that orchestrates all jobs on push/PR
lint.yml - Code linting
build.yml - Build all packages
test.yml - Run tests
publish-npm.yml - Publish packages to npm (decoupled from UI)
deploy-ui.yml - Deploy UI to fossiq.github.io (decoupled from npm)

Workflow Triggers:

ci.yml - Runs on push to main and all PRs
- Runs lint → build → test
- On main branch only: publish-npm + deploy-ui (parallel)
deploy-ui.yml - Also runs independently on push to main when UI-related files change
- Triggers: changes in packages/ui/, packages/kql-lezer/, packages/kql-to-duckdb/, packages/kql-ast/

Secrets Required:

NPM_TOKEN - npm authentication for publishing packages
DEPLOY_KEY_GITHUB_IO - SSH deploy key for fossiq.github.io repository
GITHUB_TOKEN - Automatically provided by GitHub Actions

Debugging CI:

gh run view <run-id> - See job status
gh run view --log-failed --job=<job-id> - See failed logs
Check workflow definitions in .github/workflows/
Inspect scripts in .github/scripts/

Common CI Failures:

Dependency installation (check lock file)
Cache issues (update cache keys)
Build order (check turbo.json)
Missing secrets (check repository settings → Secrets)
Deploy key permissions (check fossiq.github.io deploy keys)

Decoupled Publishing Architecture

The publishing and deployment workflows are intentionally decoupled:

Why Decoupled:

npm package releases and UI deployments have different cadences
UI can be deployed independently when fixing visual bugs
npm packages can be published without re-deploying UI
Failures in one pipeline don't block the other
Both can run in parallel for faster releases

Workflow:

Push to main
    ↓
CI Workflow
    ├─→ Lint
    ├─→ Build
    └─→ Test
         ├─→ Publish npm (if on main)
         │   ├─→ Publish packages to npm
         │   └─→ Create GitHub release
         │
         └─→ Deploy UI (if on main)
             └─→ Deploy to fossiq.github.io

UI changes detected
    ↓
Deploy UI Workflow (independent)
    └─→ Deploy to fossiq.github.io

Manual Triggers:

# Trigger npm publish only (via GitHub UI)
gh workflow run publish-npm.yml

# Trigger UI deploy only (via GitHub UI)
gh workflow run deploy-ui.yml

Performance Tips

Development

Use bun run dev in ui for hot reload
Use --filter to build only changed packages
Use lint fix: bun run lint:fix vs manual fixes

Production

All packages pre-built for npm
UI uses Vite's optimized build
DuckDB WASM lazy-loaded
Table virtualization for large results

Resources

Bun: https://bun.sh/docs
Lezer: https://lezer.codemirror.net/
SolidJS: https://www.solidjs.com/docs
DuckDB WASM: https://duckdb.org/docs/api/wasm
CodeMirror 6: https://codemirror.net/docs/
Changesets: https://github.com/changesets/changesets

Uh oh!

FilesExpand file tree

DEVELOPMENT.md

Latest commit

History

DEVELOPMENT.md

File metadata and controls

Development Guide

Quick Reference

Development Workflow

1. Setup

2. Making Changes

3. Testing Strategy

4. Documentation Updates

Package-Specific Workflows

kql-lezer Development

kql-to-duckdb Development

ui Development

Common Tasks

Adding a New KQL Operator

Fixing a Build Issue

Creating a Pull Request

Gotchas & Lessons Learned

General

kql-lezer

kql-to-duckdb

ui (SolidJS)

Troubleshooting

Tests Failing

Build Errors

UI Not Loading

Git Workflow

Branch Naming

Commit Messages

GitHub CLI Usage

Monorepo Management

Turborepo

Changesets

Manual Release Process

Code Style

TypeScript

Imports

Comments

CI/CD

GitHub Actions

Decoupled Publishing Architecture

Performance Tips

Development

Production

Resources