docs

Author: litecreator

.gitignore

@@ -1 +1,20 @@
-target/
+# Build artifacts
+/target/
+
+# Coverage and profiling
+/coverage/
+*.profraw
+*.profdata
+
+# Logs and temp files
+*.log
+*.tmp
+*.temp
+
+# Editor and OS files
+/.idea/
+/.vscode/
+*.swp
+*.swo
+.DS_Store
+Thumbs.db

docs/README.md

@@ -0,0 +1,22 @@
+# dustfmt Documentation
+
+This directory contains complete Markdown documentation for `dustfmt`.
+
+## Documentation Index
+
+- `getting_started.md`: build, install, and first formatting runs.
+- `cli_reference.md`: command-line contract and exit behavior.
+- `architecture.md`: formatter pipeline and module boundaries.
+- `formatting_rules.md`: canonical formatting behavior implemented today.
+- `frontend_validation.md`: parser/lexer validation path used before formatting.
+- `formatter_lexer.md`: comment-preserving formatter lexer behavior.
+- `library_api.md`: public Rust API (`format_source`).
+- `testing.md`: current test coverage and test caveats.
+- `known_limitations.md`: implementation limits and behavior mismatches to track.
+- `developer_guide.md`: local workflow for contributors.
+
+## Scope
+
+`dustfmt` is the Rust formatter for Dust source files. It validates input through an embedded frontend parser, then re-lexes source with a formatter-specific lexer to preserve comments while normalizing layout.
+
+This documentation describes current behavior from source in `crates/dustfmt/src`.

docs/architecture.md

@@ -0,0 +1,33 @@
+# Architecture
+
+## Crate Layout
+
+- `src/main.rs`: CLI and file/stdin orchestration.
+- `src/lib.rs`: public formatting API (`format_source`).
+- `src/format.rs`: token-to-text canonical formatter.
+- `src/lexer.rs`: formatter lexer that preserves comments.
+- `src/frontend/*`: embedded frontend lexer/parser/AST used for validation.
+
+## Pipeline
+
+`format_source(src)` executes two phases:
+
+1. **Validation phase**
+   - run `frontend::lexer::Lexer::lex_all`
+   - run `frontend::parser::Parser::parse_file`
+   - fail early on lex/parse errors
+
+2. **Formatting phase**
+   - run formatter lexer (`src/lexer.rs`) to retain comments
+   - run `format::format_tokens`
+   - emit canonical formatted text with trailing newline
+
+## Design Intent
+
+- avoid semantic changes by requiring syntactically valid input
+- preserve comments while normalizing whitespace/layout
+- deterministic, idempotent formatting output
+
+## Implementation Note
+
+The embedded frontend module is a copied/adjusted parser stack inside this crate rather than an external dependency on the full `dust` compiler crate.

docs/cli_reference.md

@@ -0,0 +1,47 @@
+# CLI Reference
+
+Source: `crates/dustfmt/src/main.rs`
+
+## Invocation
+
+```text
+dustfmt [--check|-c] [files...]
+```
+
+## Flags
+
+- `--check`: check-only mode
+- `-c`: short alias for check-only mode
+
+No other special flags are handled in current implementation.
+
+## Modes
+
+## File Mode
+
+When one or more files are provided:
+
+- normal mode: each file is formatted and rewritten in place
+- check mode: formatting differences produce an error and exit code `1`
+
+## STDIN Mode
+
+When no files are provided:
+
+- reads entire stdin
+- normal mode: writes formatted text to stdout
+- check mode: emits message and exits `1` when formatted output differs
+
+## Exit Behavior
+
+- `0`: successful formatting/check with no differences (in check mode)
+- `1`: check failed, parse/lex/format error, file read/write error, or explicit process error path
+
+## Diagnostics
+
+Representative stderr output patterns:
+
+- `format error: <details>`
+- `<file>: format error: <details>`
+- `check failed: input differs from formatted output`
+- `check failed: <file> differs from formatted output`

docs/developer_guide.md

@@ -0,0 +1,35 @@
+# Developer Guide
+
+## Build
+
+```bash
+cargo build --workspace --verbose
+```
+
+## Test
+
+```bash
+cargo test --workspace --verbose
+```
+
+## Quick Manual Checks
+
+```bash
+cargo run -p dustfmt -- --check crates/dustfmt/tests/fixtures/sample.ds
+cargo run -p dustfmt -- crates/dustfmt/tests/fixtures/sample.ds
+```
+
+(Adjust file paths to local fixtures that exist.)
+
+## Suggested Workflow for Formatter Changes
+
+1. update `src/lexer.rs` or `src/format.rs`
+2. run tests
+3. add/update tests in `crates/dustfmt/tests/format.rs`
+4. update docs in `dustfmt/docs`
+
+## Priority Fix Areas
+
+- align line-comment behavior with intended preservation (`//` fidelity)
+- expand parser coverage to match formatter keyword surface
+- improve CLI diagnostics for file-system failures

docs/formatter_lexer.md

@@ -0,0 +1,30 @@
+# Formatter Lexer
+
+Source: `crates/dustfmt/src/lexer.rs`
+
+## Token Model
+
+Formatter lexer tokens include:
+
+- structural tokens: delimiters/operators/newlines
+- lexical atoms: identifiers, keywords, regimes, numbers, bools, strings
+- comments: line and block comments preserved as token payload
+- fallback tokens: unknown characters
+
+## Comment Handling
+
+- line comments and block comments are tokenized for preservation
+- unlike the validation lexer, comments are not discarded
+
+## Keyword Coverage
+
+Keyword matching includes a broad set (core DPL plus several v0.2-oriented terms), independent of whether parser/formatter fully supports all constructs semantically.
+
+## String and Number Reading
+
+- strings are read until closing `"` without escape processing parity with compiler lexer
+- number scanning allows digits and `.`
+
+## Operational Role
+
+The formatter lexer is a layout-oriented tokenizer. It is not authoritative for syntax validity; syntax validity is delegated to the embedded frontend validation pass.

docs/formatting_rules.md

@@ -0,0 +1,39 @@
+# Formatting Rules (Current Implementation)
+
+Source: `crates/dustfmt/src/format.rs`
+
+## Indentation
+
+- indentation level is tracked by `{` and `}` tokens
+- each level uses 4 spaces
+
+## Braces
+
+- `{` is emitted on current line (with a preceding space if needed)
+- `}` is emitted on its own line at decremented indentation
+
+## Statement Breaks
+
+- `;` forces line flush
+- comments force current line flush before comment emission
+
+## Commas and Colons
+
+- `,` and `:` are followed by one space
+
+## Line Normalization
+
+- trailing whitespace is trimmed per emitted line
+- final newline is always ensured (`result` ends with an empty final line)
+
+## Newline Tokens
+
+- lexer newline tokens are ignored by formatter logic; output newlines are structural
+
+## Unknown Tokens
+
+- unknown characters are emitted as standalone indented lines
+
+## Spacing Heuristic
+
+Spacing between tokens uses a local previous-token heuristic (for example around operators, delimiters, and scope operators). This is rule-based and not AST-aware.

docs/frontend_validation.md

@@ -0,0 +1,34 @@
+# Frontend Validation
+
+Sources:
+
+- `crates/dustfmt/src/frontend/lexer.rs`
+- `crates/dustfmt/src/frontend/parser.rs`
+- `crates/dustfmt/src/frontend/ast.rs`
+
+## Purpose
+
+Before formatting, `dustfmt` validates that source can be lexed and parsed by the embedded frontend parser.
+
+## Validation Surface
+
+`format_source` rejects input when:
+
+- frontend lexing fails (`LexError`)
+- frontend parsing fails (`ParseError`)
+
+Error strings are normalized as:
+
+- `lex error: <kind> at <start>..<end>`
+- `parse error: <message> at <start>..<end>`
+
+## Parser Coverage in This Crate
+
+The embedded parser is intentionally limited in current state:
+
+- supports forge parsing and top-level shorthand procs
+- supports `proc` declarations and shorthand `K/Q/F name { ... }`
+- statement support is narrow (`emit` statement path in parser)
+- many keywords/features return explicit unsupported errors
+
+Because validation runs before formatting, unsupported-but-lexable syntax can still fail format with parser errors.

docs/getting_started.md

@@ -0,0 +1,60 @@
+# Getting Started
+
+## Prerequisites
+
+- Rust toolchain with `cargo`.
+- Dust source files (`.ds`) to format.
+
+## Build
+
+From `dustfmt/`:
+
+```bash
+cargo build --release
+```
+
+Binary path (Windows):
+
+```text
+target\release\dustfmt.exe
+```
+
+## Format Files In Place
+
+```bash
+cargo run -p dustfmt -- file1.ds file2.ds
+```
+
+Behavior:
+
+- each listed file is read
+- formatted output overwrites the same file
+- stops on first formatting/parse/lex error
+
+## Check Mode
+
+```bash
+cargo run -p dustfmt -- --check file.ds
+```
+
+or:
+
+```bash
+cargo run -p dustfmt -- -c file.ds
+```
+
+Behavior:
+
+- no writes
+- exits `1` if any file differs from canonical formatted output
+
+## STDIN Mode
+
+If no files are supplied, input is read from stdin.
+
+```bash
+cat file.ds | cargo run -p dustfmt --
+```
+
+- normal mode: prints formatted source to stdout
+- check mode: compares stdin text to formatted result and exits `1` on mismatch

docs/known_limitations.md

@@ -0,0 +1,21 @@
+# Known Limitations
+
+## Parser Coverage vs Keyword Surface
+
+`src/lexer.rs` recognizes many keywords, but embedded parser coverage is narrower. Formatting can fail on valid-looking inputs when parser support is incomplete.
+
+## Comment Tokenization Quirk
+
+Current formatter lexer comment path consumes one `/` before line-comment capture, and existing tests encode this behavior. This can produce non-ideal comment rendering.
+
+## Formatting Is Token-Heuristic, Not Full AST Pretty-Print
+
+`format_tokens` applies local token spacing/newline rules; it does not perform deep AST-aware layout decisions.
+
+## Limited CLI Feature Set
+
+Only `--check`/`-c` is implemented. There are no configuration files, width options, or selective rule toggles in current code.
+
+## Error Handling Style
+
+Several file I/O paths use `expect(...)`, resulting in immediate process termination on file read/write failures rather than structured recoverable diagnostics.