Repository navigation
fix: comprehensive tech debt cleanup — security, CI, docs, tests, refactors - #535
Conversation
…injection, attestation verify Batch 1 of tech debt cleanup — P1 security fixes: - Sanitizer: add XXE prevention (empty entity map), input size limit (100MB), and DTD directive stripping (#65) - Sanitizer: replace custom ASCII-only toLower/containsIgnoreCase with strings.ToLower/strings.Contains for consistency (#66) - Export: replace naive dotDotCount path traversal check with proper absolute path resolution + CWD containment check (#67) - action.yml: move all inputs to env block to prevent shell injection, add set -euo pipefail and input validation (#89) - release.yml: fix attestation verification silently swallowing failures via || echo pattern — now fails the step on verification errors (#94) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…eleaser safety Batch 2 of tech debt cleanup — P1 quality + CI fixes: - Sort map keys in formatMapValue for deterministic markdown output, fixing GOTCHAS.md §3.1 violation (#68) - Fix README: remove STIG mentions (enterprise-only), replace dead rule detection example with implemented capabilities, remove nonexistent --validate flag (#69) - Wire test-race into ci-check and add CI race detector job with continue-on-error for GitHub runner compatibility (#70) - GoReleaser: use skip_push template instead of conditional tags that produce empty strings on prerelease builds (#90) - Release workflow: skip docker on snapshot builds to prevent accidental latest/v0 pushes from workflow_dispatch (#91) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…on, snapshot warning Batch 3 of tech debt cleanup: - Scope bare ldap_* FieldPatterns to authserver ancestor path to prevent over-matching on unrelated XML elements (#59) - Pre-lowercase field patterns at rule engine construction time, eliminating per-comparison allocations on the hot path (#77) - Add format/command allowlists and explicit docker pull with actionable errors to action.yml (#92) - Add workflow summary warning for snapshot builds so maintainers see that no artifacts were published (#95) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Batch 4 of tech debt cleanup: - Add CodeQL workflow for Go with weekly scheduled scan and PR/push triggers (#71) - Create codecov.yml with 70% project / 60% patch thresholds and enable fail_ci_if_error (#72) - Pin cyclonedx-gomod to v1.10.0 and goreleaser to 2.15.2 in release workflow and mise.toml (#79) - Revert slog migration for sanitizer escapeXMLText — sloglint forbids global slog and the function has no struct receiver; keeping log.Printf for this unreachable error path (#84 — won't fix as designed) - Remove custom compliance profiles from README (enterprise-only feature) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…Y.md versions Batch 5 of tech debt cleanup — documentation fixes: - Document sanitizer rule ordering contract (authserver_config must precede password) in rules.go and GOTCHAS.md §19 (#61) - Fix stale "Credential-only" label in sanitize CLI help to match updated scope including authserver values (#62) - Move all-contributors badge to top badge group in README (#63) - Update SECURITY.md supported versions: 1.3.x supported, 1.2.x end-of-life (#64) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…e sync Batch 6 of tech debt cleanup — documentation fixes: - Emit CLI warning when --mode red selected and add experimental admonition in audit.md; document stubs in GOTCHAS.md §8.4 (#75) - Document --failures-only flag in audit.md flags table with usage example (#85) - Fix pfSense incorrectly labeled as "third-party plugin" in convert.md — it's a built-in parser (#86) - Add missing SetFailuresOnly method to ReportComposer interface listing in CONTRIBUTING.md (#87) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…n docs Batch 7 of tech debt cleanup: - Document plugin loading trust model in plugin-development.md and GOTCHAS.md §2.5 — no verification on .so loading (#88) - Add t.Run() subtests to TestRedact_AuthServerConfig for per-case isolation (#60) - Replace strings.Join with running currentPath in sanitizer for reduced allocations on hot path (#78) - Fix action.yml/README docs: add pfSense mentions, list all formats and subcommands, clarify @main vs @v1, document args whitespace splitting (#93) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Batch 8 of tech debt cleanup — test coverage: - Add comprehensive cmd/diff tests: validateDiffFlags, parseConfigFile, outputDiffResult, PreRunE validation, completions (83%+ coverage for diff.go) (#76) - Add 26 DHCP diff comparison tests covering CompareDHCP section guards, per-interface comparison, compareStaticMappings, and edge cases (18.2% -> 100% coverage) (#81) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Batch 9 of tech debt cleanup — cmd/ coverage target: Add 10 test files covering DetermineExitCode, helper functions, config show/validate, shared XML flag completions, root setup, sanitize output path, audit integration (runAudit, generateAuditOutput, emitAuditResult), and validate updateMaxExitCode. cmd/ package coverage: 48.5% -> 82.9% (#80) Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…ckage Batch 10 of tech debt cleanup — large refactors: - Split builder.go (1,378 → 285 lines) into builder_system.go, builder_network.go, builder_security.go, builder_nat.go, builder_audit.go following existing builder_vpn.go pattern (#73) - Split analyzer.go (1,011 → 12 lines) into 7 domain files: analyzer_system.go, analyzer_firewall.go, analyzer_nat.go, analyzer_network.go, analyzer_services.go, analyzer_users.go, analyzer_routing.go (#74) - Remove internal/markdown deprecated wrapper package — moved RenderMarkdown/ValidateMarkdown to internal/converter/html.go, updated 3 callers (#82) - Evaluated shared converter extraction (#83) — 12 identical functions (~350 LOC) but extraction requires new package, interface design, and careful handling of pfSense-specific divergences; cost exceeds benefit. Keeping as tracked low-priority item. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…ardening
Fix issues identified by code-reviewer, silent-failure-hunter, and
pr-test-analyzer agents:
Critical:
- action.yml: remove completion/config/man from command allowlist —
pointless in CI and mismatched with error message
- sanitizer: fix currentPath desync on XML elements containing dots —
use pathStack instead of strings.LastIndex to prevent field name dots
from corrupting the running path
Important:
- export: add warning log when resolveWithMissingTail falls back to
unresolved path (symlink protection degraded)
- sanitizer: document LDAP pattern scope assumption for non-OPNsense
device types in rules.go comment
- export: add symlink + dotdot path traversal test and direct
resolveWithMissingTail unit tests
- cmd/diff: fix vacuous context cancellation test — add NotPanics
wrapper and assert device nil on error
Medium:
- action.yml: add config-file and output path traversal validation
(reject absolute paths and ".." segments)
- action.yml: warn when args input contains quotes that won't be
preserved by whitespace splitting
- audit_integration_test: use json.Valid() instead of Contains("{")
- audit_integration_test: use t.Fatal instead of t.Skip for required
testdata fixtures
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (76)
Disabled knowledge base sources:
📝 WalkthroughSummary by CodeRabbit
WalkthroughReplaces the legacy internal/markdown facade with a domain-split internal/converter builder (system, network, security, NAT, audit), reimplements the diff Analyzer as focused Compare* modules, hardens sanitizer and export path handling, tightens CI/release/tooling, and adds many CLI tests and docs. No public Go API signatures intentionally changed. Changes
Sequence Diagram(s)sequenceDiagram
participant User as User/CLI
participant Cmd as cmd/audit
participant Builder as internal/converter/builder
participant Analyzer as internal/diff/Analyzer
participant Converter as internal/converter
participant Export as internal/export
User->>Cmd: invoke audit (config, --mode, --failures-only)
Cmd->>Cmd: PreRunE validate flags (warn on red)
Cmd->>Builder: runAudit(ctx, cfg, mode, failuresOnly)
Builder->>Analyzer: Compare*() (system, firewall, nat, interfaces, dhcp, users, routes)
Analyzer-->>Builder: []Change
Builder->>Builder: BuildSystem/Network/Security/Audit sections (tables)
Builder->>Converter: ValidateMarkdown(content) / RenderMarkdown(content)
Converter-->>Builder: validated/HTML result
Builder->>Export: ExportFile(content, path)
Export->>Export: checkPathTraversal(path) -> resolveWithMissingTail (symlink-aware)
Export-->>User: file written or error
sequenceDiagram
participant Builder as Converter/Builder
participant Sys as builder_system.go
participant Net as builder_network.go
participant Sec as builder_security.go
participant Nat as builder_nat.go
participant Audit as builder_audit.go
Note over Builder: BuildComprehensiveReport(device, failuresOnly)
Builder->>Sys: BuildSystemSection(device)
Sys-->>Builder: system markdown
Builder->>Net: BuildNetworkSection(device)
Net-->>Builder: network markdown
Builder->>Sec: BuildSecuritySection(device)
Sec-->>Builder: security markdown + firewall/NAT tables
Sec->>Nat: WriteInboundNATTable / WriteOutboundNATTable
Nat-->>Sec: nat tables
Builder->>Audit: BuildAuditSection(device)
Audit-->>Builder: audit markdown (plugin tables, summary, metadata)
Builder-->>Builder: concat -> final document
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 Enforce conventional commitWonderful, this rule succeeded.Require conventional commit format per https://www.conventionalcommits.org/en/v1.0.0/. Skipped for dependabot and dosubot.
🟢 Full CI must passWonderful, this rule succeeded.All CI checks must pass. Activates for non-bot authors, or dependabot when files exist outside .github/workflows/.
🟢 Do not merge outdated PRsWonderful, this rule succeeded.Make sure PRs are within 10 commits of the base branch before merging
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
Related Documentation 2 document(s) may need updating based on files changed in this PR: opnDossier architecture
|
There was a problem hiding this comment.
Pull request overview
Tech-debt cleanup focused on hardening XML sanitization and export path handling, tightening CI/release security checks, updating docs, and removing deprecated markdown wrapper code while expanding test coverage.
Changes:
- Hardened sanitizer and export pipeline (XXE/DTD handling, input size limits, safer path traversal checks, updated markdown render/validate location).
- CI/release improvements (race detection wiring, CodeQL workflow, Codecov thresholds, pinned tool versions, stricter attestation verification).
- Refactors and documentation updates (removed
internal/markdownwrapper package, updated user/dev docs and contribution contracts).
Reviewed changes
Copilot reviewed 70 out of 71 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| SECURITY.md | Updates supported versions table and support policy wording. |
| README.md | Updates feature/docs text and GitHub Action usage examples/inputs. |
| mise.toml | Pins goreleaser tool version. |
| mise.lock | Updates locked tool versions/checksums (including goreleaser). |
| justfile | Adds test-race to ci-check group. |
| internal/sanitizer/sanitizer.go | Adds input size cap, disables entity expansion, fixes path tracking, strips DTD directives. |
| internal/sanitizer/sanitizer_test.go | Adds XXE/size/directive tests and a benchmark. |
| internal/sanitizer/rules.go | Pre-lowercases patterns, removes custom ASCII helpers, documents rule ordering contract, scopes authserver ldap patterns. |
| internal/sanitizer/rules_test.go | Updates authserver field paths, refactors tests/benchmarks for new matching behavior. |
| internal/markdown/testdata/sample.config.3.xml | Removes markdown package test fixture (package removal). |
| internal/markdown/testdata/sample.config.1.xml | Removes markdown package test fixture (package removal). |
| internal/markdown/testdata/config.xml | Removes markdown package test fixture (package removal). |
| internal/markdown/README.md | Removes deprecated wrapper package docs. |
| internal/markdown/options.go | Removes wrapper re-exports to converter options/types. |
| internal/markdown/hybrid_generator.go | Removes wrapper hybrid generator shim. |
| internal/markdown/hybrid_generator_test.go | Removes wrapper tests. |
| internal/markdown/generator.go | Removes wrapper generator shim. |
| internal/markdown/generator_test.go | Removes wrapper tests. |
| internal/markdown/errors.go | Removes wrapper error re-exports. |
| internal/markdown/adapter.go | Removes wrapper adapter. |
| internal/export/file.go | Replaces naive traversal detection with abs+symlink resolution helper. |
| internal/export/file_test.go | Switches markdown validation/rendering calls to internal/converter. |
| internal/export/file_additional_test.go | Expands traversal tests (dotdot + symlink scenarios) and adds resolve helper tests. |
| internal/diff/formatters/html.go | Moves markdown→HTML rendering to internal/converter. |
| internal/diff/analyzer_users.go | Adds user comparison analyzer. |
| internal/diff/analyzer_test.go | Removes DHCP tests moved into dedicated file. |
| internal/diff/analyzer_system.go | Adds system comparison analyzer. |
| internal/diff/analyzer_services.go | Adds DHCP comparison analyzer and static mapping diff logic. |
| internal/diff/analyzer_routing.go | Adds routing comparison analyzer. |
| internal/diff/analyzer_network.go | Adds interfaces + VLAN comparison analyzers. |
| internal/diff/analyzer_nat.go | Adds NAT comparison analyzer. |
| internal/diff/analyzer_firewall.go | Adds firewall rule comparison analyzer (UUID + fallback by position). |
| internal/converter/html.go | Adds rich markdown renderer + RenderMarkdown/ValidateMarkdown in converter package. |
| internal/converter/builder/writer_test.go | Updates validation calls to converter.ValidateMarkdown. |
| internal/converter/builder/builder_system.go | New system section builder + user/group/sysctl table builders. |
| internal/converter/builder/builder_security.go | New security section builder (NAT + firewall + IDS). |
| internal/converter/builder/builder_network.go | New network section builder + interface/VLAN/static route tables. |
| internal/converter/builder/builder_nat.go | Adds inbound/outbound NAT table builders. |
| internal/converter/builder/builder_audit.go | Updates audit section rendering (controls table, failures-only behavior, summaries/metadata). |
| GOTCHAS.md | Documents plugin trust model, red mode stubs, sanitizer rule ordering contract. |
| docs/user-guide/commands/convert.md | Updates device type docs (OPNsense + pfSense built-in) and plugin wording. |
| docs/user-guide/commands/audit.md | Documents --failures-only and adds red mode “experimental” warning. |
| docs/development/plugin-development.md | Adds explicit dynamic plugin security model and recommendations. |
| CONTRIBUTING.md | Updates ReportComposer contract (SetFailuresOnly). |
| codecov.yml | Adds project/patch coverage thresholds and comment layout settings. |
| cmd/validate_coverage_test.go | Adds targeted unit tests to increase coverage for validate internals. |
| cmd/shared_flags_xml_test.go | Adds tests for XML file shell completion helper. |
| cmd/sanitize.go | Updates sanitize help text for minimal mode behavior. |
| cmd/sanitize_coverage_test.go | Adds tests for sanitize output path selection/overwrite prompts. |
| cmd/root_coverage_test.go | Adds tests for root build metadata accessors and lightweight context setup. |
| cmd/help_test.go | Adds tests for fuzzy suggestions, flag grouping, and example formatting. |
| cmd/exitcodes_test.go | Adds tests for exit code mapping and JSON error output helpers. |
| cmd/config_validate_coverage_test.go | Adds tests for plain context output and unknown key warnings. |
| cmd/config_show_coverage_test.go | Adds tests for plain config output grouping/formatting. |
| cmd/audit.go | Adds stderr warning when red mode is selected. |
| cmd/audit_integration_test.go | Adds integration-style tests for audit pipeline (JSON capture, file output, multifile). |
| action.yml | Hardens action runner step (env inputs, allowlists, traversal validation, safer bash settings). |
| .goreleaser.yaml | Disables stable docker tags for prereleases; simplifies stable tag list. |
| .github/workflows/release.yml | Pins cyclonedx tool, skips docker on snapshot, fixes attestation failure handling, adds snapshot warnings. |
| .github/workflows/codeql.yml | Adds CodeQL SAST workflow for Go (PR/push + weekly). |
| .github/workflows/ci.yml | Makes Codecov upload fail CI on error; adds race detector job. |
| .github/copilot-instructions.md | Updates architecture notes to remove internal/markdown from data flow. |
Summary
Resolves 35 tech debt items identified during PR #521 review and a full codebase audit. Two items deferred as won't-fix (shared converter extraction: cost > benefit; slog migration: sloglint constraint).
Impact: 79 files changed (+5,764 / -6,474) — net reduction of 710 lines
Categories: 27 source, 21 test, 17 docs, 7 CI/config
Review rounds: 3 (internal agents, Copilot, fixes applied)
Changes by Area
Security Hardening
sanitizer.gotoLower/containsIgnoreCasewithstrings.ToLower/strings.Containsrules.godotDotCountwith abs resolution + CWD containment viaEvalSymlinksfile.go..pathsfile.goenv:block, addset -euo pipefail, command/format allowlists, path validationaction.yml|| echothat swallowed failures in release workflowrelease.ymlcurrentPathcorruption when XML elements contain literal dots — usepathStacksanitizer.goldap_*patterns toauthserver.prefix to prevent cross-rule false matchesrules.goCI/Release
test-raceintojust ci-check+ add CI race detector job (continue-on-error: true)codecov.ymlwith 70% project / 60% patch thresholds; enablefail_ci_if_errorcyclonedx-gomod(v1.10.0),goreleaser(2.15.2)disabletemplate onghcr-stablefor prerelease builds (replaces empty-tag approach)--skip=docker) + add workflow summary warningPerformance
strings.Join(elementStack)with runningcurrentPath+pathStackDocumentation
--validateflag--failures-only, add red mode experimental warningReportComposerinterface (addSetFailuresOnly)Test Coverage
cmd/packagecmd/diff.goNew tests: sanitizer XXE/size/directive, path traversal symlink,
resolveWithMissingTail, benchmarksRefactors
builder.goanalyzer.gointernal/markdown/RenderMarkdown/ValidateMarkdown→converter/html.goWon't Fix
sloglintforbids globalslogand the function has no struct receiver. Keepinglog.Printffor unreachable error path.Review Feedback Addressed
checkPathTraversalskips symlink checks without..resolveWithMissingTailwrites to stderr(string, error)LF()afterPlainTextf()in builder_system.goLF()calls + updated golden filest.Parallel()range variable capturecurrentPathdesync on dot-containing elementspathStackapproachNotPanics+ assertionsContains("{")JSON assertionjson.Valid()t.Skipfor required testdatat.FatalTest plan
just ci-checkpasses (lint, modernize, test, race detector, pre-commit)just release-checkpasses (goreleaser config validation)Used Claude Code to perform code review with compound-engineering agents; identified issues were resolved manually by @unclesp1d3r, who mostly understands his own code.