diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 33c0dd2f..2bb6841b 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -5,6 +5,25 @@ on: pull_request: jobs: + lint: + name: Lint (Python) + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v6 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Install ruff + run: pip install ruff + + - name: Lint Python (undefined names, syntax errors) + working-directory: libs/openant-core + run: ruff check . + python-tests: name: Python tests (${{ matrix.os }}) runs-on: ${{ matrix.os }} @@ -51,7 +70,7 @@ jobs: - name: Run Python and parser tests working-directory: libs/openant-core - run: python -m pytest tests/test_token_tracker.py tests/test_parser_adapter.py tests/test_python_parser.py tests/test_js_parser.py -v + run: python -m pytest tests/ -v go-tests: name: Go build + integration (${{ matrix.os }}) @@ -93,6 +112,10 @@ jobs: working-directory: apps/openant-cli run: go vet ./... + - name: Run Go unit tests + working-directory: apps/openant-cli + run: go test ./... -v + - name: Build (Linux/macOS) if: runner.os != 'Windows' working-directory: apps/openant-cli diff --git a/.gitignore b/.gitignore index 5aa0e7b3..599ac159 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,4 @@ __pycache__/ node_modules/ apps/openant-cli/bin/ libs/openant-core/parsers/go/go_parser/go_parser -# docs/ +_docs/ diff --git a/CHANGELOG.md b/CHANGELOG.md index f09f2651..bbe7a9a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,97 @@ All notable changes to OpenAnt are documented in this file. +## [2026-05-10] — Windows compatibility & CI hardening + +### Fixed + +- **JavaScript parser no longer returns zero functions on Windows.** + `path.relative()` and `path.resolve()` produce backslash-separated + paths there, and ts-morph treats `\` as an escape character when + matching paths it has already added — the analyzer silently emitted + an empty result. The TypeScript analyzer now normalises every path + it hands to ts-morph (and every value stored as a `functionId` + component) to forward slashes via a `toPosixPath()` helper. A + static-scanner test in `libs/openant-core/tests/test_windows_path_handling.py` + enforces the contract on every commit. +- **`--files-from` no longer drops every path on Windows.** File lists + written with CRLF line endings used to leave a trailing `\r` on each + entry, which `addSourceFileAtPath` then failed to resolve. The + TypeScript analyzer now splits on `/\r?\n/` and trims each line. +- **Pipeline status output no longer crashes on cp1252 consoles.** + `parsers/{javascript,go}/test_pipeline.py` previously printed + `✓ ✗ →` directly, which raised `UnicodeEncodeError` on the Windows + default code page. Both pipelines now probe `sys.stdout.encoding` at + import time and fall back to ASCII (`OK` / `FAIL` / `->`) only when + the terminal can't encode the Unicode glyphs — UTF-8 terminals keep + the prettier output. +- **`'charmap' codec can't decode byte ...` errors on Windows.** Bare + `open()` calls and `subprocess.run(..., text=True)` invocations + across `libs/openant-core/` defaulted to the system locale encoding + (cp1252 on Windows), crashing on any source code containing non-ASCII + characters (curly quotes U+2019, accented characters, CJK). All ~190 + call sites now go through new helpers in + `libs/openant-core/utilities/file_io.py` (`open_utf8`, `read_json`, + `write_json`, `run_utf8`) that pin UTF-8 explicitly. Four regression + scanners in `tests/test_file_io.py` prevent reintroduction by failing + CI on any new bare `open(`, `.read_text(`/`.write_text(`, `.open(`, + or `subprocess.run(..., text=True)` call without an explicit + `encoding=`. +- **Token tracker NameError on resume.** `core/analyzer.py` called + `tracker.add_prior_usage(...)` without `tracker` being defined in the + surrounding `run_analysis()` function. The path was reached only when + resuming a scan with non-zero prior token usage — a dormant bug + uncovered by the new lint step. Now uses `get_global_tracker()` to + match the existing pattern in the same function. +- **Managed venv path is wrong on Windows.** `venvPython()` in + `apps/openant-cli/internal/python/runtime.go` hard-coded + `bin/python`, which doesn't exist in a Windows venv (the layout there + is `Scripts\python.exe`). The CLI now branches on `runtime.GOOS` and + returns the OS-correct path, so `~/.openant/venv/` is usable on + Windows without setting `OPENANT_PYTHON`. New `runtime_test.go` + covers both layouts. +- **Python parser test pipelines fail when invoked as subprocesses.** + `parsers/{javascript,go}/test_pipeline.py` import from `utilities.*` + but, when the Go CLI runs them as subprocesses with a different + working directory, `openant-core/` was not on `sys.path`. Both files + now prepend the openant-core root to `sys.path` before the + `utilities` import. +- **Anthropic SDK auth-error test broken by SDK update.** + `tests/test_silent_401.py` constructed `AuthenticationError("...")` + with a positional message; the current SDK requires + `AuthenticationError(message=, response=, body=)`. The test now + builds a mock `httpx.Response` and uses the keyword form, and + temporarily restores the real `anthropic` module so the real + exception class is used. +- **`run_utf8` explicit-encoding test crashed on Windows.** + `test_run_utf8_does_not_override_explicit_encoding` used + `print('café')` from a `-c` snippet, which itself fails to encode + on a cp1252 console before `run_utf8` even runs. The test now writes + raw `latin-1` bytes via `sys.stdout.buffer.write(...)` so the + encoding-override path is the thing under test on every platform. +- **`withTempHome` test helper didn't work on Windows.** Both copies + (`apps/openant-cli/cmd/mode_test.go` and + `apps/openant-cli/internal/config/scan_meta_test.go`) only set + `HOME`, but `os.UserHomeDir()` on Windows reads `USERPROFILE`. The + helpers now branch on `runtime.GOOS` and set the correct env var. + +### Added + +- **CI now lints for missing imports and undefined names.** A + `ruff check .` step runs in the `python-tests` job before `pytest`, + with `select = ["F821", "F811"]` (undefined name, redefined unused + name). Both rules are zero-false-positive runtime-bug catchers, so + contributors get fast static feedback on the kind of mistake Python + won't surface until the affected code path executes. Scoped narrowly + on purpose — widening to additional pyflakes rules can come later. +- **CI now runs Go unit tests on every platform.** A new + `go test ./... -v` step runs in the `go-tests` job before the build, + on Ubuntu, macOS, and Windows. Catches regressions like the venv + path bug above before the binary is built. The Python step also + switched from a hand-curated test list to `pytest tests/`, picking + up ten previously-CI-invisible test files (UTF-8 file I/O, Windows + path handling, dedup, cwe-tagging, evidence-tier, and others). + ## [2026-05-07] — Incremental scans + scan pipeline rewire ### Changed diff --git a/README.md b/README.md index 7a8d877d..66c5806d 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,16 @@ openant set-api-key **The key must have access to the Claude Opus 4.6 model.** Get a key at [console.anthropic.com](https://console.anthropic.com/settings/keys). +### Python runtime + +OpenAnt's parsing, enhancement, analysis, and reporting code is Python 3.11+. The Go CLI picks an interpreter in this order: + +1. `OPENANT_PYTHON` env var (set this to pin a specific interpreter — e.g. `OPENANT_PYTHON=python3.11`). +2. Managed venv at `~/.openant/venv/` (auto-created on first use). The CLI uses `bin/python` on Linux/macOS and `Scripts\python.exe` on Windows. +3. `python3` / `python` on `PATH`. + +If none yield Python 3.11+, the command exits with an error pointing at [python.org](https://www.python.org/downloads/). To rebuild a stale managed venv (e.g. after upgrading Python), delete `~/.openant/venv/` and rerun any `openant` command. + ## Data directories OpenAnt creates two directories: diff --git a/apps/openant-cli/cmd/mode_test.go b/apps/openant-cli/cmd/mode_test.go index 9844afb3..f98c273b 100644 --- a/apps/openant-cli/cmd/mode_test.go +++ b/apps/openant-cli/cmd/mode_test.go @@ -2,6 +2,7 @@ package cmd import ( "bytes" + "runtime" "strings" "testing" "time" @@ -123,13 +124,16 @@ func TestSelectModeNoFlagsNoBaselineGoesFull(t *testing.T) { } } -// Reuse helper from scan_meta_test.go's withTempHome. The cmd package -// can't import _test.go files from another package, so we redeclare a -// minimal copy here. +// Helper to set up a temporary home directory for tests. +// On Unix: sets HOME. On Windows: sets USERPROFILE. func withTempHome(t *testing.T) string { t.Helper() dir := t.TempDir() - t.Setenv("HOME", dir) + if runtime.GOOS == "windows" { + t.Setenv("USERPROFILE", dir) + } else { + t.Setenv("HOME", dir) + } return dir } diff --git a/apps/openant-cli/internal/config/scan_meta_test.go b/apps/openant-cli/internal/config/scan_meta_test.go index e5a9e0e2..f39979aa 100644 --- a/apps/openant-cli/internal/config/scan_meta_test.go +++ b/apps/openant-cli/internal/config/scan_meta_test.go @@ -3,16 +3,22 @@ package config import ( "os" "path/filepath" + "runtime" "testing" "time" ) -// withTempHome points HOME at a temp dir for the duration of the test so -// ProjectDir / ScanRunDir resolve under there. Restores HOME on cleanup. +// withTempHome points the home directory at a temp dir for the duration of the test so +// ProjectDir / ScanRunDir resolve under there. Restores on cleanup. +// On Unix: sets HOME. On Windows: sets USERPROFILE. func withTempHome(t *testing.T) string { t.Helper() dir := t.TempDir() - t.Setenv("HOME", dir) + if runtime.GOOS == "windows" { + t.Setenv("USERPROFILE", dir) + } else { + t.Setenv("HOME", dir) + } return dir } diff --git a/apps/openant-cli/internal/python/runtime.go b/apps/openant-cli/internal/python/runtime.go index ba8d1310..20a16317 100644 --- a/apps/openant-cli/internal/python/runtime.go +++ b/apps/openant-cli/internal/python/runtime.go @@ -6,6 +6,7 @@ import ( "os" "os/exec" "path/filepath" + "runtime" "strconv" "strings" ) @@ -40,7 +41,11 @@ func venvDir() string { // venvPython returns the path to the Python binary inside the managed venv. func venvPython() string { - return filepath.Join(venvDir(), "bin", "python") + base := venvDir() + if runtime.GOOS == "windows" { + return filepath.Join(base, "Scripts", "python.exe") + } + return filepath.Join(base, "bin", "python") } // DetectRuntime finds a suitable Python 3.11+ installation. @@ -51,10 +56,9 @@ func venvPython() string { // 2. Managed venv at ~/.openant/venv/ (if it exists and is valid) // 3. python3 / python on PATH // -// Note: the managed-venv path (strategy 2) uses "bin/python" which is correct -// on Linux/macOS. On Windows the venv layout uses "Scripts\python.exe"; users -// on Windows who rely on the managed venv should set OPENANT_PYTHON explicitly -// to point at the desired interpreter. +// The managed-venv path (strategy 2) automatically detects the correct Python +// binary location based on the OS: "bin/python" on Unix-like systems, or +// "Scripts/python.exe" on Windows. func DetectRuntime() (*RuntimeInfo, error) { // Strategy 0: honour explicit override via OPENANT_PYTHON env var. // If the override is set but unusable, warn and fall through rather than diff --git a/apps/openant-cli/internal/python/runtime_test.go b/apps/openant-cli/internal/python/runtime_test.go new file mode 100644 index 00000000..573814e5 --- /dev/null +++ b/apps/openant-cli/internal/python/runtime_test.go @@ -0,0 +1,47 @@ +package python + +import ( + "os" + "path/filepath" + "runtime" + "strings" + "testing" +) + +func TestVenvPython_Windows(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("test only runs on Windows") + } + + vp := venvPython() + expected := filepath.Join(os.Getenv("USERPROFILE"), ".openant", "venv", "Scripts", "python.exe") + if vp != expected { + t.Errorf("venvPython() = %q, want %q", vp, expected) + } + + if !filepath.IsAbs(vp) { + t.Errorf("venvPython() should return absolute path, got %q", vp) + } +} + +func TestVenvPython_Unix(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("test only runs on Unix-like systems") + } + + vp := venvPython() + if !filepath.IsAbs(vp) { + t.Errorf("venvPython() should return absolute path, got %q", vp) + } + + if !strings.HasSuffix(vp, filepath.Join("bin", "python")) { + t.Errorf("venvPython() on Unix should end with bin/python, got %q", vp) + } +} + +func TestVenvDir_ReturnsAbsolutePath(t *testing.T) { + vd := venvDir() + if !filepath.IsAbs(vd) { + t.Errorf("venvDir() should return absolute path, got %q", vd) + } +} diff --git a/libs/openant-core/context/application_context.py b/libs/openant-core/context/application_context.py index f7fa55d6..11940db1 100644 --- a/libs/openant-core/context/application_context.py +++ b/libs/openant-core/context/application_context.py @@ -31,6 +31,7 @@ from anthropic import Anthropic from dotenv import load_dotenv +from utilities.file_io import open_utf8, read_json, write_json # Load environment variables load_dotenv() @@ -208,7 +209,8 @@ def gather_context_sources(repo_path: Path) -> dict[str, str]: filepath = repo_path / filename if filepath.exists(): try: - content = filepath.read_text(errors="ignore") + with open_utf8(filepath, errors="ignore") as _f: + content = _f.read() # Limit size to avoid token overflow if len(content) > 10000: content = content[:10000] + "\n\n[... truncated ...]" @@ -289,7 +291,8 @@ def detect_entry_points(repo_path: Path) -> str: continue try: - content = py_file.read_text(errors="ignore") + with open_utf8(py_file, errors="ignore") as _f: + content = _f.read() rel_path = py_file.relative_to(repo_path) for category, patterns in ENTRY_POINT_PATTERNS.items(): @@ -308,7 +311,8 @@ def detect_entry_points(repo_path: Path) -> str: continue try: - content = js_file.read_text(errors="ignore") + with open_utf8(js_file, errors="ignore") as _f: + content = _f.read() rel_path = js_file.relative_to(repo_path) if re.search(r"express\(\)|require\(['\"]express['\"]\)", content): @@ -340,15 +344,17 @@ def check_manual_override(repo_path: Path) -> ApplicationContext | None: continue try: - content = filepath.read_text() - if filename.endswith('.json'): # Direct JSON format - data = json.loads(content) + data = read_json(filepath) data['source'] = 'manual' return ApplicationContext(**data) - elif filename.endswith('.md'): + # .md files need raw text so regex can extract the embedded JSON block. + with open_utf8(filepath) as _f: + content = _f.read() + + if filename.endswith('.md'): # Markdown format - check for JSON code block json_match = re.search(r'```json\s*(.*?)\s*```', content, re.DOTALL) if json_match: @@ -545,8 +551,7 @@ def save_context(context: ApplicationContext, output_path: Path) -> None: output_path = Path(output_path) output_path.parent.mkdir(parents=True, exist_ok=True) - with open(output_path, 'w') as f: - json.dump(asdict(context), f, indent=2) + write_json(output_path, asdict(context)) print(f"Context saved to {output_path}", file=sys.stderr) @@ -560,9 +565,7 @@ def load_context(input_path: Path) -> ApplicationContext: Returns: ApplicationContext loaded from file. """ - with open(input_path) as f: - data = json.load(f) - + data = read_json(input_path) # Mark as manual to skip validation (already validated when saved) original_source = data.get('source', 'llm') data['source'] = 'manual' # Temporarily bypass validation diff --git a/libs/openant-core/core/analyzer.py b/libs/openant-core/core/analyzer.py index 7fb59661..f8255f13 100644 --- a/libs/openant-core/core/analyzer.py +++ b/libs/openant-core/core/analyzer.py @@ -27,6 +27,7 @@ # Import existing analysis machinery from utilities.llm_client import AnthropicClient, get_global_tracker +from utilities.file_io import read_json, write_json from utilities.json_corrector import JSONCorrector from utilities.rate_limiter import get_rate_limiter, is_rate_limit_error, is_retryable_error @@ -330,9 +331,7 @@ def run_analysis( # Load dataset print(f"[Analyze] Loading dataset: {dataset_path}", file=sys.stderr) - with open(dataset_path) as f: - dataset = json.load(f) - + dataset = read_json(dataset_path) units = dataset.get("units", []) # Diff filter: if upstream parse stamped diff_selected on units (PR-diff @@ -390,7 +389,7 @@ def _usage_dict(): # Inject prior usage into tracker so step_report captures the total if _summary_input_tokens or _summary_output_tokens: - tracker.add_prior_usage( + get_global_tracker().add_prior_usage( _summary_input_tokens, _summary_output_tokens, _summary_cost_usd) # Write initial summary @@ -513,9 +512,7 @@ def _summary_callback(finding, usage=None): "code_by_route": code_by_route, } - with open(results_path, "w") as f: - json.dump(experiment_result, f, indent=2) - + write_json(results_path, experiment_result) print(f"\n[Analyze] Results written to {results_path}", file=sys.stderr) # Checkpoints are preserved as a permanent artifact alongside results. diff --git a/libs/openant-core/core/checkpoint.py b/libs/openant-core/core/checkpoint.py index 7c42f529..3b2015a0 100644 --- a/libs/openant-core/core/checkpoint.py +++ b/libs/openant-core/core/checkpoint.py @@ -27,6 +27,7 @@ from datetime import datetime, timezone from utilities.safe_filename import safe_filename +from utilities.file_io import read_json, write_json from pathlib import Path @@ -79,8 +80,7 @@ def load(self) -> dict[str, dict]: continue filepath = os.path.join(self.dir, filename) try: - with open(filepath, "r") as f: - data = json.load(f) + data = read_json(filepath) unit_id = data.get("id") if unit_id: results[unit_id] = data @@ -130,9 +130,7 @@ def save(self, unit_id: str, data: dict): filename = self._safe_filename(unit_id) + ".json" filepath = os.path.join(self.dir, filename) data["id"] = unit_id # ensure id is always present - with open(filepath, "w") as f: - json.dump(data, f, indent=2) - + write_json(filepath, data) def write_summary( self, total_units: int, @@ -168,9 +166,7 @@ def write_summary( } if usage is not None: data["usage"] = usage - with open(filepath, "w") as f: - json.dump(data, f, indent=2) - + write_json(filepath, data) @staticmethod def read_summary(checkpoint_dir: str) -> dict | None: """Read _summary.json from a checkpoint directory. @@ -182,8 +178,7 @@ def read_summary(checkpoint_dir: str) -> dict | None: if not os.path.isfile(filepath): return None try: - with open(filepath, "r") as f: - return json.load(f) + return read_json(filepath) except (json.JSONDecodeError, OSError): return None @@ -241,8 +236,7 @@ def status(checkpoint_dir: str) -> dict: continue filepath = os.path.join(checkpoint_dir, filename) try: - with open(filepath, "r") as f: - data = json.load(f) + data = read_json(filepath) except (json.JSONDecodeError, OSError): errors += 1 error_breakdown["unreadable"] = error_breakdown.get("unreadable", 0) + 1 diff --git a/libs/openant-core/core/diff_filter.py b/libs/openant-core/core/diff_filter.py index bd939173..07b832cb 100644 --- a/libs/openant-core/core/diff_filter.py +++ b/libs/openant-core/core/diff_filter.py @@ -30,10 +30,11 @@ from __future__ import annotations -import json import sys from dataclasses import dataclass, asdict +from utilities.file_io import read_json + # Scope constants (must match internal/git/manifest.go). SCOPE_CHANGED_FILES = "changed_files" @@ -65,8 +66,7 @@ def to_dict(self) -> dict: def load_manifest(path: str) -> dict: """Read and minimally validate a diff manifest file.""" - with open(path, "r", encoding="utf-8") as f: - m = json.load(f) + m = read_json(path) scope = m.get("scope") if scope not in _VALID_SCOPES: raise ValueError( diff --git a/libs/openant-core/core/dynamic_tester.py b/libs/openant-core/core/dynamic_tester.py index 9f9c10db..41b1a104 100644 --- a/libs/openant-core/core/dynamic_tester.py +++ b/libs/openant-core/core/dynamic_tester.py @@ -12,6 +12,7 @@ from core.schemas import DynamicTestStepResult, UsageInfo from core import tracking +from utilities.file_io import read_json, write_json def run_tests( @@ -51,9 +52,7 @@ def run_tests( os.makedirs(output_dir, exist_ok=True) # Check how many findings to test - with open(pipeline_output_path) as f: - pipeline_data = json.load(f) - + pipeline_data = read_json(pipeline_output_path) findings = pipeline_data.get("findings", []) testable = [ f for f in findings @@ -65,8 +64,7 @@ def run_tests( if not testable: results_path = os.path.join(output_dir, "dynamic_test_results.json") - with open(results_path, "w") as f: - json.dump({"findings_tested": 0, "results": []}, f, indent=2) + write_json(results_path, {"findings_tested": 0, "results": []}) return DynamicTestStepResult( results_json_path=results_path, diff --git a/libs/openant-core/core/enhancer.py b/libs/openant-core/core/enhancer.py index fef1453c..70879b81 100644 --- a/libs/openant-core/core/enhancer.py +++ b/libs/openant-core/core/enhancer.py @@ -17,6 +17,7 @@ from core import tracking from core.progress import ProgressReporter from utilities.rate_limiter import configure_rate_limiter +from utilities.file_io import read_json, write_json def enhance_dataset( @@ -69,9 +70,7 @@ def enhance_dataset( # Load dataset print(f"[Enhance] Loading dataset: {dataset_path}", file=sys.stderr) - with open(dataset_path) as f: - dataset = json.load(f) - + dataset = read_json(dataset_path) units = dataset.get("units", []) print(f"[Enhance] Units to enhance: {len(units)}", file=sys.stderr) @@ -138,9 +137,7 @@ def _on_restored(count: int): # Write enhanced dataset os.makedirs(os.path.dirname(os.path.abspath(output_path)), exist_ok=True) - with open(output_path, "w") as f: - json.dump(enhanced, f, indent=2) - + write_json(output_path, enhanced) print(f"[Enhance] Enhanced dataset: {output_path}", file=sys.stderr) print(f"[Enhance] Classifications: {classifications}", file=sys.stderr) if error_count: diff --git a/libs/openant-core/core/parser_adapter.py b/libs/openant-core/core/parser_adapter.py index 314d4704..46fc08c9 100644 --- a/libs/openant-core/core/parser_adapter.py +++ b/libs/openant-core/core/parser_adapter.py @@ -16,6 +16,7 @@ from pathlib import Path from core.schemas import ParseResult +from utilities.file_io import read_json, write_json # Root of openant-core (where parsers/ lives) _CORE_ROOT = Path(__file__).parent.parent @@ -161,9 +162,7 @@ def _maybe_apply_diff_filter( ) return - with open(result.dataset_path, "r") as f: - dataset = json.load(f) - + dataset = read_json(result.dataset_path) # Dataset may be a dict with "units" or a raw list. if isinstance(dataset, dict): units = dataset.get("units", []) @@ -172,14 +171,11 @@ def _maybe_apply_diff_filter( stats = apply_diff_filter(units, manifest) - with open(result.dataset_path, "w") as f: - json.dump(dataset, f, indent=2) - + write_json(result.dataset_path, dataset) # Expose stats on the ParseResult via a side-channel file; the parse # step_context reads this when assembling parse.report.json. diff_report_path = os.path.join(output_dir, "diff_filter.report.json") - with open(diff_report_path, "w") as f: - json.dump(stats.to_dict(), f, indent=2) + write_json(diff_report_path, stats.to_dict()) print( f" Diff filter ({stats.scope}): {stats.selected}/{stats.total} units selected" @@ -245,9 +241,7 @@ def _load_module(name, filename): print(f"\n[Reachability Filter] Filtering to {processing_level} units...", file=sys.stderr) - with open(call_graph_path, "r") as f: - call_graph_data = json.load(f) - + call_graph_data = read_json(call_graph_path) functions = call_graph_data.get("functions", {}) call_graph = call_graph_data.get("call_graph", {}) reverse_call_graph = call_graph_data.get("reverse_call_graph", {}) @@ -352,12 +346,8 @@ def _parse_python(repo_path: str, output_dir: str, processing_level: str, skip_t dataset = _apply_reachability_filter(dataset, output_dir, processing_level) # Write outputs - with open(dataset_path, "w") as f: - json.dump(dataset, f, indent=2) - - with open(analyzer_output_path, "w") as f: - json.dump(analyzer_output, f, indent=2) - + write_json(dataset_path, dataset) + write_json(analyzer_output_path, analyzer_output) units_count = len(dataset.get("units", [])) print(f" Python parser complete: {units_count} units", file=sys.stderr) @@ -413,8 +403,7 @@ def _parse_javascript(repo_path: str, output_dir: str, processing_level: str, sk # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" JavaScript parser complete: {units_count} units", file=sys.stderr) @@ -470,8 +459,7 @@ def _parse_go(repo_path: str, output_dir: str, processing_level: str, skip_tests # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" Go parser complete: {units_count} units", file=sys.stderr) @@ -530,8 +518,7 @@ def _parse_c(repo_path: str, output_dir: str, processing_level: str, skip_tests: # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" C/C++ parser complete: {units_count} units", file=sys.stderr) @@ -590,8 +577,7 @@ def _parse_ruby(repo_path: str, output_dir: str, processing_level: str, skip_tes # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" Ruby parser complete: {units_count} units", file=sys.stderr) @@ -650,8 +636,7 @@ def _parse_php(repo_path: str, output_dir: str, processing_level: str, skip_test # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" PHP parser complete: {units_count} units", file=sys.stderr) @@ -710,8 +695,7 @@ def _parse_zig(repo_path: str, output_dir: str, processing_level: str, skip_test # Count units units_count = 0 if os.path.exists(dataset_path): - with open(dataset_path) as f: - data = json.load(f) + data = read_json(dataset_path) units_count = len(data.get("units", [])) print(f" Zig parser complete: {units_count} units", file=sys.stderr) diff --git a/libs/openant-core/core/reporter.py b/libs/openant-core/core/reporter.py index 7153dab1..9536c4de 100644 --- a/libs/openant-core/core/reporter.py +++ b/libs/openant-core/core/reporter.py @@ -19,6 +19,7 @@ from pathlib import Path from core.schemas import ReportResult +from utilities.file_io import open_utf8, read_json, write_json # Root of openant-core _CORE_ROOT = Path(__file__).parent.parent @@ -34,8 +35,7 @@ def _load_diff_metadata(scan_dir: str) -> dict | None: if not os.path.exists(manifest_path): return None try: - with open(manifest_path) as f: - manifest = json.load(f) + manifest = read_json(manifest_path) except (json.JSONDecodeError, OSError): return None out = { @@ -50,8 +50,7 @@ def _load_diff_metadata(scan_dir: str) -> dict | None: filter_report = os.path.join(scan_dir, "diff_filter.report.json") if os.path.exists(filter_report): try: - with open(filter_report) as f: - stats = json.load(f) + stats = read_json(filter_report) out["units_in_diff"] = stats.get("selected") out["units_total_parsed"] = stats.get("total") out["callers_added"] = stats.get("callers_added") or 0 @@ -129,8 +128,7 @@ def _dedup_caller_callee( return confirmed try: - with open(call_graph_path) as f: - cg_data = json.load(f) + cg_data = read_json(call_graph_path) except (json.JSONDecodeError, OSError): return confirmed @@ -212,9 +210,7 @@ def build_pipeline_output( """ print(f"[Report] Building pipeline_output.json...", file=sys.stderr) - with open(results_path) as f: - experiment = json.load(f) - + experiment = read_json(results_path) all_results = experiment.get("results", []) code_by_route = experiment.get("code_by_route", {}) metrics = experiment.get("metrics", {}) @@ -371,9 +367,7 @@ def build_pipeline_output( print(_banner, file=sys.stderr) os.makedirs(os.path.dirname(os.path.abspath(output_path)), exist_ok=True) - with open(output_path, "w") as f: - json.dump(pipeline_output, f, indent=2, ensure_ascii=False) - + write_json(output_path, pipeline_output, ensure_ascii=False) print(f" pipeline_output.json: {len(findings_data)} findings", file=sys.stderr) print(f" Written to {output_path}", file=sys.stderr) @@ -469,9 +463,7 @@ def generate_summary_report( print("[Report] Generating summary report (LLM)...", file=sys.stderr) - with open(results_path) as f: - pipeline_data = json.load(f) - + pipeline_data = read_json(results_path) # Merge dynamic test results if available pipeline_data = merge_dynamic_results(pipeline_data, results_path) @@ -483,7 +475,7 @@ def generate_summary_report( report_text, usage = _generate_summary(pipeline_data) os.makedirs(os.path.dirname(os.path.abspath(output_path)), exist_ok=True) - with open(output_path, "w") as f: + with open_utf8(output_path, "w") as f: f.write(report_text) print(f" Summary report: {output_path}", file=sys.stderr) @@ -517,9 +509,7 @@ def generate_disclosure_docs( print("[Report] Generating disclosure documents (LLM)...", file=sys.stderr) - with open(results_path) as f: - pipeline_data = json.load(f) - + pipeline_data = read_json(results_path) # Merge dynamic test results if available pipeline_data = merge_dynamic_results(pipeline_data, results_path) @@ -552,7 +542,7 @@ def _one(args): safe_name = finding["short_name"].replace(" ", "_").upper() filename = f"DISCLOSURE_{i:02d}_{safe_name}.md" filepath = os.path.join(output_dir, filename) - with open(filepath, "w") as f: + with open_utf8(filepath, "w") as f: f.write(disclosure_text) return finding["short_name"], filepath, usage diff --git a/libs/openant-core/core/scanner.py b/libs/openant-core/core/scanner.py index f0813529..2eba6eeb 100644 --- a/libs/openant-core/core/scanner.py +++ b/libs/openant-core/core/scanner.py @@ -27,6 +27,7 @@ ) from core.step_report import step_context from core import tracking +from utilities.file_io import read_json # Import app context generator (optional) try: @@ -149,8 +150,7 @@ def _step_label(name: str) -> str: _diff_report = os.path.join(output_dir, "diff_filter.report.json") if os.path.exists(_diff_report): try: - with open(_diff_report) as _f: - ctx.summary["diff_stats"] = json.load(_f) + ctx.summary["diff_stats"] = read_json(_diff_report) except (json.JSONDecodeError, OSError): pass ctx.outputs = { @@ -542,8 +542,7 @@ def _load_step_report(output_dir: str, step: str) -> dict: """Load a step report JSON from disk. Returns empty dict on failure.""" path = os.path.join(output_dir, f"{step}.report.json") try: - with open(path) as f: - return json.load(f) + return read_json(path) except Exception: return {"step": step, "status": "unknown"} @@ -551,8 +550,7 @@ def _load_step_report(output_dir: str, step: str) -> dict: def _read_app_type(app_context_path: str) -> str | None: """Read application_type from an app context JSON file.""" try: - with open(app_context_path) as f: - data = json.load(f) + data = read_json(app_context_path) return data.get("application_type") except Exception: return None diff --git a/libs/openant-core/core/schemas.py b/libs/openant-core/core/schemas.py index 88d30d45..43886ebf 100644 --- a/libs/openant-core/core/schemas.py +++ b/libs/openant-core/core/schemas.py @@ -10,12 +10,13 @@ standardized metadata (timing, cost, inputs, outputs). """ -import json import os from dataclasses import dataclass, field, asdict from datetime import datetime, timezone from typing import Any +from utilities.file_io import write_json + # --------------------------------------------------------------------------- # JSON Envelope @@ -268,6 +269,5 @@ def write(self, output_dir: str) -> str: """Write ``{step}.report.json`` to *output_dir*. Returns the path.""" os.makedirs(output_dir, exist_ok=True) path = os.path.join(output_dir, f"{self.step}.report.json") - with open(path, "w") as f: - json.dump(self.to_dict(), f, indent=2) + write_json(path, self.to_dict()) return path diff --git a/libs/openant-core/core/verifier.py b/libs/openant-core/core/verifier.py index fa7a43f2..705ca4a3 100644 --- a/libs/openant-core/core/verifier.py +++ b/libs/openant-core/core/verifier.py @@ -20,6 +20,7 @@ from core.progress import ProgressReporter from utilities.llm_client import TokenTracker, get_global_tracker +from utilities.file_io import read_json, write_json from utilities.finding_verifier import FindingVerifier from utilities.agentic_enhancer.repository_index import load_index_from_file @@ -80,9 +81,7 @@ def run_verification( # Load Stage 1 results print(f"[Verify] Loading results: {results_path}", file=sys.stderr) - with open(results_path) as f: - experiment = json.load(f) - + experiment = read_json(results_path) all_results = experiment.get("results", []) code_by_route = experiment.get("code_by_route", {}) @@ -268,10 +267,7 @@ def _write_verified_results( output["metrics"] = {"total": len(merged_results), **counts} - with open(path, "w") as f: - json.dump(output, f, indent=2, ensure_ascii=False) - - + write_json(path, output, ensure_ascii=False) def _build_code_by_route(results: list) -> dict: """Build code_by_route from result entries (fallback).""" code_by_route = {} diff --git a/libs/openant-core/experiment.py b/libs/openant-core/experiment.py index 359d41f1..7eb8dda5 100644 --- a/libs/openant-core/experiment.py +++ b/libs/openant-core/experiment.py @@ -35,6 +35,7 @@ from pathlib import Path from utilities.llm_client import AnthropicClient, get_global_tracker +from utilities.file_io import read_json, write_json from prompts.prompt_selector import get_analysis_prompt from prompts.vulnerability_analysis import get_system_prompt as get_stage1_system_prompt from utilities.context_corrector import ContextCorrector @@ -211,8 +212,7 @@ def load_dataset(name: str, enhanced: bool = False) -> dict: if not path or not os.path.exists(path): raise ValueError(f"Dataset not found: {name} (enhanced={enhanced})") - with open(path, "r") as f: - return json.load(f) + return read_json(path) def load_ground_truth(name: str) -> dict: @@ -221,8 +221,7 @@ def load_ground_truth(name: str) -> dict: if not path or not os.path.exists(path): return {} - with open(path, "r") as f: - return json.load(f) + return read_json(path) def get_ground_truth_verdict(ground_truth: dict, route_key: str) -> str: @@ -1034,9 +1033,7 @@ def main(): suffix = "" if args.no_enhanced else "_enhanced" output_path = f"experiment_{args.dataset}_{args.model}{suffix}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json" - with open(output_path, "w") as f: - json.dump(experiment, f, indent=2) - + write_json(output_path, experiment) print() print(f"Results saved to: {output_path}") diff --git a/libs/openant-core/export_csv.py b/libs/openant-core/export_csv.py index 8b693005..b330a456 100644 --- a/libs/openant-core/export_csv.py +++ b/libs/openant-core/export_csv.py @@ -29,6 +29,7 @@ import json import os import sys +from utilities.file_io import read_json def _load_diff_block(experiment_path: str) -> dict | None: @@ -41,8 +42,7 @@ def _load_diff_block(experiment_path: str) -> dict | None: if not os.path.exists(candidate): return None try: - with open(candidate) as f: - data = json.load(f) + data = read_json(candidate) except (json.JSONDecodeError, OSError): return None diff = data.get("diff") @@ -67,8 +67,7 @@ def _format_diff_banner(diff: dict) -> str: def load_json(path: str) -> dict: """Load JSON file.""" - with open(path, 'r') as f: - return json.load(f) + return read_json(path) def extract_file(unit_id: str) -> str: diff --git a/libs/openant-core/generate_report.py b/libs/openant-core/generate_report.py index 633cd9b1..5af97f9e 100644 --- a/libs/openant-core/generate_report.py +++ b/libs/openant-core/generate_report.py @@ -31,6 +31,7 @@ import anthropic from dotenv import load_dotenv +from utilities.file_io import read_json # Load environment variables from .env file load_dotenv() @@ -42,8 +43,7 @@ def load_json(path: str) -> dict: """Load JSON file.""" - with open(path, 'r') as f: - return json.load(f) + return read_json(path) def extract_file(unit_id: str) -> str: @@ -83,8 +83,7 @@ def _load_pipeline_metadata(experiment_path: str) -> tuple[dict | None, dict | N if not os.path.exists(candidate): return None, None try: - with open(candidate, 'r') as f: - data = json.load(f) + data = read_json(candidate) except (json.JSONDecodeError, OSError): return None, None return data.get("repository"), data.get("diff") diff --git a/libs/openant-core/openant/cli.py b/libs/openant-core/openant/cli.py index b0ce3455..e521b22f 100644 --- a/libs/openant-core/openant/cli.py +++ b/libs/openant-core/openant/cli.py @@ -22,6 +22,8 @@ import sys import tempfile +from utilities.file_io import read_json + def _output_json(data: dict): """Write JSON to stdout.""" @@ -39,8 +41,7 @@ def _load_step_reports(directory: str) -> list[dict]: reports = [] for path in glob.glob(os.path.join(directory, "*.report.json")): try: - with open(path) as f: - reports.append(json.load(f)) + reports.append(read_json(path)) except (json.JSONDecodeError, OSError): continue return reports @@ -82,8 +83,7 @@ def cmd_scan(args): # is the same one written into pipeline_output.json by reporter.py. if result.pipeline_output_path and os.path.exists(result.pipeline_output_path): try: - with open(result.pipeline_output_path) as f: - po = json.load(f) + po = read_json(result.pipeline_output_path) diff_block = po.get("diff") if isinstance(diff_block, dict) and diff_block.get("mode") == "incremental": scan_payload["diff"] = diff_block @@ -135,8 +135,7 @@ def cmd_parse(args): diff_report = os.path.join(output_dir, "diff_filter.report.json") if os.path.exists(diff_report): try: - with open(diff_report) as f: - ctx.summary["diff_stats"] = json.load(f) + ctx.summary["diff_stats"] = read_json(diff_report) except (json.JSONDecodeError, OSError): pass ctx.outputs = { @@ -607,10 +606,8 @@ def cmd_report_data(args): "dataset_path": os.path.abspath(dataset_path), }) as ctx: # Load data - with open(results_path) as f: - experiment = json.load(f) - with open(dataset_path) as f: - dataset = json.load(f) + experiment = read_json(results_path) + dataset = read_json(dataset_path) # --- Load dynamic test results if available --- # Dynamic tests use VULN-XXX IDs from pipeline_output.json, @@ -620,10 +617,8 @@ def cmd_report_data(args): dt_path = os.path.join(results_dir, "dynamic_test_results.json") po_path = os.path.join(results_dir, "pipeline_output.json") if os.path.exists(dt_path) and os.path.exists(po_path): - with open(dt_path) as f: - dt_data = json.load(f) - with open(po_path) as f: - po_data = json.load(f) + dt_data = read_json(dt_path) + po_data = read_json(po_path) # Map VULN-ID → route_key from pipeline_output vuln_id_to_route = {} @@ -876,8 +871,7 @@ def _linkify_finding(m): diff_block = None if os.path.exists(po_path): try: - with open(po_path) as f: - po = json.load(f) + po = read_json(po_path) repo_info = po.get("repository", {}) repo_name = repo_info.get("name", "") commit_sha = repo_info.get("commit_sha", "") diff --git a/libs/openant-core/parsers/c/call_graph_builder.py b/libs/openant-core/parsers/c/call_graph_builder.py index 84e59881..f5940bad 100644 --- a/libs/openant-core/parsers/c/call_graph_builder.py +++ b/libs/openant-core/parsers/c/call_graph_builder.py @@ -40,6 +40,7 @@ import tree_sitter_c as tsc import tree_sitter_cpp as tscpp from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 C_LANGUAGE = Language(tsc.language()) @@ -423,9 +424,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - extractor_output = json.load(f) - + extractor_output = read_json(args.input_file) print(f"Processing {len(extractor_output.get('functions', {}))} functions...", file=sys.stderr) builder = CallGraphBuilder(extractor_output, {'max_depth': args.depth}) @@ -444,7 +443,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Output written to: {args.output}", file=sys.stderr) else: diff --git a/libs/openant-core/parsers/c/function_extractor.py b/libs/openant-core/parsers/c/function_extractor.py index 10b5f70f..8e5b1cf1 100644 --- a/libs/openant-core/parsers/c/function_extractor.py +++ b/libs/openant-core/parsers/c/function_extractor.py @@ -42,6 +42,7 @@ import tree_sitter_c as tsc import tree_sitter_cpp as tscpp from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 C_LANGUAGE = Language(tsc.language()) @@ -575,8 +576,7 @@ def main(): extractor = FunctionExtractor(args.repo_path) if args.scan_file: - with open(args.scan_file) as f: - scan_result = json.load(f) + scan_result = read_json(args.scan_file) result = extractor.extract_from_scan(scan_result) else: result = extractor.extract_all() @@ -584,7 +584,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Extraction complete. Results written to: {args.output}", file=sys.stderr) print(f"Total functions: {result['statistics']['total_functions']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/c/repository_scanner.py b/libs/openant-core/parsers/c/repository_scanner.py index 6706f926..a6ec2418 100644 --- a/libs/openant-core/parsers/c/repository_scanner.py +++ b/libs/openant-core/parsers/c/repository_scanner.py @@ -30,6 +30,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 class RepositoryScanner: @@ -225,7 +226,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Scan complete. Results written to: {args.output}", file=sys.stderr) print(f"Total files found: {result['statistics']['total_files']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/c/test_pipeline.py b/libs/openant-core/parsers/c/test_pipeline.py index 3f186359..5072d680 100644 --- a/libs/openant-core/parsers/c/test_pipeline.py +++ b/libs/openant-core/parsers/c/test_pipeline.py @@ -42,6 +42,7 @@ from enum import Enum from pathlib import Path from typing import Set +from utilities.file_io import open_utf8, read_json, run_utf8, write_json # Add parent directory to path for utilities import sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) @@ -139,8 +140,7 @@ def run_parser_pipeline(self) -> bool: # Save scan results self.scan_results_file = os.path.join(self.output_dir, 'scan_results.json') - with open(self.scan_results_file, 'w') as f: - json.dump(scan_result, f, indent=2) + write_json(self.scan_results_file, scan_result) # Stage 2: Extract functions print(" [2/4] Extracting functions via tree-sitter...") @@ -178,13 +178,11 @@ def run_parser_pipeline(self) -> bool: print(f" Avg upstream deps: {dataset['statistics']['avg_upstream']}") # Write dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) # Write analyzer output analyzer_output = generator.generate_analyzer_output() - with open(self.analyzer_output_file, 'w') as f: - json.dump(analyzer_output, f, indent=2) + write_json(self.analyzer_output_file, analyzer_output) elapsed = (datetime.now() - start_time).total_seconds() @@ -242,8 +240,7 @@ def apply_reachability_filter(self) -> bool: start_time = datetime.now() try: - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) @@ -262,8 +259,7 @@ def apply_reachability_filter(self) -> bool: } # Build call graph from dataset unit metadata - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) call_graph = {} reverse_call_graph = {} @@ -313,8 +309,7 @@ def apply_reachability_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -379,7 +374,7 @@ def run_codeql_analysis(self) -> bool: '--overwrite' ] - result = subprocess.run( + result = run_utf8( create_db_cmd, capture_output=True, text=True, @@ -410,7 +405,7 @@ def run_codeql_analysis(self) -> bool: f'codeql/{language}-queries:codeql-suites/{language}-security-extended.qls' ] - result = subprocess.run( + result = run_utf8( analyze_cmd, capture_output=True, text=True, @@ -443,8 +438,7 @@ def run_codeql_analysis(self) -> bool: } return False - with open(sarif_output, 'r') as f: - sarif_data = json.load(f) + sarif_data = read_json(sarif_output) self.codeql_findings = [] @@ -555,8 +549,7 @@ def apply_codeql_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Build mapping of file -> [(start_line, end_line, func_id)] file_functions = {} @@ -605,8 +598,7 @@ def apply_codeql_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -662,8 +654,7 @@ def run_context_enhancer(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) enhancer = ContextEnhancer() @@ -695,8 +686,7 @@ def run_context_enhancer(self) -> bool: 'data_flows_extracted': enhancer.stats['data_flows_extracted'] } - with open(self.dataset_file, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(self.dataset_file, enhanced) elapsed = (datetime.now() - start_time).total_seconds() @@ -740,8 +730,7 @@ def apply_exploitable_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -767,8 +756,7 @@ def apply_exploitable_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -908,7 +896,7 @@ def run_full_pipeline(self): # Save results summary results_file = os.path.join(self.output_dir, 'pipeline_results.json') - with open(results_file, 'w') as f: + with open_utf8(results_file, 'w') as f: clean_results = { 'repository': self.results['repository'], 'test_time': self.results['test_time'], diff --git a/libs/openant-core/parsers/c/unit_generator.py b/libs/openant-core/parsers/c/unit_generator.py index a0391d75..fcca5065 100644 --- a/libs/openant-core/parsers/c/unit_generator.py +++ b/libs/openant-core/parsers/c/unit_generator.py @@ -28,6 +28,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 # File boundary marker for enhanced code (C-style comment, matching Go parser) @@ -343,9 +344,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - call_graph_data = json.load(f) - + call_graph_data = read_json(args.input_file) options = { 'max_depth': args.depth, } @@ -373,7 +372,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"\nOutput written to: {args.output}", file=sys.stderr) else: @@ -382,8 +381,7 @@ def main(): # Write analyzer output if requested if args.analyzer_output: analyzer = generator.generate_analyzer_output() - with open(args.analyzer_output, 'w') as f: - json.dump(analyzer, f, indent=2) + write_json(args.analyzer_output, analyzer) print(f"Analyzer output written to: {args.analyzer_output}", file=sys.stderr) except Exception as e: diff --git a/libs/openant-core/parsers/go/test_pipeline.py b/libs/openant-core/parsers/go/test_pipeline.py index 8fe05b88..7e2aa118 100644 --- a/libs/openant-core/parsers/go/test_pipeline.py +++ b/libs/openant-core/parsers/go/test_pipeline.py @@ -43,6 +43,39 @@ from pathlib import Path from typing import Set +# Add parent directories to path so utilities can be found when run as a subprocess +_parser_dir = Path(__file__).parent +_core_root = _parser_dir.parent.parent +if str(_core_root) not in sys.path: + sys.path.insert(0, str(_core_root)) + +from utilities.file_io import open_utf8, read_json, run_utf8, write_json + + +def _stdout_supports_unicode() -> bool: + """Return True if sys.stdout can emit the symbols we use for status. + + Returns False when stdout is piped or redirected (common in CI) and + the encoding cannot be determined — this degrades output to plain ASCII + rather than raising UnicodeEncodeError at runtime. + """ + encoding = getattr(sys.stdout, "encoding", None) + if not encoding: + return False + try: + # Probe with the actual symbols we emit. This catches cp1252 and + # other limited code pages without us having to enumerate them. + "✓✗→".encode(encoding) + return True + except (UnicodeEncodeError, LookupError): + return False + + +_UNICODE_OK = _stdout_supports_unicode() +SYM_OK = "✓" if _UNICODE_OK else "OK" +SYM_FAIL = "✗" if _UNICODE_OK else "FAIL" +SYM_ARROW = "→" if _UNICODE_OK else "->" + # Add parent directory to path for utilities import sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) from utilities.context_enhancer import ContextEnhancer @@ -115,11 +148,11 @@ def setup(self): if not os.path.exists(self.go_parser): print("Building Go parser...") go_parser_dir = os.path.join(self.parser_dir, 'go_parser') - result = subprocess.run( + result = run_utf8( ['go', 'build', '-o', 'go_parser', '.'], cwd=go_parser_dir, capture_output=True, - text=True + text=True, ) if result.returncode != 0: print(f"Error building Go parser: {result.stderr}") @@ -140,7 +173,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: start_time = datetime.now() try: - result = subprocess.run( + result = run_utf8( command, capture_output=True, text=True, @@ -158,7 +191,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: } if result.returncode == 0: - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print() # Print stderr (often contains summary info) if result.stderr: @@ -168,11 +201,10 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: # Load and summarize output if os.path.exists(output_file): - with open(output_file, 'r') as f: - data = json.load(f) + data = read_json(output_file) stage_result['summary'] = self._summarize_output(name, data) else: - print(f"✗ Failed (exit code {result.returncode})") + print(f"{SYM_FAIL} Failed (exit code {result.returncode})") print() if result.stderr: print("STDERR:") @@ -185,7 +217,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") return { 'success': False, 'elapsed_seconds': elapsed, @@ -244,11 +276,9 @@ def run_go_parser_all(self) -> bool: # Post-process: apply dataset name if specified (Go binary doesn't support --name) if result.get('success', False) and self.dataset_name and os.path.exists(self.dataset_file): try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) dataset['name'] = self.dataset_name - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) except Exception as e: print(f"Warning: Could not apply dataset name: {e}") @@ -282,8 +312,7 @@ def apply_reachability_filter(self) -> bool: try: # Load analyzer output for call graph - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) @@ -304,8 +333,7 @@ def apply_reachability_filter(self) -> bool: } # Load call graph from dataset (go_parser puts it in statistics) - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Build call graph from unit metadata call_graph = {} @@ -359,8 +387,7 @@ def apply_reachability_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -378,9 +405,9 @@ def apply_reachability_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Entry points detected: {len(self.entry_points)}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['reachability_filter'] = result @@ -388,7 +415,7 @@ def apply_reachability_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -434,7 +461,7 @@ def run_codeql_analysis(self) -> bool: '--overwrite' ] - result = subprocess.run( + result = run_utf8( create_db_cmd, capture_output=True, text=True, @@ -442,7 +469,7 @@ def run_codeql_analysis(self) -> bool: ) if result.returncode != 0: - print(f"✗ CodeQL database creation failed") + print(f"{SYM_FAIL} CodeQL database creation failed") print(f" stderr: {result.stderr[:500] if result.stderr else 'none'}") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { @@ -465,7 +492,7 @@ def run_codeql_analysis(self) -> bool: f'codeql/{language}-queries:codeql-suites/{language}-security-extended.qls' ] - result = subprocess.run( + result = run_utf8( analyze_cmd, capture_output=True, text=True, @@ -473,7 +500,7 @@ def run_codeql_analysis(self) -> bool: ) if result.returncode != 0: - print(f"✗ CodeQL analysis failed") + print(f"{SYM_FAIL} CodeQL analysis failed") print(f" stderr: {result.stderr[:500] if result.stderr else 'none'}") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { @@ -489,7 +516,7 @@ def run_codeql_analysis(self) -> bool: # Step 3: Parse SARIF output print("Parsing results...") if not os.path.exists(sarif_output): - print("✗ SARIF output not found") + print(f"{SYM_FAIL} SARIF output not found") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { 'success': False, @@ -498,8 +525,7 @@ def run_codeql_analysis(self) -> bool: } return False - with open(sarif_output, 'r') as f: - sarif_data = json.load(f) + sarif_data = read_json(sarif_output) # Extract findings and map to file:line self.codeql_findings = [] @@ -550,7 +576,7 @@ def run_codeql_analysis(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Total findings: {len(self.codeql_findings)}") print(f" Unique files: {summary['unique_files']}") if summary['by_level']: @@ -562,7 +588,7 @@ def run_codeql_analysis(self) -> bool: except FileNotFoundError: elapsed = (datetime.now() - start_time).total_seconds() - print("✗ CodeQL not found. Please install CodeQL CLI.") + print(f"{SYM_FAIL} CodeQL not found. Please install CodeQL CLI.") print(" See: https://docs.github.com/en/code-security/codeql-cli") self.results['stages']['codeql_analysis'] = { 'success': False, @@ -573,7 +599,7 @@ def run_codeql_analysis(self) -> bool: except subprocess.TimeoutExpired: elapsed = (datetime.now() - start_time).total_seconds() - print("✗ CodeQL analysis timed out") + print(f"{SYM_FAIL} CodeQL analysis timed out") self.results['stages']['codeql_analysis'] = { 'success': False, 'elapsed_seconds': elapsed, @@ -583,7 +609,7 @@ def run_codeql_analysis(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() self.results['stages']['codeql_analysis'] = { @@ -620,8 +646,7 @@ def apply_codeql_filter(self) -> bool: try: # Load dataset to get function line ranges - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Build mapping of file -> [(start_line, end_line, func_id)] file_functions = {} @@ -675,8 +700,7 @@ def apply_codeql_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -695,10 +719,10 @@ def apply_codeql_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" CodeQL findings: {len(self.codeql_findings)}") print(f" Flagged function units: {len(self.codeql_flagged_units)}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['codeql_filter'] = result @@ -706,7 +730,7 @@ def apply_codeql_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -733,8 +757,7 @@ def run_context_enhancer(self) -> bool: try: # Load dataset - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Enhance with LLM enhancer = ContextEnhancer() @@ -771,8 +794,7 @@ def run_context_enhancer(self) -> bool: } # Write back - with open(self.dataset_file, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(self.dataset_file, enhanced) elapsed = (datetime.now() - start_time).total_seconds() @@ -784,14 +806,14 @@ def run_context_enhancer(self) -> bool: } print() - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") self.results['stages']['context_enhancer'] = result return True except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -824,8 +846,7 @@ def apply_exploitable_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -854,8 +875,7 @@ def apply_exploitable_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -873,12 +893,12 @@ def apply_exploitable_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Classification breakdown:") for cls, count in sorted(classification_counts.items()): - marker = "→" if cls == "exploitable" else " " + marker = SYM_ARROW if cls == "exploitable" else " " print(f" {marker} {cls}: {count}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['exploitable_filter'] = result @@ -886,7 +906,7 @@ def apply_exploitable_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -966,13 +986,13 @@ def run_full_pipeline(self): self.results['success'] = all_success if all_success: - print("✓ All stages completed successfully") + print(f"{SYM_OK} All stages completed successfully") else: - print("✗ Some stages failed") + print(f"{SYM_FAIL} Some stages failed") print() for stage_name, stage_result in self.results['stages'].items(): - status = "✓" if stage_result.get('success') else "✗" + status = SYM_OK if stage_result.get('success') else SYM_FAIL elapsed = stage_result.get('elapsed_seconds', 0) print(f" {status} {stage_name}: {elapsed:.2f}s") @@ -1002,7 +1022,7 @@ def run_full_pipeline(self): # Save results summary results_file = os.path.join(self.output_dir, 'pipeline_results.json') - with open(results_file, 'w') as f: + with open_utf8(results_file, 'w') as f: # Remove stdout/stderr from saved results (too verbose) clean_results = { 'repository': self.results['repository'], diff --git a/libs/openant-core/parsers/javascript/path_utils.js b/libs/openant-core/parsers/javascript/path_utils.js new file mode 100644 index 00000000..c89c273b --- /dev/null +++ b/libs/openant-core/parsers/javascript/path_utils.js @@ -0,0 +1,28 @@ +"use strict"; + +/** + * Convert a filesystem path to forward slashes. + * + * ts-morph stores source-file paths internally with forward slashes and + * also treats backslashes as escape characters when matching paths it + * has already added. On Windows, Node's `path.relative()` and + * `path.resolve()` return backslash-separated paths, which causes + * ts-morph to silently fail to find any files (resulting in 0 functions + * extracted). Always normalise to forward slashes before handing paths + * to ts-morph or storing them as functionId components. + * + * UNC paths (`\\server\share\...`) are correctly converted to + * `//server/share/...`, which TypeScript understands on Windows. + * + * CONTRACT FOR CONTRIBUTORS: every path that will be passed to ts-morph + * (addSourceFileAtPath, getSourceFile, etc.) or stored as a functionId + * component MUST go through toPosixPath() first. This applies to the + * result of any path.resolve(), path.relative(), or path.join() call. + * Skipping this step silently breaks Windows: ts-morph finds zero files + * and the analyzer emits an empty result without an error. + */ +function toPosixPath(p) { + return p.replace(/\\/g, "/"); +} + +module.exports = { toPosixPath }; diff --git a/libs/openant-core/parsers/javascript/test_pipeline.py b/libs/openant-core/parsers/javascript/test_pipeline.py index 77ab9c43..667bf1fd 100644 --- a/libs/openant-core/parsers/javascript/test_pipeline.py +++ b/libs/openant-core/parsers/javascript/test_pipeline.py @@ -42,6 +42,39 @@ from pathlib import Path from typing import Set, Tuple +# Add parent directories to path so utilities can be found when run as a subprocess +_parser_dir = Path(__file__).parent +_core_root = _parser_dir.parent.parent +if str(_core_root) not in sys.path: + sys.path.insert(0, str(_core_root)) + +from utilities.file_io import open_utf8, read_json, run_utf8, write_json + + +def _stdout_supports_unicode() -> bool: + """Return True if sys.stdout can emit the symbols we use for status. + + Returns False when stdout is piped or redirected (common in CI) and + the encoding cannot be determined — this degrades output to plain ASCII + rather than raising UnicodeEncodeError at runtime. + """ + encoding = getattr(sys.stdout, "encoding", None) + if not encoding: + return False + try: + # Probe with the actual symbols we emit. This catches cp1252 and + # other limited code pages without us having to enumerate them. + "✓✗→".encode(encoding) + return True + except (UnicodeEncodeError, LookupError): + return False + + +_UNICODE_OK = _stdout_supports_unicode() +SYM_OK = "✓" if _UNICODE_OK else "OK" +SYM_FAIL = "✗" if _UNICODE_OK else "FAIL" +SYM_ARROW = "→" if _UNICODE_OK else "->" + # Add parent directory to path for utilities import sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) from utilities.context_enhancer import ContextEnhancer @@ -126,7 +159,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: start_time = datetime.now() try: - result = subprocess.run( + result = run_utf8( command, capture_output=True, text=True, @@ -144,7 +177,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: } if result.returncode == 0: - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print() # Print stderr (often contains summary info) if result.stderr: @@ -154,11 +187,10 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: # Load and summarize output if os.path.exists(output_file): - with open(output_file, 'r') as f: - data = json.load(f) + data = read_json(output_file) stage_result['summary'] = self._summarize_output(name, data) else: - print(f"✗ Failed (exit code {result.returncode})") + print(f"{SYM_FAIL} Failed (exit code {result.returncode})") print() if result.stderr: print("STDERR:") @@ -171,7 +203,7 @@ def run_stage(self, name: str, command: list, output_file: str) -> dict: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") return { 'success': False, 'elapsed_seconds': elapsed, @@ -242,8 +274,7 @@ def run_typescript_analyzer(self, files: list = None) -> bool: # If no specific files, use ALL files from scan results if not files and self.scan_results_file and os.path.exists(self.scan_results_file): - with open(self.scan_results_file, 'r') as f: - scan_data = json.load(f) + scan_data = read_json(self.scan_results_file) files = [f['path'] for f in scan_data.get('files', [])] if not files: @@ -252,7 +283,7 @@ def run_typescript_analyzer(self, files: list = None) -> bool: # Write file list to a temporary file to avoid command-line length limits file_list_path = os.path.join(self.output_dir, 'file_list.txt') - with open(file_list_path, 'w') as f: + with open_utf8(file_list_path, 'w') as f: for file_path in files: # Convert relative path to absolute if not os.path.isabs(file_path): @@ -289,7 +320,7 @@ def run_stage_with_stdout_capture(self, name: str, command: list, output_file: s start_time = datetime.now() try: - result = subprocess.run( + result = run_utf8( command, capture_output=True, text=True, @@ -300,10 +331,10 @@ def run_stage_with_stdout_capture(self, name: str, command: list, output_file: s if result.returncode == 0: # Write stdout to output file - with open(output_file, 'w') as f: + with open_utf8(output_file, 'w') as f: f.write(result.stdout) - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print() # Print stderr (often contains summary info) if result.stderr: @@ -313,8 +344,7 @@ def run_stage_with_stdout_capture(self, name: str, command: list, output_file: s # Load and summarize output if os.path.exists(output_file): - with open(output_file, 'r') as f: - data = json.load(f) + data = read_json(output_file) summary = self._summarize_output(name, data) else: summary = {} @@ -327,7 +357,7 @@ def run_stage_with_stdout_capture(self, name: str, command: list, output_file: s 'stderr': result.stderr } else: - print(f"✗ Failed (exit code {result.returncode})") + print(f"{SYM_FAIL} Failed (exit code {result.returncode})") print() if result.stderr: print("STDERR:") @@ -345,7 +375,7 @@ def run_stage_with_stdout_capture(self, name: str, command: list, output_file: s except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") return { 'success': False, 'elapsed_seconds': elapsed, @@ -391,8 +421,7 @@ def run_context_enhancer(self) -> bool: try: # Load dataset - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Enhance with LLM enhancer = ContextEnhancer() @@ -432,8 +461,7 @@ def run_context_enhancer(self) -> bool: } # Write back - with open(self.dataset_file, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(self.dataset_file, enhanced) elapsed = (datetime.now() - start_time).total_seconds() @@ -445,14 +473,14 @@ def run_context_enhancer(self) -> bool: } print() - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") self.results['stages']['context_enhancer'] = result return True except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -490,8 +518,7 @@ def apply_reachability_filter(self) -> bool: try: # Load analyzer output for call graph - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) call_graph = analyzer.get("call_graph", analyzer.get("callGraph", {})) @@ -510,8 +537,7 @@ def apply_reachability_filter(self) -> bool: self.reachable_units = reachability.get_all_reachable() # Load and filter dataset - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -539,8 +565,7 @@ def apply_reachability_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -558,9 +583,9 @@ def apply_reachability_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Entry points detected: {len(self.entry_points)}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['reachability_filter'] = result @@ -568,7 +593,7 @@ def apply_reachability_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -590,8 +615,7 @@ def _detect_codeql_language(self) -> str: return "javascript" # Default try: - with open(self.scan_results_file, 'r') as f: - scan_data = json.load(f) + scan_data = read_json(self.scan_results_file) stats = scan_data.get('statistics', {}) by_extension = stats.get('byExtension', {}) @@ -642,7 +666,7 @@ def run_codeql_analysis(self) -> bool: '--overwrite' ] - result = subprocess.run( + result = run_utf8( create_db_cmd, capture_output=True, text=True, @@ -650,7 +674,7 @@ def run_codeql_analysis(self) -> bool: ) if result.returncode != 0: - print(f"✗ CodeQL database creation failed") + print(f"{SYM_FAIL} CodeQL database creation failed") print(f" stderr: {result.stderr[:500] if result.stderr else 'none'}") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { @@ -673,7 +697,7 @@ def run_codeql_analysis(self) -> bool: f'codeql/{language}-queries:codeql-suites/{language}-security-extended.qls' ] - result = subprocess.run( + result = run_utf8( analyze_cmd, capture_output=True, text=True, @@ -681,7 +705,7 @@ def run_codeql_analysis(self) -> bool: ) if result.returncode != 0: - print(f"✗ CodeQL analysis failed") + print(f"{SYM_FAIL} CodeQL analysis failed") print(f" stderr: {result.stderr[:500] if result.stderr else 'none'}") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { @@ -697,7 +721,7 @@ def run_codeql_analysis(self) -> bool: # Step 3: Parse SARIF output print("Parsing results...") if not os.path.exists(sarif_output): - print("✗ SARIF output not found") + print(f"{SYM_FAIL} SARIF output not found") elapsed = (datetime.now() - start_time).total_seconds() self.results['stages']['codeql_analysis'] = { 'success': False, @@ -706,8 +730,7 @@ def run_codeql_analysis(self) -> bool: } return False - with open(sarif_output, 'r') as f: - sarif_data = json.load(f) + sarif_data = read_json(sarif_output) # Extract findings and map to file:line self.codeql_findings = [] @@ -760,7 +783,7 @@ def run_codeql_analysis(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Total findings: {len(self.codeql_findings)}") print(f" Unique files: {summary['unique_files']}") if summary['by_level']: @@ -772,7 +795,7 @@ def run_codeql_analysis(self) -> bool: except FileNotFoundError: elapsed = (datetime.now() - start_time).total_seconds() - print("✗ CodeQL not found. Please install CodeQL CLI.") + print(f"{SYM_FAIL} CodeQL not found. Please install CodeQL CLI.") print(" See: https://docs.github.com/en/code-security/codeql-cli") self.results['stages']['codeql_analysis'] = { 'success': False, @@ -783,7 +806,7 @@ def run_codeql_analysis(self) -> bool: except subprocess.TimeoutExpired: elapsed = (datetime.now() - start_time).total_seconds() - print("✗ CodeQL analysis timed out") + print(f"{SYM_FAIL} CodeQL analysis timed out") self.results['stages']['codeql_analysis'] = { 'success': False, 'elapsed_seconds': elapsed, @@ -793,7 +816,7 @@ def run_codeql_analysis(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() self.results['stages']['codeql_analysis'] = { @@ -830,8 +853,7 @@ def apply_codeql_filter(self) -> bool: try: # Load analyzer output to get function line ranges - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) @@ -869,8 +891,7 @@ def apply_codeql_filter(self) -> bool: self.codeql_flagged_units.add(func_id) # Load and filter dataset - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -891,8 +912,7 @@ def apply_codeql_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -911,10 +931,10 @@ def apply_codeql_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" CodeQL findings: {len(self.codeql_findings)}") print(f" Flagged function units: {len(self.codeql_flagged_units)}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['codeql_filter'] = result @@ -922,7 +942,7 @@ def apply_codeql_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -955,8 +975,7 @@ def apply_exploitable_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -985,8 +1004,7 @@ def apply_exploitable_filter(self) -> bool: } # Write filtered dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -1004,12 +1022,12 @@ def apply_exploitable_filter(self) -> bool: 'summary': summary } - print(f"✓ Success ({elapsed:.2f}s)") + print(f"{SYM_OK} Success ({elapsed:.2f}s)") print(f" Classification breakdown:") for cls, count in sorted(classification_counts.items()): - marker = "→" if cls == "exploitable" else " " + marker = SYM_ARROW if cls == "exploitable" else " " print(f" {marker} {cls}: {count}") - print(f" Units: {original_count} → {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") + print(f" Units: {original_count} {SYM_ARROW} {len(filtered_units)} ({summary['reduction_percentage']}% reduction)") print() self.results['stages']['exploitable_filter'] = result @@ -1017,7 +1035,7 @@ def apply_exploitable_filter(self) -> bool: except Exception as e: elapsed = (datetime.now() - start_time).total_seconds() - print(f"✗ Error: {e}") + print(f"{SYM_FAIL} Error: {e}") import traceback traceback.print_exc() result = { @@ -1105,13 +1123,13 @@ def run_full_pipeline(self): self.results['success'] = all_success if all_success: - print("✓ All stages completed successfully") + print(f"{SYM_OK} All stages completed successfully") else: - print("✗ Some stages failed") + print(f"{SYM_FAIL} Some stages failed") print() for stage_name, stage_result in self.results['stages'].items(): - status = "✓" if stage_result.get('success') else "✗" + status = SYM_OK if stage_result.get('success') else SYM_FAIL elapsed = stage_result.get('elapsed_seconds', 0) print(f" {status} {stage_name}: {elapsed:.2f}s") @@ -1143,7 +1161,7 @@ def run_full_pipeline(self): # Save results summary results_file = os.path.join(self.output_dir, 'pipeline_results.json') - with open(results_file, 'w') as f: + with open_utf8(results_file, 'w') as f: # Remove stdout/stderr from saved results (too verbose) clean_results = { 'repository': self.results['repository'], diff --git a/libs/openant-core/parsers/javascript/typescript_analyzer.js b/libs/openant-core/parsers/javascript/typescript_analyzer.js index a41a80d8..7121acdb 100644 --- a/libs/openant-core/parsers/javascript/typescript_analyzer.js +++ b/libs/openant-core/parsers/javascript/typescript_analyzer.js @@ -28,6 +28,7 @@ const { Project } = require("ts-morph"); const { ts } = require("@ts-morph/common"); const path = require("path"); +const { toPosixPath } = require("./path_utils"); /** * Maximally permissive compiler options for AST extraction. @@ -50,7 +51,9 @@ const PERMISSIVE_COMPILER_OPTIONS = { class TypeScriptAnalyzer { constructor(repoPath) { - this.repoPath = repoPath; + // Normalise immediately so all later path operations (path.relative, + // path.join) work with a consistent forward-slash base on Windows. + this.repoPath = toPosixPath(path.resolve(repoPath)); this.project = new Project({ compilerOptions: PERMISSIVE_COMPILER_OPTIONS, }); @@ -136,10 +139,15 @@ class TypeScriptAnalyzer { ? filePath : path.join(this.repoPath, filePath); + // ts-morph treats backslashes as escape characters when matching + // paths it has already added. Normalise to forward slashes so + // Windows-native paths (with `\`) resolve consistently. + const normalised = toPosixPath(fullPath); + try { - this.project.addSourceFileAtPath(fullPath); + this.project.addSourceFileAtPath(normalised); } catch (error) { - console.error(`Failed to add file ${fullPath}: ${error.message}`); + console.error(`Failed to add file ${normalised}: ${error.message}`); } } @@ -163,7 +171,12 @@ class TypeScriptAnalyzer { * Extract all functions/methods from a source file */ extractFunctionsFromFile(sourceFile) { - const relativePath = path.relative(this.repoPath, sourceFile.getFilePath()); + // Always emit POSIX-style relative paths so functionId values are + // stable across platforms (Python downstream consumers and dataset + // diffs key off these strings). + const relativePath = toPosixPath( + path.relative(this.repoPath, sourceFile.getFilePath()), + ); // Extract function declarations for (const func of sourceFile.getFunctions()) { @@ -407,7 +420,9 @@ class TypeScriptAnalyzer { * For each function, find what other functions it calls */ buildCallGraphForFile(sourceFile) { - const relativePath = path.relative(this.repoPath, sourceFile.getFilePath()); + const relativePath = toPosixPath( + path.relative(this.repoPath, sourceFile.getFilePath()), + ); // Analyze function declarations for (const func of sourceFile.getFunctions()) { @@ -488,9 +503,13 @@ class TypeScriptAnalyzer { function extractSingleFunction(filePath, functionRef) { const fs = require("fs"); - // Check if file exists - if (!fs.existsSync(filePath)) { - console.error(`File not found: ${filePath}`); + // Normalise to forward slashes so ts-morph can match the path it stores + // internally. On Windows, filePath may arrive with backslashes. + const normalisedFilePath = toPosixPath(path.resolve(filePath)); + + // Check if file exists using the normalised path for consistent error messages. + if (!fs.existsSync(normalisedFilePath)) { + console.error(`File not found: ${normalisedFilePath}`); process.exit(1); } @@ -499,7 +518,7 @@ function extractSingleFunction(filePath, functionRef) { }); try { - const sourceFile = project.addSourceFileAtPath(filePath); + const sourceFile = project.addSourceFileAtPath(normalisedFilePath); // Parse function reference (e.g., "sessionHandler.handleLogin" or just "handleLogin") let className = null; @@ -689,16 +708,21 @@ function extractSingleFunction(filePath, functionRef) { ); if (requireMatch) { const requiredPath = requireMatch[1]; - // Resolve the path relative to current file - const currentDir = path.dirname(filePath); - let resolvedPath = path.resolve(currentDir, requiredPath); + // Resolve the path relative to current file; use the + // already-normalised path to avoid mixed separators. + const currentDir = path.dirname(normalisedFilePath); + let resolvedPath = toPosixPath( + path.resolve(currentDir, requiredPath), + ); // Try with .js extension if not present if (!fs.existsSync(resolvedPath)) { resolvedPath = resolvedPath + ".js"; } if (!fs.existsSync(resolvedPath)) { - resolvedPath = path.resolve(currentDir, requiredPath + ".ts"); + resolvedPath = toPosixPath( + path.resolve(currentDir, requiredPath + ".ts"), + ); } if (fs.existsSync(resolvedPath)) { @@ -889,9 +913,14 @@ if (require.main === module) { process.exit(1); } const content = fs.readFileSync(listFile, "utf-8"); + // Split on either CRLF or LF and trim residual whitespace so + // file lists written on Windows (with \r\n line endings) don't + // leave a trailing \r on each path, which would make + // addSourceFileAtPath fail. filePaths = content - .split("\n") - .filter((line) => line.trim().length > 0); + .split(/\r?\n/) + .map((line) => line.trim()) + .filter((line) => line.length > 0); console.error(`Loaded ${filePaths.length} files from ${listFile}`); i += 2; } else if (args[i] === "--output" && i + 1 < args.length) { diff --git a/libs/openant-core/parsers/php/call_graph_builder.py b/libs/openant-core/parsers/php/call_graph_builder.py index dfa441e0..42e37bbf 100644 --- a/libs/openant-core/parsers/php/call_graph_builder.py +++ b/libs/openant-core/parsers/php/call_graph_builder.py @@ -39,6 +39,7 @@ import tree_sitter_php as ts_php from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 PHP_LANGUAGE = Language(ts_php.language_php()) @@ -482,9 +483,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - extractor_output = json.load(f) - + extractor_output = read_json(args.input_file) print(f"Processing {len(extractor_output.get('functions', {}))} functions...", file=sys.stderr) builder = CallGraphBuilder(extractor_output, {'max_depth': args.depth}) @@ -503,7 +502,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Output written to: {args.output}", file=sys.stderr) else: diff --git a/libs/openant-core/parsers/php/function_extractor.py b/libs/openant-core/parsers/php/function_extractor.py index bdedecf7..2c9039ad 100644 --- a/libs/openant-core/parsers/php/function_extractor.py +++ b/libs/openant-core/parsers/php/function_extractor.py @@ -42,6 +42,7 @@ import tree_sitter_php as ts_php from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 PHP_LANGUAGE = Language(ts_php.language_php()) @@ -547,8 +548,7 @@ def main(): extractor = FunctionExtractor(args.repo_path) if args.scan_file: - with open(args.scan_file) as f: - scan_result = json.load(f) + scan_result = read_json(args.scan_file) result = extractor.extract_from_scan(scan_result) else: result = extractor.extract_all() @@ -556,7 +556,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Extraction complete. Results written to: {args.output}", file=sys.stderr) print(f"Total functions: {result['statistics']['total_functions']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/php/repository_scanner.py b/libs/openant-core/parsers/php/repository_scanner.py index bd8a2d94..89781ffb 100644 --- a/libs/openant-core/parsers/php/repository_scanner.py +++ b/libs/openant-core/parsers/php/repository_scanner.py @@ -30,6 +30,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 class RepositoryScanner: @@ -236,7 +237,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Scan complete. Results written to: {args.output}", file=sys.stderr) print(f"Total files found: {result['statistics']['total_files']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/php/test_pipeline.py b/libs/openant-core/parsers/php/test_pipeline.py index fd104777..7529ea96 100644 --- a/libs/openant-core/parsers/php/test_pipeline.py +++ b/libs/openant-core/parsers/php/test_pipeline.py @@ -42,6 +42,7 @@ from enum import Enum from pathlib import Path from typing import Set +from utilities.file_io import open_utf8, read_json, run_utf8, write_json # Add parent directory to path for utilities import sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) @@ -139,8 +140,7 @@ def run_parser_pipeline(self) -> bool: # Save scan results self.scan_results_file = os.path.join(self.output_dir, 'scan_results.json') - with open(self.scan_results_file, 'w') as f: - json.dump(scan_result, f, indent=2) + write_json(self.scan_results_file, scan_result) # Stage 2: Extract functions print(" [2/4] Extracting functions via tree-sitter...") @@ -178,13 +178,11 @@ def run_parser_pipeline(self) -> bool: print(f" Avg upstream deps: {dataset['statistics']['avg_upstream']}") # Write dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) # Write analyzer output analyzer_output = generator.generate_analyzer_output() - with open(self.analyzer_output_file, 'w') as f: - json.dump(analyzer_output, f, indent=2) + write_json(self.analyzer_output_file, analyzer_output) elapsed = (datetime.now() - start_time).total_seconds() @@ -242,8 +240,7 @@ def apply_reachability_filter(self) -> bool: start_time = datetime.now() try: - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) @@ -262,8 +259,7 @@ def apply_reachability_filter(self) -> bool: } # Build call graph from dataset unit metadata - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) call_graph = {} reverse_call_graph = {} @@ -313,8 +309,7 @@ def apply_reachability_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -379,7 +374,7 @@ def run_codeql_analysis(self) -> bool: '--overwrite' ] - result = subprocess.run( + result = run_utf8( create_db_cmd, capture_output=True, text=True, @@ -410,7 +405,7 @@ def run_codeql_analysis(self) -> bool: f'codeql/{language}-queries:codeql-suites/{language}-security-extended.qls' ] - result = subprocess.run( + result = run_utf8( analyze_cmd, capture_output=True, text=True, @@ -443,8 +438,7 @@ def run_codeql_analysis(self) -> bool: } return False - with open(sarif_output, 'r') as f: - sarif_data = json.load(f) + sarif_data = read_json(sarif_output) self.codeql_findings = [] @@ -555,8 +549,7 @@ def apply_codeql_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Build mapping of file -> [(start_line, end_line, func_id)] file_functions = {} @@ -605,8 +598,7 @@ def apply_codeql_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -662,8 +654,7 @@ def run_context_enhancer(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) enhancer = ContextEnhancer() @@ -695,8 +686,7 @@ def run_context_enhancer(self) -> bool: 'data_flows_extracted': enhancer.stats['data_flows_extracted'] } - with open(self.dataset_file, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(self.dataset_file, enhanced) elapsed = (datetime.now() - start_time).total_seconds() @@ -740,8 +730,7 @@ def apply_exploitable_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -767,8 +756,7 @@ def apply_exploitable_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -908,7 +896,7 @@ def run_full_pipeline(self): # Save results summary results_file = os.path.join(self.output_dir, 'pipeline_results.json') - with open(results_file, 'w') as f: + with open_utf8(results_file, 'w') as f: clean_results = { 'repository': self.results['repository'], 'test_time': self.results['test_time'], diff --git a/libs/openant-core/parsers/php/unit_generator.py b/libs/openant-core/parsers/php/unit_generator.py index 9b36684d..63f9fffa 100644 --- a/libs/openant-core/parsers/php/unit_generator.py +++ b/libs/openant-core/parsers/php/unit_generator.py @@ -28,6 +28,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 # File boundary marker for enhanced code (PHP uses // comments) @@ -344,9 +345,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - call_graph_data = json.load(f) - + call_graph_data = read_json(args.input_file) options = { 'max_depth': args.depth, } @@ -374,7 +373,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"\nOutput written to: {args.output}", file=sys.stderr) else: @@ -383,8 +382,7 @@ def main(): # Write analyzer output if requested if args.analyzer_output: analyzer = generator.generate_analyzer_output() - with open(args.analyzer_output, 'w') as f: - json.dump(analyzer, f, indent=2) + write_json(args.analyzer_output, analyzer) print(f"Analyzer output written to: {args.analyzer_output}", file=sys.stderr) except Exception as e: diff --git a/libs/openant-core/parsers/python/ast_parser.py b/libs/openant-core/parsers/python/ast_parser.py index e4cdc219..7f9b7c86 100644 --- a/libs/openant-core/parsers/python/ast_parser.py +++ b/libs/openant-core/parsers/python/ast_parser.py @@ -17,6 +17,7 @@ import sys from pathlib import Path from typing import Dict, List, Optional, Tuple +from utilities.file_io import read_json, write_json, open_utf8 class PythonRouteParser: @@ -35,7 +36,8 @@ def detect_framework(self) -> str: for f in files: try: - content = f.read_text() + with open_utf8(f, errors="replace") as _f: + content = _f.read() if "from django" in content or "django.urls" in content: return "django" if "from flask" in content or "Flask(" in content: @@ -76,7 +78,8 @@ def _read_file(self, file_path: Path) -> str: path_str = str(file_path) if path_str not in self.file_cache: try: - self.file_cache[path_str] = file_path.read_text() + with open_utf8(file_path, errors="replace") as _f: + self.file_cache[path_str] = _f.read() except Exception as e: print(f"Error reading {file_path}: {e}") self.file_cache[path_str] = "" @@ -461,8 +464,7 @@ def main(): result = parser.parse() if output_file: - with open(output_file, 'w') as f: - json.dump(result, f, indent=2) + write_json(output_file, result) print(f"Output written to {output_file}") else: print(json.dumps(result, indent=2)) diff --git a/libs/openant-core/parsers/python/call_graph_builder.py b/libs/openant-core/parsers/python/call_graph_builder.py index 3d92b251..a6741cc1 100644 --- a/libs/openant-core/parsers/python/call_graph_builder.py +++ b/libs/openant-core/parsers/python/call_graph_builder.py @@ -38,6 +38,7 @@ import textwrap from pathlib import Path from typing import Any, Dict, List, Optional, Set, Tuple +from utilities.file_io import read_json, write_json, open_utf8 class CallGraphBuilder: @@ -492,9 +493,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - extractor_output = json.load(f) - + extractor_output = read_json(args.input_file) print(f"Processing {len(extractor_output.get('functions', {}))} functions...", file=sys.stderr) builder = CallGraphBuilder(extractor_output, {'max_depth': args.depth}) @@ -513,7 +512,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Output written to: {args.output}", file=sys.stderr) else: diff --git a/libs/openant-core/parsers/python/dataset_enhancer.py b/libs/openant-core/parsers/python/dataset_enhancer.py index d41f8a81..73efe062 100644 --- a/libs/openant-core/parsers/python/dataset_enhancer.py +++ b/libs/openant-core/parsers/python/dataset_enhancer.py @@ -13,6 +13,7 @@ import sys from pathlib import Path from typing import Dict, List, Optional, Set, Tuple +from utilities.file_io import read_json, write_json, open_utf8 class PythonDependencyResolver: @@ -29,7 +30,8 @@ def _read_file(self, file_path: Path) -> str: path_str = str(file_path) if path_str not in self.file_cache: try: - self.file_cache[path_str] = file_path.read_text() + with open_utf8(file_path, errors="replace") as _f: + self.file_cache[path_str] = _f.read() except Exception as e: self.file_cache[path_str] = "" return self.file_cache[path_str] @@ -226,9 +228,7 @@ def resolve_recursive(current_file: Path, current_code: str, depth: int): def enhance_dataset(dataset_path: str, repo_path: str, output_path: str = None): """Enhance a dataset with resolved dependencies.""" - with open(dataset_path, 'r') as f: - dataset = json.load(f) - + dataset = read_json(dataset_path) resolver = PythonDependencyResolver(repo_path) enhanced_units = [] @@ -263,8 +263,7 @@ def enhance_dataset(dataset_path: str, repo_path: str, output_path: str = None): dataset['enhanced'] = True if output_path: - with open(output_path, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(output_path, dataset) print(f"Enhanced dataset written to {output_path}") else: print(json.dumps(dataset, indent=2)) diff --git a/libs/openant-core/parsers/python/function_extractor.py b/libs/openant-core/parsers/python/function_extractor.py index 574ba080..8714e9dc 100644 --- a/libs/openant-core/parsers/python/function_extractor.py +++ b/libs/openant-core/parsers/python/function_extractor.py @@ -64,6 +64,7 @@ from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional, Set, Tuple +from utilities.file_io import read_json, write_json, open_utf8 class FunctionExtractor: @@ -596,8 +597,7 @@ def main(): extractor = FunctionExtractor(args.repo_path) if args.scan_file: - with open(args.scan_file) as f: - scan_result = json.load(f) + scan_result = read_json(args.scan_file) result = extractor.extract_from_scan(scan_result) else: result = extractor.extract_all() @@ -605,7 +605,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Extraction complete. Results written to: {args.output}", file=sys.stderr) print(f"Total functions: {result['statistics']['total_functions']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/python/parse_repository.py b/libs/openant-core/parsers/python/parse_repository.py index 45af852b..18a61b74 100644 --- a/libs/openant-core/parsers/python/parse_repository.py +++ b/libs/openant-core/parsers/python/parse_repository.py @@ -52,6 +52,7 @@ from function_extractor import FunctionExtractor from call_graph_builder import CallGraphBuilder from unit_generator import UnitGenerator +from utilities.file_io import read_json, write_json, open_utf8 def generate_analyzer_output(extractor_result: dict) -> dict: @@ -138,8 +139,7 @@ def parse_repository(repo_path: str, options: dict = None) -> tuple: if output_dir: scan_file = Path(output_dir) / 'scan_result.json' - with open(scan_file, 'w') as f: - json.dump(scan_result, f, indent=2) + write_json(scan_file, scan_result) print(f" Saved: {scan_file}", file=sys.stderr) # Phase 2: Extract functions @@ -154,8 +154,7 @@ def parse_repository(repo_path: str, options: dict = None) -> tuple: if output_dir: extract_file = Path(output_dir) / 'functions.json' - with open(extract_file, 'w') as f: - json.dump(extractor_result, f, indent=2) + write_json(extract_file, extractor_result) print(f" Saved: {extract_file}", file=sys.stderr) # Phase 3: Build call graph @@ -171,8 +170,7 @@ def parse_repository(repo_path: str, options: dict = None) -> tuple: if output_dir: graph_file = Path(output_dir) / 'call_graph.json' - with open(graph_file, 'w') as f: - json.dump(call_graph_result, f, indent=2) + write_json(graph_file, call_graph_result) print(f" Saved: {graph_file}", file=sys.stderr) # Phase 4: Generate units @@ -199,8 +197,7 @@ def parse_repository(repo_path: str, options: dict = None) -> tuple: if output_dir: analyzer_file = Path(output_dir) / 'analyzer_output.json' - with open(analyzer_file, 'w') as f: - json.dump(analyzer_output, f, indent=2) + write_json(analyzer_file, analyzer_output) print(f" Saved: {analyzer_file}", file=sys.stderr) print(f"\n" + "=" * 60, file=sys.stderr) @@ -253,7 +250,7 @@ def main(): # Save dataset dataset_json = json.dumps(dataset, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(dataset_json) print(f"\nDataset written to: {args.output}", file=sys.stderr) else: @@ -261,8 +258,7 @@ def main(): # Save analyzer output if requested if args.analyzer_output: - with open(args.analyzer_output, 'w') as f: - json.dump(analyzer_output, f, indent=2) + write_json(args.analyzer_output, analyzer_output) print(f"Analyzer output written to: {args.analyzer_output}", file=sys.stderr) except Exception as e: diff --git a/libs/openant-core/parsers/python/repository_scanner.py b/libs/openant-core/parsers/python/repository_scanner.py index e2ab1f03..108eac58 100644 --- a/libs/openant-core/parsers/python/repository_scanner.py +++ b/libs/openant-core/parsers/python/repository_scanner.py @@ -30,6 +30,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 class RepositoryScanner: @@ -289,7 +290,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Scan complete. Results written to: {args.output}", file=sys.stderr) print(f"Total files found: {result['statistics']['total_files']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/python/unit_generator.py b/libs/openant-core/parsers/python/unit_generator.py index a7d26807..19af301a 100644 --- a/libs/openant-core/parsers/python/unit_generator.py +++ b/libs/openant-core/parsers/python/unit_generator.py @@ -53,6 +53,7 @@ from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 # File boundary marker for enhanced code @@ -400,9 +401,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - call_graph_data = json.load(f) - + call_graph_data = read_json(args.input_file) options = { 'max_depth': args.depth, } @@ -430,7 +429,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"\nOutput written to: {args.output}", file=sys.stderr) else: diff --git a/libs/openant-core/parsers/ruby/call_graph_builder.py b/libs/openant-core/parsers/ruby/call_graph_builder.py index 3c4b3ea4..7e5d533b 100644 --- a/libs/openant-core/parsers/ruby/call_graph_builder.py +++ b/libs/openant-core/parsers/ruby/call_graph_builder.py @@ -39,6 +39,7 @@ import tree_sitter_ruby as ts_ruby from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 RUBY_LANGUAGE = Language(ts_ruby.language()) @@ -441,9 +442,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - extractor_output = json.load(f) - + extractor_output = read_json(args.input_file) print(f"Processing {len(extractor_output.get('functions', {}))} functions...", file=sys.stderr) builder = CallGraphBuilder(extractor_output, {'max_depth': args.depth}) @@ -462,7 +461,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Output written to: {args.output}", file=sys.stderr) else: diff --git a/libs/openant-core/parsers/ruby/function_extractor.py b/libs/openant-core/parsers/ruby/function_extractor.py index f2f1dc30..798945bb 100644 --- a/libs/openant-core/parsers/ruby/function_extractor.py +++ b/libs/openant-core/parsers/ruby/function_extractor.py @@ -42,6 +42,7 @@ import tree_sitter_ruby as ts_ruby from tree_sitter import Language, Parser +from utilities.file_io import read_json, write_json, open_utf8 RUBY_LANGUAGE = Language(ts_ruby.language()) @@ -444,8 +445,7 @@ def main(): extractor = FunctionExtractor(args.repo_path) if args.scan_file: - with open(args.scan_file) as f: - scan_result = json.load(f) + scan_result = read_json(args.scan_file) result = extractor.extract_from_scan(scan_result) else: result = extractor.extract_all() @@ -453,7 +453,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Extraction complete. Results written to: {args.output}", file=sys.stderr) print(f"Total functions: {result['statistics']['total_functions']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/ruby/repository_scanner.py b/libs/openant-core/parsers/ruby/repository_scanner.py index 65b9a14a..d561e5bd 100644 --- a/libs/openant-core/parsers/ruby/repository_scanner.py +++ b/libs/openant-core/parsers/ruby/repository_scanner.py @@ -30,6 +30,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 class RepositoryScanner: @@ -240,7 +241,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"Scan complete. Results written to: {args.output}", file=sys.stderr) print(f"Total files found: {result['statistics']['total_files']}", file=sys.stderr) diff --git a/libs/openant-core/parsers/ruby/test_pipeline.py b/libs/openant-core/parsers/ruby/test_pipeline.py index cffe880f..947d495b 100644 --- a/libs/openant-core/parsers/ruby/test_pipeline.py +++ b/libs/openant-core/parsers/ruby/test_pipeline.py @@ -42,6 +42,7 @@ from enum import Enum from pathlib import Path from typing import Set +from utilities.file_io import open_utf8, read_json, run_utf8, write_json # Add parent directory to path for utilities import sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) @@ -139,8 +140,7 @@ def run_parser_pipeline(self) -> bool: # Save scan results self.scan_results_file = os.path.join(self.output_dir, 'scan_results.json') - with open(self.scan_results_file, 'w') as f: - json.dump(scan_result, f, indent=2) + write_json(self.scan_results_file, scan_result) # Stage 2: Extract functions print(" [2/4] Extracting functions via tree-sitter...") @@ -178,13 +178,11 @@ def run_parser_pipeline(self) -> bool: print(f" Avg upstream deps: {dataset['statistics']['avg_upstream']}") # Write dataset - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) # Write analyzer output analyzer_output = generator.generate_analyzer_output() - with open(self.analyzer_output_file, 'w') as f: - json.dump(analyzer_output, f, indent=2) + write_json(self.analyzer_output_file, analyzer_output) elapsed = (datetime.now() - start_time).total_seconds() @@ -242,8 +240,7 @@ def apply_reachability_filter(self) -> bool: start_time = datetime.now() try: - with open(self.analyzer_output_file, 'r') as f: - analyzer = json.load(f) + analyzer = read_json(self.analyzer_output_file) functions = analyzer.get("functions", {}) @@ -262,8 +259,7 @@ def apply_reachability_filter(self) -> bool: } # Build call graph from dataset unit metadata - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) call_graph = {} reverse_call_graph = {} @@ -313,8 +309,7 @@ def apply_reachability_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -379,7 +374,7 @@ def run_codeql_analysis(self) -> bool: '--overwrite' ] - result = subprocess.run( + result = run_utf8( create_db_cmd, capture_output=True, text=True, @@ -410,7 +405,7 @@ def run_codeql_analysis(self) -> bool: f'codeql/{language}-queries:codeql-suites/{language}-security-extended.qls' ] - result = subprocess.run( + result = run_utf8( analyze_cmd, capture_output=True, text=True, @@ -443,8 +438,7 @@ def run_codeql_analysis(self) -> bool: } return False - with open(sarif_output, 'r') as f: - sarif_data = json.load(f) + sarif_data = read_json(sarif_output) self.codeql_findings = [] @@ -555,8 +549,7 @@ def apply_codeql_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) # Build mapping of file -> [(start_line, end_line, func_id)] file_functions = {} @@ -605,8 +598,7 @@ def apply_codeql_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -662,8 +654,7 @@ def run_context_enhancer(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) enhancer = ContextEnhancer() @@ -695,8 +686,7 @@ def run_context_enhancer(self) -> bool: 'data_flows_extracted': enhancer.stats['data_flows_extracted'] } - with open(self.dataset_file, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(self.dataset_file, enhanced) elapsed = (datetime.now() - start_time).total_seconds() @@ -740,8 +730,7 @@ def apply_exploitable_filter(self) -> bool: start_time = datetime.now() try: - with open(self.dataset_file, 'r') as f: - dataset = json.load(f) + dataset = read_json(self.dataset_file) units = dataset.get("units", []) original_count = len(units) @@ -767,8 +756,7 @@ def apply_exploitable_filter(self) -> bool: "reduction_percentage": round((1 - len(filtered_units) / original_count) * 100, 1) if original_count > 0 else 0 } - with open(self.dataset_file, 'w') as f: - json.dump(dataset, f, indent=2) + write_json(self.dataset_file, dataset) elapsed = (datetime.now() - start_time).total_seconds() @@ -908,7 +896,7 @@ def run_full_pipeline(self): # Save results summary results_file = os.path.join(self.output_dir, 'pipeline_results.json') - with open(results_file, 'w') as f: + with open_utf8(results_file, 'w') as f: clean_results = { 'repository': self.results['repository'], 'test_time': self.results['test_time'], diff --git a/libs/openant-core/parsers/ruby/unit_generator.py b/libs/openant-core/parsers/ruby/unit_generator.py index 184a2219..424d215a 100644 --- a/libs/openant-core/parsers/ruby/unit_generator.py +++ b/libs/openant-core/parsers/ruby/unit_generator.py @@ -28,6 +28,7 @@ from datetime import datetime from pathlib import Path from typing import Dict, List, Optional, Set +from utilities.file_io import read_json, write_json, open_utf8 # File boundary marker for enhanced code (Ruby uses # comments) @@ -344,9 +345,7 @@ def main(): args = parser.parse_args() try: - with open(args.input_file) as f: - call_graph_data = json.load(f) - + call_graph_data = read_json(args.input_file) options = { 'max_depth': args.depth, } @@ -374,7 +373,7 @@ def main(): output = json.dumps(result, indent=2) if args.output: - with open(args.output, 'w') as f: + with open_utf8(args.output, 'w') as f: f.write(output) print(f"\nOutput written to: {args.output}", file=sys.stderr) else: @@ -383,8 +382,7 @@ def main(): # Write analyzer output if requested if args.analyzer_output: analyzer = generator.generate_analyzer_output() - with open(args.analyzer_output, 'w') as f: - json.dump(analyzer, f, indent=2) + write_json(args.analyzer_output, analyzer) print(f"Analyzer output written to: {args.analyzer_output}", file=sys.stderr) except Exception as e: diff --git a/libs/openant-core/parsers/zig/call_graph_builder.py b/libs/openant-core/parsers/zig/call_graph_builder.py index 52f661de..fbd6fd59 100644 --- a/libs/openant-core/parsers/zig/call_graph_builder.py +++ b/libs/openant-core/parsers/zig/call_graph_builder.py @@ -4,11 +4,12 @@ Builds bidirectional call graphs showing function dependencies. """ -import json import re from collections import defaultdict from typing import Dict, Any, List, Set +from utilities.file_io import write_json + import tree_sitter_zig as ts_zig from tree_sitter import Language, Parser, Node @@ -321,5 +322,4 @@ def _resolve_call( def save_results(self, output_path: str, results: Dict[str, Any]) -> None: """Save call graph to a JSON file.""" - with open(output_path, "w") as f: - json.dump(results, f, indent=2) + write_json(output_path, results) diff --git a/libs/openant-core/parsers/zig/function_extractor.py b/libs/openant-core/parsers/zig/function_extractor.py index f3348a0f..647f0cd5 100644 --- a/libs/openant-core/parsers/zig/function_extractor.py +++ b/libs/openant-core/parsers/zig/function_extractor.py @@ -4,11 +4,12 @@ Extracts functions, methods, and structs from Zig source files using tree-sitter. """ -import json from datetime import datetime from pathlib import Path from typing import Dict, Any, Optional, List +from utilities.file_io import write_json + import tree_sitter_zig as ts_zig from tree_sitter import Language, Parser, Node @@ -276,5 +277,4 @@ def _classify_function(self, name: str, file_path: str) -> str: def save_results(self, output_path: str, results: Dict[str, Any]) -> None: """Save extraction results to a JSON file.""" - with open(output_path, "w") as f: - json.dump(results, f, indent=2) + write_json(output_path, results) diff --git a/libs/openant-core/parsers/zig/repository_scanner.py b/libs/openant-core/parsers/zig/repository_scanner.py index ae095645..bb988194 100644 --- a/libs/openant-core/parsers/zig/repository_scanner.py +++ b/libs/openant-core/parsers/zig/repository_scanner.py @@ -5,11 +5,12 @@ """ import os -import json from datetime import datetime from pathlib import Path from typing import List, Dict, Any, Optional +from utilities.file_io import write_json + class RepositoryScanner: """Scans a repository for Zig source files.""" @@ -131,5 +132,4 @@ def _is_test_file(self, filepath: str) -> bool: def save_results(self, output_path: str, results: Dict[str, Any]) -> None: """Save scan results to a JSON file.""" - with open(output_path, "w") as f: - json.dump(results, f, indent=2) + write_json(output_path, results) diff --git a/libs/openant-core/parsers/zig/test_pipeline.py b/libs/openant-core/parsers/zig/test_pipeline.py index b4a98325..d9e06217 100644 --- a/libs/openant-core/parsers/zig/test_pipeline.py +++ b/libs/openant-core/parsers/zig/test_pipeline.py @@ -20,6 +20,7 @@ import json import sys from pathlib import Path +from utilities.file_io import write_json # Add parent directories to path for imports sys.path.insert(0, str(Path(__file__).parent.parent.parent)) @@ -96,10 +97,8 @@ def main(): "statistics": {"total_units": 0, "by_type": {}}, "metadata": {"generator": "zig_unit_generator.py"}, } - with open(output_dir / "dataset.json", "w") as f: - json.dump(empty_dataset, f, indent=2) - with open(output_dir / "analyzer_output.json", "w") as f: - json.dump({"repository": str(repo_path), "functions": {}}, f, indent=2) + write_json(output_dir / "dataset.json", empty_dataset) + write_json(output_dir / "analyzer_output.json", {"repository": str(repo_path), "functions": {}}) return 0 # Stage 2: Function Extractor diff --git a/libs/openant-core/parsers/zig/unit_generator.py b/libs/openant-core/parsers/zig/unit_generator.py index de1ce1ca..71a306e8 100644 --- a/libs/openant-core/parsers/zig/unit_generator.py +++ b/libs/openant-core/parsers/zig/unit_generator.py @@ -4,11 +4,12 @@ Creates self-contained analysis units with dependency context. """ -import json from datetime import datetime from pathlib import Path from typing import Dict, Any, List, Optional, Set +from utilities.file_io import write_json + class UnitGenerator: """Generates analysis units from call graph data.""" @@ -246,8 +247,6 @@ def save_results( output_path = Path(output_dir) output_path.mkdir(parents=True, exist_ok=True) - with open(output_path / "dataset.json", "w") as f: - json.dump(dataset, f, indent=2) + write_json(output_path / "dataset.json", dataset) - with open(output_path / "analyzer_output.json", "w") as f: - json.dump(analyzer_output, f, indent=2) + write_json(output_path / "analyzer_output.json", analyzer_output) diff --git a/libs/openant-core/pyproject.toml b/libs/openant-core/pyproject.toml index 266e7dba..bf0377a8 100644 --- a/libs/openant-core/pyproject.toml +++ b/libs/openant-core/pyproject.toml @@ -21,6 +21,7 @@ dependencies = [ [project.optional-dependencies] dev = [ "pytest>=8.0.0", + "ruff>=0.8.0", ] [project.scripts] @@ -30,6 +31,14 @@ openant = "openant.cli:main" requires = ["hatchling"] build-backend = "hatchling.build" +[tool.ruff] +target-version = "py311" + +[tool.ruff.lint] +# Only rules that catch actual bugs (will break at runtime) +# F821: undefined name, F811: redefined unused name, F823: local var referenced before assignment +select = ["F821", "F811", "F823"] + [tool.hatch.build.targets.wheel] packages = [ "openant", diff --git a/libs/openant-core/report/__main__.py b/libs/openant-core/report/__main__.py index fbe65151..1ed32ce4 100644 --- a/libs/openant-core/report/__main__.py +++ b/libs/openant-core/report/__main__.py @@ -9,17 +9,17 @@ """ import argparse -import json import sys from pathlib import Path from .generator import generate_summary_report, generate_disclosure, generate_all from .schema import validate_pipeline_output, ValidationError +from utilities.file_io import open_utf8, read_json def cmd_summary(args): """Generate summary report.""" - pipeline_data = json.loads(Path(args.input).read_text()) + pipeline_data = read_json(args.input) try: validate_pipeline_output(pipeline_data) @@ -32,14 +32,15 @@ def cmd_summary(args): output_path = Path(args.output) if args.output else Path("SUMMARY_REPORT.md") output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text(report) + with open_utf8(output_path, "w") as f: + f.write(report) print(f" -> {output_path}") print(f" Cost: ${usage['cost_usd']:.4f} ({usage['total_tokens']:,} tokens)") def cmd_disclosures(args): """Generate disclosure documents.""" - pipeline_data = json.loads(Path(args.input).read_text()) + pipeline_data = read_json(args.input) try: validate_pipeline_output(pipeline_data) @@ -62,7 +63,8 @@ def cmd_disclosures(args): safe_name = finding["short_name"].replace(" ", "_").upper() filename = f"DISCLOSURE_{i:02d}_{safe_name}.md" - (output_dir / filename).write_text(disclosure) + with open_utf8(output_dir / filename, "w") as f: + f.write(disclosure) print(f" -> {output_dir / filename}") count += 1 diff --git a/libs/openant-core/report/generator.py b/libs/openant-core/report/generator.py index c996250c..9f08b873 100644 --- a/libs/openant-core/report/generator.py +++ b/libs/openant-core/report/generator.py @@ -13,6 +13,7 @@ from dotenv import load_dotenv from .schema import validate_pipeline_output, ValidationError +from utilities.file_io import open_utf8, read_json load_dotenv() @@ -63,7 +64,8 @@ def _check_api_key(): def load_prompt(name: str) -> str: """Load a prompt template from the prompts directory.""" - return (PROMPTS_DIR / f"{name}.txt").read_text() + with open_utf8(PROMPTS_DIR / f"{name}.txt") as f: + return f.read() def merge_dynamic_results(pipeline_data: dict, pipeline_path: str) -> dict: @@ -76,7 +78,7 @@ def merge_dynamic_results(pipeline_data: dict, pipeline_path: str) -> dict: if not dynamic_path.exists(): return pipeline_data - dynamic_data = json.loads(dynamic_path.read_text()) + dynamic_data = read_json(dynamic_path) results_by_id = {} for result in dynamic_data.get("results", []): fid = result.get("finding_id") @@ -233,7 +235,7 @@ def generate_disclosure(vulnerability_data: dict, product_name: str) -> tuple[st def generate_all(pipeline_path: str, output_dir: str) -> None: """Generate all reports from a pipeline output file.""" - pipeline_data = json.loads(Path(pipeline_path).read_text()) + pipeline_data = read_json(pipeline_path) try: validate_pipeline_output(pipeline_data) @@ -247,7 +249,8 @@ def generate_all(pipeline_path: str, output_dir: str) -> None: # Generate summary report print("Generating summary report...") summary, _usage = generate_summary_report(pipeline_data) - (output_path / "SUMMARY_REPORT.md").write_text(summary) + with open_utf8(output_path / "SUMMARY_REPORT.md", "w") as f: + f.write(summary) print(f" -> {output_path / 'SUMMARY_REPORT.md'}") # Generate disclosure for each confirmed vulnerability @@ -265,7 +268,8 @@ def generate_all(pipeline_path: str, output_dir: str) -> None: safe_name = finding["short_name"].replace(" ", "_").upper() filename = f"DISCLOSURE_{i:02d}_{safe_name}.md" - (disclosures_dir / filename).write_text(disclosure) + with open_utf8(disclosures_dir / filename, "w") as f: + f.write(disclosure) print(f" -> {disclosures_dir / filename}") diff --git a/libs/openant-core/tests/test_file_io.py b/libs/openant-core/tests/test_file_io.py new file mode 100644 index 00000000..a47bc4fb --- /dev/null +++ b/libs/openant-core/tests/test_file_io.py @@ -0,0 +1,392 @@ +"""Tests for utilities.file_io UTF-8 helpers and a regression scan.""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +CORE_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(CORE_ROOT)) + +from utilities.file_io import open_utf8, read_json, run_utf8, write_json # noqa: E402 + + +NON_ASCII = "héllo 日本語 — café" + + +# --------------------------------------------------------------------------- +# Helper unit tests +# --------------------------------------------------------------------------- + +def test_open_utf8_round_trip(tmp_path: Path): + p = tmp_path / "x.txt" + with open_utf8(p, "w") as f: + f.write(NON_ASCII) + with open_utf8(p) as f: + assert f.read() == NON_ASCII + + +def test_open_utf8_passes_through_binary_mode(tmp_path: Path): + """Binary mode should not get encoding= injected.""" + p = tmp_path / "raw.bin" + payload = NON_ASCII.encode("utf-8") + with open_utf8(p, "wb") as f: + f.write(payload) + with open_utf8(p, "rb") as f: + assert f.read() == payload + + +def test_open_utf8_caller_encoding_wins(tmp_path: Path): + """If caller explicitly passes encoding=, helper must not override it.""" + p = tmp_path / "y.txt" + p.write_bytes("café".encode("latin-1")) + with open_utf8(p, encoding="latin-1") as f: + assert f.read() == "café" + + +def test_read_json_round_trip(tmp_path: Path): + p = tmp_path / "data.json" + obj = {"greeting": NON_ASCII, "list": ["a", NON_ASCII, "b"]} + write_json(p, obj) + assert read_json(p) == obj + + +def test_write_json_uses_utf8(tmp_path: Path): + """write_json must encode non-ASCII as UTF-8 bytes (not cp1252).""" + p = tmp_path / "data.json" + write_json(p, {"k": NON_ASCII}) + raw = p.read_bytes() + # The non-ASCII characters should appear as their UTF-8 encoding (or as + # JSON-escaped \uXXXX sequences — both are valid; the key is that the + # file does not contain a cp1252-encoded ?-replacement). + decoded = raw.decode("utf-8") + parsed = json.loads(decoded) + assert parsed["k"] == NON_ASCII + + +def test_write_json_default_indent(tmp_path: Path): + """write_json should pretty-print by default for human readability.""" + p = tmp_path / "data.json" + write_json(p, {"a": 1, "b": 2}) + text = p.read_text(encoding="utf-8") + # Indented output spans multiple lines. + assert "\n" in text + + +# --------------------------------------------------------------------------- +# run_utf8 subprocess test +# --------------------------------------------------------------------------- + +def test_run_utf8_captures_non_ascii_text(): + """run_utf8 with text=True must decode UTF-8 stdout without raising on cp1252.""" + code = ( + "import sys; " + "sys.stdout.buffer.write('" + + NON_ASCII + + "'.encode('utf-8'))" + ) + result = run_utf8( + [sys.executable, "-c", code], + capture_output=True, + text=True, + timeout=30, + ) + assert result.returncode == 0 + assert result.stdout == NON_ASCII + + +def test_run_utf8_universal_newlines_alias(tmp_path: Path): + """universal_newlines=True is an alias for text=True; must also get UTF-8.""" + code = ( + "import sys; " + "sys.stdout.buffer.write('" + + NON_ASCII + + "'.encode('utf-8'))" + ) + result = run_utf8( + [sys.executable, "-c", code], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + universal_newlines=True, + timeout=30, + ) + assert result.returncode == 0 + assert result.stdout == NON_ASCII + + +def test_run_utf8_invalid_bytes_replaced_not_raised(): + """errors='replace' default means invalid bytes don't raise.""" + code = ( + "import sys; " + "sys.stdout.buffer.write(b'good\\x9d_bad')" + ) + result = run_utf8( + [sys.executable, "-c", code], + capture_output=True, + text=True, + timeout=30, + ) + assert result.returncode == 0 + # Invalid byte 0x9d is replaced by U+FFFD rather than raising. + assert "good" in result.stdout + assert "bad" in result.stdout + + +def test_run_utf8_caller_can_override_errors_default_strict(): + """Without text=True, run_utf8 should not inject errors='replace'. + + Confirms that the encoding/errors injection only fires for text-mode + captures, leaving binary subprocess invocations untouched. + """ + result = run_utf8( + [sys.executable, "-c", "import sys; sys.stdout.buffer.write(b'\\x9d')"], + capture_output=True, + timeout=30, + ) + assert result.returncode == 0 + assert result.stdout == b"\x9d" + + +def test_run_utf8_does_not_override_explicit_encoding(): + """If caller passes encoding= explicitly, run_utf8 must not overwrite it.""" + result = run_utf8( + [ + sys.executable, + "-c", + "import sys; sys.stdout.buffer.write('café\\n'.encode('latin-1'))", + ], + capture_output=True, + text=True, + encoding="latin-1", + timeout=30, + ) + assert result.returncode == 0 + assert "café" in result.stdout + + +# --------------------------------------------------------------------------- +# Regression scan: no bare open() calls reappear in non-test code +# --------------------------------------------------------------------------- + +def _iter_python_sources(root: Path): + for p in root.rglob("*.py"): + rel = p.relative_to(root).as_posix() + if rel.startswith("tests/"): + continue + if rel == "utilities/file_io.py": + continue + # Skip vendored/build artifacts + if any(part in {".venv", "venv", "build", "dist", "__pycache__"} for part in p.parts): + continue + yield p + + +_OPEN_CALL_RE = re.compile(r"(? str: + """Replace string literals and comments with spaces so identifier matches inside + docstrings/comments don't trigger the regression check.""" + out = [] + i = 0 + n = len(text) + in_str = None + triple = False + while i < n: + c = text[i] + if in_str: + if c == "\\" and not triple: + out.append(" ") + i += 2 + continue + if triple and text[i:i + 3] == in_str: + out.append(" ") + in_str = None + triple = False + i += 3 + continue + if not triple and c == in_str: + in_str = None + out.append(" ") + i += 1 + continue + if not triple and c == "\n": + in_str = None + out.append("\n") + i += 1 + continue + out.append("\n" if c == "\n" else " ") + i += 1 + continue + if c == "#": + nl = text.find("\n", i) + if nl == -1: + out.append(" " * (n - i)) + break + out.append(" " * (nl - i)) + i = nl + continue + if text[i:i + 3] in ('"""', "'''"): + in_str = text[i:i + 3] + triple = True + out.append(" ") + i += 3 + continue + if c in ("'", '"'): + in_str = c + out.append(" ") + i += 1 + continue + out.append(c) + i += 1 + return "".join(out) + + +def _has_encoding(call_args: str) -> bool: + return re.search(r"\bencoding\s*=", call_args) is not None + + +def _has_binary_mode(call_args: str) -> bool: + return re.search(r"""(['"])([rwax+]*b[rwax+]*)\1""", call_args) is not None + + +def _scan_calls(scrubbed: str, original: str, call_re: re.Pattern): + """Yield (line_number, args_text, original_line) for each call match.""" + for m in call_re.finditer(scrubbed): + i = m.end() + depth = 1 + while i < len(scrubbed) and depth: + ch = scrubbed[i] + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + i += 1 + if depth != 0: + continue + args = original[m.end():i - 1] + line = original[:m.start()].count("\n") + 1 + yield line, args, original.splitlines()[line - 1].strip() + + +def test_no_bare_open_in_non_test_code(): + """Regression: every text-mode `open(` call in non-test code must specify + encoding=, otherwise Windows defaults to cp1252 and crashes on non-ASCII + source code. + """ + offenders: list[str] = [] + for path in _iter_python_sources(CORE_ROOT): + text = path.read_text(encoding="utf-8") + scrubbed = _strip_strings_and_comments(text) + for line, args, src in _scan_calls(scrubbed, text, _OPEN_CALL_RE): + if _has_binary_mode(args) or _has_encoding(args): + continue + rel = path.relative_to(CORE_ROOT).as_posix() + offenders.append(f"{rel}:{line}: {src}") + + assert not offenders, ( + "Found bare open() calls without encoding= in non-test code. " + "Use utilities.file_io.open_utf8 / read_json / write_json or pass " + "encoding='utf-8' explicitly:\n " + "\n ".join(offenders) + ) + + +# Match `.read_text(` / `.write_text(` method calls (any object, including +# Path objects). Don't match `text=` kwargs or other identifiers ending in +# read_text/write_text. +_PATH_TEXT_RE = re.compile(r"\.(?:read_text|write_text)\s*\(") + + +# Match `path.open(`-style method calls. The bare ``open(`` case is handled +# above, so here we look explicitly for ``.open(`` (Path or file-like object +# method form) which has the same Windows cp1252 default behaviour as +# ``open()`` and is not caught by the bare-open regex. +_DOT_OPEN_RE = re.compile(r"\.open\s*\(") + + +def test_no_bare_pathlib_text_io_in_non_test_code(): + """Regression: ``Path.read_text()`` / ``write_text()`` default to the + system locale encoding on Python <3.10 and to ``locale.getpreferredencoding(False)`` + in 3.10+ unless ``-X utf8`` mode is on. On Windows that is cp1252, which + crashes on non-ASCII content. Every call in non-test code must pass + ``encoding=`` explicitly. + """ + offenders: list[str] = [] + for path in _iter_python_sources(CORE_ROOT): + text = path.read_text(encoding="utf-8") + scrubbed = _strip_strings_and_comments(text) + for line, args, src in _scan_calls(scrubbed, text, _PATH_TEXT_RE): + if _has_encoding(args): + continue + rel = path.relative_to(CORE_ROOT).as_posix() + offenders.append(f"{rel}:{line}: {src}") + + assert not offenders, ( + "Found Path.read_text()/write_text() calls without encoding= in " + "non-test code. Pass encoding='utf-8' explicitly:\n " + + "\n ".join(offenders) + ) + + +def test_no_bare_dot_open_in_non_test_code(): + """Regression: ``path.open()`` (the Path / file-like method form) defaults + to system locale encoding the same way ``open()`` does, and is not caught + by the bare-open regex above. Every text-mode call must pass ``encoding=``. + """ + offenders: list[str] = [] + for path in _iter_python_sources(CORE_ROOT): + text = path.read_text(encoding="utf-8") + scrubbed = _strip_strings_and_comments(text) + for line, args, src in _scan_calls(scrubbed, text, _DOT_OPEN_RE): + if _has_binary_mode(args) or _has_encoding(args): + continue + rel = path.relative_to(CORE_ROOT).as_posix() + offenders.append(f"{rel}:{line}: {src}") + + assert not offenders, ( + "Found .open() calls without encoding= in non-test code. " + "Pass encoding='utf-8' explicitly:\n " + "\n ".join(offenders) + ) + + +# Match `subprocess.run(` (covers `subprocess.run` and `sp.run` etc. via the +# right-hand identifier — restrict to the explicit form to avoid noise). +_SUBPROCESS_RUN_RE = re.compile(r"(? bool: + return ( + re.search(r"\btext\s*=\s*True", call_args) is not None + or re.search(r"\buniversal_newlines\s*=\s*True", call_args) is not None + ) + + +def test_no_bare_text_mode_subprocess_in_non_test_code(): + """Regression: ``subprocess.run(..., text=True)`` decodes stdout/stderr + with the system locale on Windows (cp1252), which crashes on non-ASCII + output from parsers, codeql, etc. Every text-mode subprocess call must + pass ``encoding=`` explicitly (or use ``utilities.file_io.run_utf8``). + """ + offenders: list[str] = [] + for path in _iter_python_sources(CORE_ROOT): + text = path.read_text(encoding="utf-8") + scrubbed = _strip_strings_and_comments(text) + for line, args, src in _scan_calls(scrubbed, text, _SUBPROCESS_RUN_RE): + if not _has_text_mode(args): + continue + if _has_encoding(args): + continue + rel = path.relative_to(CORE_ROOT).as_posix() + offenders.append(f"{rel}:{line}: {src}") + + assert not offenders, ( + "Found subprocess.run(..., text=True) calls without encoding= in " + "non-test code. Pass encoding='utf-8', errors='replace' explicitly " + "(or use utilities.file_io.run_utf8):\n " + "\n ".join(offenders) + ) diff --git a/libs/openant-core/tests/test_go_cli.py b/libs/openant-core/tests/test_go_cli.py index fc92113b..42ad294e 100644 --- a/libs/openant-core/tests/test_go_cli.py +++ b/libs/openant-core/tests/test_go_cli.py @@ -129,17 +129,13 @@ def test_parse_js_repo(self, sample_js_repo, tmp_path): "--language", "javascript", "--json", ) - if result.returncode != 0: - if "No module named" in result.stderr: - if sys.platform == "win32": - pytest.skip("Go CLI using system Python without required packages (Windows)") - else: - pytest.fail("Go CLI resolved wrong Python (missing required packages)") - if "UnicodeEncodeError" in result.stderr: - if sys.platform == "win32": - pytest.skip("Pre-existing Unicode bug in JS test_pipeline.py on Windows") - else: - pytest.fail("UnicodeEncodeError from JS parser on non-Windows (unexpected regression)") + if result.returncode != 0 and "No module named" in result.stderr: + if sys.platform == "win32": + pytest.skip("Go CLI using system Python without required packages (Windows)") + else: + pytest.fail("Go CLI resolved wrong Python (missing required packages)") + if result.returncode != 0 and "UnicodeEncodeError" in result.stderr: + pytest.fail("UnicodeEncodeError from JS parser (unexpected regression)") assert result.returncode == 0 envelope = json.loads(result.stdout) assert envelope["status"] == "success" diff --git a/libs/openant-core/tests/test_js_parser.py b/libs/openant-core/tests/test_js_parser.py index 25bf9515..44e5d67f 100644 --- a/libs/openant-core/tests/test_js_parser.py +++ b/libs/openant-core/tests/test_js_parser.py @@ -6,7 +6,6 @@ import json import subprocess import shutil -import sys from pathlib import Path import pytest @@ -66,13 +65,6 @@ def test_skip_tests_flag(self, tmp_path): class TestTypeScriptAnalyzer: - # Known issue: ts-morph fails to resolve files with backslash paths on Windows - _windows_path_xfail = pytest.mark.xfail( - sys.platform == "win32", - reason="ts-morph path resolution issue with Windows backslash paths", - strict=False, - ) - def test_analyzes_files(self, sample_js_repo, tmp_path): # First scan to get file list scan_output = tmp_path / "scan.json" @@ -94,7 +86,6 @@ def test_analyzes_files(self, sample_js_repo, tmp_path): assert result.returncode == 0 assert analyzer_output.exists() - @_windows_path_xfail def test_extracts_functions(self, sample_js_repo, tmp_path): scan_output = tmp_path / "scan.json" run_node("repository_scanner.js", sample_js_repo, "--output", str(scan_output)) @@ -190,14 +181,6 @@ def test_units_have_required_fields(self, analyzer_output, tmp_path): class TestFullPipeline: """End-to-end test through parser_adapter.""" - # Known issue: test_pipeline.py uses Unicode checkmarks that fail on Windows cp1252 - _windows_unicode_xfail = pytest.mark.xfail( - sys.platform == "win32", - reason="JS test_pipeline.py Unicode chars fail on Windows cp1252 encoding", - strict=False, - ) - - @_windows_unicode_xfail def test_parse_js_repo(self, sample_js_repo, tmp_output_dir): from core.parser_adapter import parse_repository @@ -213,7 +196,6 @@ def test_parse_js_repo(self, sample_js_repo, tmp_output_dir): assert result.analyzer_output_path is not None assert Path(result.analyzer_output_path).exists() - @_windows_unicode_xfail def test_auto_detects_javascript(self, sample_js_repo, tmp_output_dir): from core.parser_adapter import parse_repository diff --git a/libs/openant-core/tests/test_parser_adapter.py b/libs/openant-core/tests/test_parser_adapter.py index af209cb7..0acc7f83 100644 --- a/libs/openant-core/tests/test_parser_adapter.py +++ b/libs/openant-core/tests/test_parser_adapter.py @@ -1,11 +1,11 @@ """Tests for core/parser_adapter.py — language detection and Python parsing.""" -import json import os from pathlib import Path import pytest from core.parser_adapter import detect_language, parse_repository +from utilities.file_io import read_json class TestDetectLanguage: @@ -65,8 +65,7 @@ def test_dataset_json_valid(self, sample_python_repo, tmp_output_dir): language="python", processing_level="all", ) - with open(result.dataset_path) as f: - dataset = json.load(f) + dataset = read_json(result.dataset_path) assert "units" in dataset assert len(dataset["units"]) > 0 @@ -77,8 +76,7 @@ def test_units_have_required_fields(self, sample_python_repo, tmp_output_dir): language="python", processing_level="all", ) - with open(result.dataset_path) as f: - dataset = json.load(f) + dataset = read_json(result.dataset_path) for unit in dataset["units"]: assert "id" in unit assert "code" in unit @@ -101,6 +99,5 @@ def test_analyzer_output_generated(self, sample_python_repo, tmp_output_dir): ) assert result.analyzer_output_path is not None assert Path(result.analyzer_output_path).exists() - with open(result.analyzer_output_path) as f: - data = json.load(f) + data = read_json(result.analyzer_output_path) assert "functions" in data diff --git a/libs/openant-core/tests/test_silent_401.py b/libs/openant-core/tests/test_silent_401.py index bbb9fe7c..d21041d2 100644 --- a/libs/openant-core/tests/test_silent_401.py +++ b/libs/openant-core/tests/test_silent_401.py @@ -104,14 +104,31 @@ def test_analyze_sync_raises_on_auth_error(): from utilities.llm_client import AnthropicClient - AuthError = sys.modules["anthropic"].AuthenticationError - - client = AnthropicClient.__new__(AnthropicClient) - client.client = MagicMock() - client.client.messages.create.side_effect = AuthError("invalid x-api-key") - client.model = "claude-haiku-4-5-20251001" - client.tracker = MagicMock() - client.last_call = None - - with pytest.raises(AuthError): - client.analyze_sync("test prompt") + # Remove the mock from sys.modules to get the real anthropic SDK + mock_anthropic = sys.modules.pop("anthropic", None) + try: + import importlib + importlib.invalidate_caches() + from anthropic import AuthenticationError + import httpx + + # Create a mock response object for the APIStatusError + mock_response = MagicMock(spec=httpx.Response) + mock_response.status_code = 401 + mock_response.headers = {"request-id": "test-123"} + + client = AnthropicClient.__new__(AnthropicClient) + client.client = MagicMock() + # Create the error with the correct signature + error = AuthenticationError(message="invalid x-api-key", response=mock_response, body={"error": "invalid_api_key"}) + client.client.messages.create.side_effect = error + client.model = "claude-haiku-4-5-20251001" + client.tracker = MagicMock() + client.last_call = None + + with pytest.raises(AuthenticationError): + client.analyze_sync("test prompt") + finally: + # Restore the mock for other tests + if mock_anthropic is not None: + sys.modules["anthropic"] = mock_anthropic diff --git a/libs/openant-core/tests/test_windows_path_handling.py b/libs/openant-core/tests/test_windows_path_handling.py new file mode 100644 index 00000000..dc492a12 --- /dev/null +++ b/libs/openant-core/tests/test_windows_path_handling.py @@ -0,0 +1,395 @@ +"""Tests for Windows-specific path/encoding handling in JS and Go parser pipelines. + +These tests cover three fixes that prevent OpenAnt from running correctly on +Windows. + +Coverage by platform: +- ``test_to_posix_path_normalises_backslashes`` — cross-platform; verifies the + normalisation helper directly via ``node -e`` (skipped only if Node is absent). +- ``test_typescript_analyzer_strips_crlf_from_file_list`` — cross-platform; + CRLF stripping is equally testable on POSIX. +- ``test_typescript_analyzer_accepts_backslash_paths`` — Windows-only; backslash + absolute paths are meaningless on POSIX so the end-to-end scenario can only + run there. +- ``test_pipeline_uses_ascii_fallback_on_cp1252_stdout`` and the Unicode + counterpart — cross-platform; cp1252 stdout encoding can be simulated anywhere. +- ``test_no_bare_path_calls_in_typescript_analyzer`` — cross-platform static + scanner; greps typescript_analyzer.js for path.X() calls missing toPosixPath(), + mirroring the PR #45 antipattern-prevention pattern. +""" +import importlib.util +import io +import json +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +PARSERS_DIR = Path(__file__).parent.parent / "parsers" +JS_PARSERS_DIR = PARSERS_DIR / "javascript" +GO_PARSERS_DIR = PARSERS_DIR / "go" +TS_ANALYZER = JS_PARSERS_DIR / "typescript_analyzer.js" +PATH_UTILS = JS_PARSERS_DIR / "path_utils.js" +JS_NODE_MODULES = JS_PARSERS_DIR / "node_modules" + + +# --------------------------------------------------------------------------- +# JS analyzer: backslash paths must be normalised before reaching ts-morph +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif( + not shutil.which("node"), + reason="Node.js not available", +) +def test_to_posix_path_normalises_backslashes(): + """toPosixPath() must replace every backslash with a forward slash. + + Calls the *actual* function from path_utils.js (not a reimplementation), + so a regression in the live regex is caught on all platforms. Also covers + the UNC path contract documented in path_utils.js: ``\\\\server\\share\\...`` + must become ``//server/share/...``. + """ + # Require path_utils.js directly — it has no ts-morph dependency, so + # node_modules is not required. We use \x5c (hex for backslash) in the + # JS string literals to avoid Python/Node string-escaping ambiguity. + path_utils = str(PATH_UTILS).replace("\\", "/") + script = ( + "const {toPosixPath} = require(" + json.dumps(path_utils) + ");" + + r"console.log(JSON.stringify([toPosixPath('C:\x5cUsers\x5cfoo\x5cbar.js'),toPosixPath('\x5c\x5cserver\x5cshare\x5cfoo.js')]));" + ) + result = subprocess.run( + ["node", "-e", script], + capture_output=True, + text=True, + timeout=10, + ) + assert result.returncode == 0, f"node failed: {result.stderr}" + results = json.loads(result.stdout.strip()) + assert results[0] == "C:/Users/foo/bar.js", ( + f"regular path: {results[0]!r}" + ) + assert results[1] == "//server/share/foo.js", ( + f"UNC path: {results[1]!r}" + ) + + +@pytest.mark.skipif( + sys.platform != "win32" + or not shutil.which("node") + or not JS_NODE_MODULES.exists(), + reason="Windows-only (backslash paths are non-absolute on POSIX) and " + "requires Node.js and JS parser npm dependencies", +) +def test_typescript_analyzer_accepts_backslash_paths(tmp_path): + """Regression: ts-morph silently drops files when given backslash paths. + + On Windows, ``path.relative()`` and ``path.resolve()`` produce paths + separated by ``\\``. ts-morph treats backslash as an escape character + when matching paths it has already added, so without explicit + normalisation the analyzer reports zero functions even for valid input. + This test only runs on Windows because a Linux/macOS absolute path + (``/tmp/...``) with all slashes replaced becomes ``\\tmp\\...`` which + Node does not treat as absolute, making the test unsound on POSIX. + """ + # Create a simple repo + repo = tmp_path / "repo" + src = repo / "src" + src.mkdir(parents=True) + (src / "module.js").write_text( + "function greet(name) { return `hello ${name}`; }\n" + "module.exports = { greet };\n", + encoding="utf-8", + ) + + # Write a file list using backslash separators (the Windows-native form). + # On POSIX this is otherwise meaningless input, but the analyzer's + # normalisation step should still accept it. + file_list = tmp_path / "files.txt" + abs_path = str(src / "module.js") + backslash_path = abs_path.replace("/", "\\") + file_list.write_text(backslash_path + "\n", encoding="utf-8") + + out_file = tmp_path / "analyzer_output.json" + result = subprocess.run( + [ + "node", + str(TS_ANALYZER), + str(repo), + "--files-from", + str(file_list), + "--output", + str(out_file), + ], + capture_output=True, + text=True, + timeout=30, + ) + + assert result.returncode == 0, ( + f"analyzer failed:\nSTDERR:\n{result.stderr}\nSTDOUT:\n{result.stdout}" + ) + data = json.loads(out_file.read_text(encoding="utf-8")) + + # Functions must be found, regardless of slash flavour in the input. + assert data.get("functions"), ( + f"expected at least one function; got {data.get('functions')!r}" + ) + func_names = [f.get("name") for f in data["functions"].values()] + assert "greet" in func_names + + # Function ids must be POSIX-form (forward slashes only). Backslash + # leakage into ids would break downstream Python consumers. + for func_id in data["functions"]: + assert "\\" not in func_id, f"functionId contains backslash: {func_id!r}" + + +@pytest.mark.skipif( + not shutil.which("node") or not JS_NODE_MODULES.exists(), + reason="Node.js or JS parser npm dependencies not available", +) +def test_typescript_analyzer_strips_crlf_from_file_list(tmp_path): + """Regression: file lists written on Windows have CRLF line endings. + + Splitting on ``\\n`` alone leaves a trailing ``\\r`` on each path, + which ts-morph then fails to resolve. Confirm the analyzer accepts + a CRLF-terminated file list and produces a non-empty result. + """ + repo = tmp_path / "repo" + repo.mkdir() + (repo / "a.js").write_text("function alpha() {}\n", encoding="utf-8") + (repo / "b.js").write_text("function beta() {}\n", encoding="utf-8") + + file_list = tmp_path / "files.txt" + # Explicit CRLF, plus a trailing blank line that should be tolerated. + content = "\r\n".join([str(repo / "a.js"), str(repo / "b.js"), ""]) + file_list.write_bytes(content.encode("utf-8")) + + out_file = tmp_path / "out.json" + result = subprocess.run( + [ + "node", + str(TS_ANALYZER), + str(repo), + "--files-from", + str(file_list), + "--output", + str(out_file), + ], + capture_output=True, + text=True, + timeout=30, + ) + + assert result.returncode == 0, ( + f"analyzer failed:\nSTDERR:\n{result.stderr}\nSTDOUT:\n{result.stdout}" + ) + data = json.loads(out_file.read_text(encoding="utf-8")) + func_names = [f.get("name") for f in data.get("functions", {}).values()] + assert "alpha" in func_names + assert "beta" in func_names + + +# --------------------------------------------------------------------------- +# Static regression scanner: toPosixPath() must wrap every path.X() call +# --------------------------------------------------------------------------- + +# Match path.relative/resolve/join call sites (non-method-name contexts). +# dirname/basename/normalize/isAbsolute are intentionally excluded: they +# preserve (not change) the separator style of their input, so they are safe +# as long as the *input* is already normalised — which is the precondition +# enforced by toPosixPath() at the point where Windows paths enter the system. +# If you add a path method that *produces* new separators (e.g. path.format), +# add it to this regex. +_BARE_PATH_CALL_RE = re.compile(r"\bpath\.(relative|resolve|join)\s*\(") +# Match JS comment lines so JSDoc / inline comments don't trip the scanner. +_JS_COMMENT_LINE_RE = re.compile(r"^\s*(?://|\*)") + + +def test_no_bare_path_calls_in_typescript_analyzer(): + """Regression: every path.relative/resolve/join() in typescript_analyzer.js + must be accompanied by toPosixPath() within 6 lines. + + Mirrors the pattern from PR #45 (test_no_bare_open, test_no_bare_pathlib_text_io, + etc.) that prevents contributors from reintroducing encoding antipatterns. + Here the guarded antipattern is passing a raw path.X() result to ts-morph + or storing it as a functionId component without normalising backslashes first. + + The ±6-line window covers all current wrapping patterns: + - same-line: ``toPosixPath(path.resolve(...))`` + - split-line: ``toPosixPath(\\n path.relative(...))`` (toPosixPath 1 line before) + - assign-then-wrap: ``const v = path.join(...);\\n...\\ntoPosixPath(v)`` (up to 5 lines after, + including interleaved comment lines) + + Scoped to typescript_analyzer.js only — other JS parser files (context_assembler, + repository_scanner, etc.) legitimately call path.X() without toPosixPath because + they don't interact with ts-morph. + """ + text = TS_ANALYZER.read_text(encoding="utf-8") + lines = text.splitlines() + offenders = [] + for i, line in enumerate(lines): + if _JS_COMMENT_LINE_RE.match(line): + continue + if _BARE_PATH_CALL_RE.search(line): + lo = max(0, i - 6) + hi = min(len(lines), i + 7) + window = "\n".join(lines[lo:hi]) + if "toPosixPath(" not in window: + offenders.append(f":{i + 1}: {line.strip()}") + assert not offenders, ( + f"Found path.relative/resolve/join() without toPosixPath() in " + f"{TS_ANALYZER.name}. Wrap the result with toPosixPath() — see " + f"the CONTRACT comment on that function.\n " + "\n ".join(offenders) + ) + + +# --------------------------------------------------------------------------- +# test_pipeline.py: status output must stay safe on a cp1252 stdout +# --------------------------------------------------------------------------- + + +def _load_pipeline_module(name, source_path): + """Import a parser test_pipeline.py module under a custom name. + + The two pipelines (JS, Go) live in sibling directories and both + expose a module named ``test_pipeline``. We import them under + distinct names so they coexist in this test process. + + The module is registered in ``sys.modules`` so that callers can + reliably remove it afterwards (via ``sys.modules.pop(name, None)``) + to prevent stale module-level state — such as ``_UNICODE_OK`` and + ``SYM_*`` globals — from leaking into subsequent tests. + + ``sys.path`` is snapshot/restored around ``exec_module`` so that the + module-level ``sys.path.insert`` calls in the pipeline files do not + accumulate extra entries on repeated calls (e.g. across parametrized + test runs). + """ + spec = importlib.util.spec_from_file_location(name, source_path) + mod = importlib.util.module_from_spec(spec) + # Register before exec so that relative imports inside the module + # resolve correctly, and so callers can pop the module after use. + sys.modules[name] = mod + # Snapshot sys.path so that module-level sys.path.insert calls inside + # the pipeline files do not leave behind permanent extra entries. + path_snapshot = sys.path[:] + try: + spec.loader.exec_module(mod) + except BaseException: + # Roll back the registration so the failed module does not pollute + # sys.modules (CPython convention for importlib loaders). + sys.modules.pop(name, None) + raise + finally: + # Restore sys.path to prevent the module's own insertions from + # accumulating across repeated calls. + sys.path[:] = path_snapshot + return mod + + +@pytest.fixture(params=["javascript", "go"]) +def pipeline_module(request, monkeypatch): + """Load the JS or Go test_pipeline module fresh under a synthetic stdout. + + We point ``sys.stdout`` at a buffer with a cp1252 encoding before the + module is imported, so the module-level ``_stdout_supports_unicode()`` + check sees the constrained encoding. We then re-import each time to + capture the fresh module-level state. + """ + # Replace stdout with a cp1252-only buffer so the module-level helper + # picks the ASCII fallback. + fake_stdout = io.TextIOWrapper(io.BytesIO(), encoding="cp1252", newline="") + monkeypatch.setattr(sys, "stdout", fake_stdout) + + parsers_root = PARSERS_DIR + parsers_parent = str(parsers_root.parent) + already_on_path = parsers_parent in sys.path + if not already_on_path: + sys.path.insert(0, parsers_parent) # so utilities.* imports work + try: + if request.param == "javascript": + path = parsers_root / "javascript" / "test_pipeline.py" + mod_name = "openant_test_js_pipeline_cp1252" + else: + path = parsers_root / "go" / "test_pipeline.py" + mod_name = "openant_test_go_pipeline_cp1252" + + # Drop any cached version so module-level symbol detection re-runs. + sys.modules.pop(mod_name, None) + mod = _load_pipeline_module(mod_name, path) + yield mod + finally: + # Remove the freshly loaded module so its stale cp1252-patched + # module-level state (_UNICODE_OK, SYM_*) doesn't leak into later + # tests that may load the same pipeline module. + sys.modules.pop(mod_name, None) + if not already_on_path: + try: + sys.path.remove(parsers_parent) + except ValueError: + pass + + +def test_pipeline_uses_ascii_fallback_on_cp1252_stdout(pipeline_module): + """Status symbols must be ASCII-only on a cp1252-encoded stdout. + + The original pipelines printed ``✓``, ``✗`` and ``→`` directly, which + crashed Python's print on cp1252 consoles (the Windows default). + """ + assert pipeline_module._UNICODE_OK is False, ( + "_stdout_supports_unicode() should report False for cp1252 stdout" + ) + assert pipeline_module.SYM_OK == "OK" + assert pipeline_module.SYM_FAIL == "FAIL" + assert pipeline_module.SYM_ARROW == "->" + + # And the ASCII fallbacks must round-trip through cp1252 without error. + for s in (pipeline_module.SYM_OK, pipeline_module.SYM_FAIL, pipeline_module.SYM_ARROW): + s.encode("cp1252") # must not raise + + +@pytest.mark.parametrize( + "mod_name,rel_path", + [ + ("openant_test_js_pipeline_utf8", "javascript/test_pipeline.py"), + ("openant_test_go_pipeline_utf8", "go/test_pipeline.py"), + ], +) +def test_pipeline_uses_unicode_when_stdout_supports_it(monkeypatch, mod_name, rel_path): + """When stdout can encode the symbols, prefer the prettier Unicode form. + + Covers both the JS and Go pipeline modules to ensure neither regresses + to ASCII when the terminal supports Unicode. + """ + # Reload under a UTF-8 stdout to confirm the other branch. + fake_stdout = io.TextIOWrapper(io.BytesIO(), encoding="utf-8", newline="") + monkeypatch.setattr(sys, "stdout", fake_stdout) + + parsers_root = PARSERS_DIR + parsers_parent = str(parsers_root.parent) + already_on_path = parsers_parent in sys.path + if not already_on_path: + sys.path.insert(0, parsers_parent) + try: + sys.modules.pop(mod_name, None) + mod = _load_pipeline_module(mod_name, parsers_root / rel_path) + assert mod._UNICODE_OK is True + # Use Unicode escape sequences rather than literal glyphs so that + # assertion failure messages are safe on cp1252 consoles — printing + # the literal characters (U+2713, U+2717, U+2192) would itself raise + # UnicodeEncodeError on the very platform these tests guard against. + assert mod.SYM_OK == "\u2713" # CHECK MARK (U+2713) + assert mod.SYM_FAIL == "\u2717" # BALLOT X (U+2717) + assert mod.SYM_ARROW == "\u2192" # RIGHTWARDS ARROW (U+2192) + finally: + sys.modules.pop(mod_name, None) + if not already_on_path: + try: + sys.path.remove(parsers_parent) + except ValueError: + pass diff --git a/libs/openant-core/utilities/agentic_enhancer/repository_index.py b/libs/openant-core/utilities/agentic_enhancer/repository_index.py index 06ef199b..5af649c8 100644 --- a/libs/openant-core/utilities/agentic_enhancer/repository_index.py +++ b/libs/openant-core/utilities/agentic_enhancer/repository_index.py @@ -14,11 +14,12 @@ load_index_from_file: Load index from analyzer_output.json file """ -import json import re from pathlib import Path from typing import Optional +from utilities.file_io import read_json + class RepositoryIndex: """ @@ -283,7 +284,6 @@ def load_index_from_file(analyzer_output_path: str, repo_path: str = None) -> Re Returns: RepositoryIndex instance """ - with open(analyzer_output_path, 'r') as f: - analyzer_output = json.load(f) + analyzer_output = read_json(analyzer_output_path) return RepositoryIndex(analyzer_output, repo_path) diff --git a/libs/openant-core/utilities/context_enhancer.py b/libs/openant-core/utilities/context_enhancer.py index 2ffbfe6a..2f7dea20 100644 --- a/libs/openant-core/utilities/context_enhancer.py +++ b/libs/openant-core/utilities/context_enhancer.py @@ -28,6 +28,7 @@ from .llm_client import AnthropicClient, TokenTracker, get_global_tracker, reset_global_tracker from .agentic_enhancer import RepositoryIndex, enhance_unit_with_agent, load_index_from_file from .rate_limiter import get_rate_limiter, is_rate_limit_error, is_retryable_error +from .file_io import read_json, write_json # Avoid circular import — import checkpoint at usage site _StepCheckpoint = None @@ -504,8 +505,7 @@ def enhance_dataset_agentic( if unit_id in processed_ids: cp_file = os.path.join(checkpoint_dir, f"{self._safe_filename(unit_id)}.json") if os.path.exists(cp_file): - with open(cp_file, 'r') as f: - cp_data = json.load(f) + cp_data = read_json(cp_file) unit["agent_context"] = cp_data.get("agent_context", {}) if "code" in cp_data: unit["code"] = cp_data["code"] @@ -538,8 +538,7 @@ def enhance_dataset_agentic( if not os.path.exists(cp_file): continue try: - with open(cp_file, 'r') as f: - cp_data = json.load(f) + cp_data = read_json(cp_file) # Sum usage from all existing checkpoints (completed + errored) cp_usage = cp_data.get("usage", {}) _summary_input_tokens += cp_usage.get("input_tokens", 0) @@ -792,8 +791,7 @@ def _save_unit_checkpoint(self, unit: dict, checkpoint_dir: str): "output_tokens": meta.get("output_tokens", 0), "cost_usd": meta.get("cost_usd", 0.0), } - with open(filepath, 'w') as f: - json.dump(cp_data, f, indent=2) + write_json(filepath, cp_data) def _load_completed_units(self, checkpoint_dir: str) -> set: """Load the set of completed unit IDs from per-unit checkpoint files.""" @@ -805,8 +803,7 @@ def _load_completed_units(self, checkpoint_dir: str) -> set: continue filepath = os.path.join(checkpoint_dir, filename) try: - with open(filepath, 'r') as f: - cp_data = json.load(f) + cp_data = read_json(filepath) unit_id = cp_data.get("id") agent_ctx = cp_data.get("agent_context", {}) if unit_id and agent_ctx and not agent_ctx.get("error"): @@ -818,8 +815,7 @@ def _load_completed_units(self, checkpoint_dir: str) -> set: def _migrate_legacy_checkpoint(self, checkpoint_path: str, checkpoint_dir: str, units: list): """Migrate a legacy single-file checkpoint to per-unit checkpoint files.""" try: - with open(checkpoint_path, 'r') as f: - checkpoint_data = json.load(f) + checkpoint_data = read_json(checkpoint_path) for cp_unit in checkpoint_data.get("units", []): if cp_unit.get("agent_context") and not cp_unit["agent_context"].get("error"): self._save_unit_checkpoint(cp_unit, checkpoint_dir) @@ -998,8 +994,7 @@ def main(): logging.error(f"Error: Input file not found: {input_path}") return 1 - with open(input_path, 'r') as f: - dataset = json.load(f) + dataset = read_json(input_path) # Enhance enhancer = ContextEnhancer() @@ -1029,8 +1024,7 @@ def main(): # Write output output_path = Path(args.output) if args.output else input_path - with open(output_path, 'w') as f: - json.dump(enhanced, f, indent=2) + write_json(output_path, enhanced) logging.info(f"Enhanced dataset written to: {output_path}") return 0 diff --git a/libs/openant-core/utilities/dynamic_tester/__init__.py b/libs/openant-core/utilities/dynamic_tester/__init__.py index e533f6ce..03922ad1 100644 --- a/libs/openant-core/utilities/dynamic_tester/__init__.py +++ b/libs/openant-core/utilities/dynamic_tester/__init__.py @@ -20,6 +20,7 @@ from utilities.dynamic_tester.result_collector import collect_result from utilities.dynamic_tester.reporter import generate_report from utilities.llm_client import get_global_tracker +from utilities.file_io import read_json, write_json, open_utf8 def run_dynamic_tests( @@ -45,9 +46,7 @@ def run_dynamic_tests( List of DynamicTestResult objects """ # Load pipeline output - with open(pipeline_output_path, "r") as f: - pipeline = json.load(f) - + pipeline = read_json(pipeline_output_path) findings = pipeline.get("findings", []) repo_info = { "name": pipeline.get("repository", {}).get("name", "unknown"), @@ -253,13 +252,13 @@ def run_dynamic_tests( report_md = generate_report(results, repo_info["name"], total_cost) report_path = os.path.join(output_dir, "DYNAMIC_TEST_RESULTS.md") - with open(report_path, "w") as f: + with open_utf8(report_path, "w") as f: f.write(report_md) print(f"\nReport written to {report_path}", file=sys.stderr) # Save structured results JSON results_path = os.path.join(output_dir, "dynamic_test_results.json") - with open(results_path, "w") as f: + with open_utf8(results_path, "w") as f: json.dump({ "repository": repo_info["name"], "total_findings": len(findings), diff --git a/libs/openant-core/utilities/dynamic_tester/docker_executor.py b/libs/openant-core/utilities/dynamic_tester/docker_executor.py index 04a45d38..87dec730 100644 --- a/libs/openant-core/utilities/dynamic_tester/docker_executor.py +++ b/libs/openant-core/utilities/dynamic_tester/docker_executor.py @@ -12,6 +12,7 @@ import tempfile import time import uuid +from utilities.file_io import open_utf8, run_utf8 # Timeouts DEFAULT_CONTAINER_TIMEOUT = 120 # seconds per container @@ -74,14 +75,14 @@ def _write_test_files(work_dir: str, generation: dict, source_file: str | None = shutil.copy2(source_file, os.path.join(work_dir, os.path.basename(source_file))) # Write Dockerfile - with open(os.path.join(work_dir, "Dockerfile"), "w") as f: + with open_utf8(os.path.join(work_dir, "Dockerfile"), "w") as f: f.write(generation["dockerfile"]) # Write test script test_filename = generation.get("test_filename", "test_exploit.py") test_path = os.path.join(work_dir, test_filename) os.makedirs(os.path.dirname(test_path), exist_ok=True) - with open(test_path, "w") as f: + with open_utf8(test_path, "w") as f: f.write(generation["test_script"]) # Write requirements/dependencies file @@ -89,7 +90,7 @@ def _write_test_files(work_dir: str, generation: dict, source_file: str | None = req_filename = generation.get("requirements_filename", "requirements.txt") req_path = os.path.join(work_dir, req_filename) os.makedirs(os.path.dirname(req_path), exist_ok=True) - with open(req_path, "w") as f: + with open_utf8(req_path, "w") as f: f.write(generation["requirements"]) # Copy attacker server if needed (before docker-compose so it's available) @@ -98,21 +99,21 @@ def _write_test_files(work_dir: str, generation: dict, source_file: str | None = os.makedirs(attacker_dir, exist_ok=True) shutil.copy2(ATTACKER_SERVER_PATH, os.path.join(attacker_dir, "server.py")) # Write attacker Dockerfile - with open(os.path.join(attacker_dir, "Dockerfile"), "w") as f: + with open_utf8(os.path.join(attacker_dir, "Dockerfile"), "w") as f: f.write("FROM python:3.11-slim\nWORKDIR /app\nCOPY server.py .\n" "EXPOSE 9999\nCMD [\"python\", \"server.py\"]\n") # Write docker-compose if multi-service, with sanitization if generation.get("docker_compose"): compose_content = _sanitize_compose(generation["docker_compose"]) - with open(os.path.join(work_dir, "docker-compose.yml"), "w") as f: + with open_utf8(os.path.join(work_dir, "docker-compose.yml"), "w") as f: f.write(compose_content) def _run_command(cmd: list[str], timeout: int, cwd: str = None) -> tuple[str, str, int, bool]: """Run a command with timeout. Returns (stdout, stderr, exit_code, timed_out).""" try: - result = subprocess.run( + result = run_utf8( cmd, capture_output=True, text=True, diff --git a/libs/openant-core/utilities/file_io.py b/libs/openant-core/utilities/file_io.py new file mode 100644 index 00000000..bc8d22f7 --- /dev/null +++ b/libs/openant-core/utilities/file_io.py @@ -0,0 +1,60 @@ +"""Centralized file I/O and subprocess helpers for Windows UTF-8 compatibility. + +On Windows, Python's default encoding is often ``cp1252`` (charmap), which +cannot decode common UTF-8 sequences found in source code. These thin +wrappers ensure that every file open and subprocess call uses UTF-8 +explicitly, preventing ``'charmap' codec can't decode byte ...`` errors. +""" + +import json +import os +import subprocess +from typing import Any, Union + +# Accept str, Path, or any os.PathLike +PathLike = Union[str, os.PathLike] + + +def open_utf8(path: PathLike, mode: str = "r", **kwargs): + """Open a file with UTF-8 encoding by default. + + Drop-in replacement for ``open()`` that sets ``encoding='utf-8'`` unless + the caller explicitly provides a different encoding or opens in binary + mode. + """ + if "b" not in mode and "encoding" not in kwargs: + kwargs["encoding"] = "utf-8" + return open(path, mode, **kwargs) + + +def read_json(path: PathLike) -> Any: + """Read and parse a JSON file using UTF-8 encoding.""" + with open_utf8(path, "r") as f: + return json.load(f) + + +def write_json(path: PathLike, data: Any, **kwargs) -> None: + """Write data as JSON to a file using UTF-8 encoding.""" + kwargs.setdefault("indent", 2) + with open_utf8(path, "w") as f: + json.dump(data, f, **kwargs) + + +def run_utf8(*args, **kwargs) -> subprocess.CompletedProcess: + """Run a subprocess with UTF-8 encoding for text mode. + + Wrapper around ``subprocess.run`` that sets ``encoding='utf-8'`` and + ``errors='replace'`` when ``text=True`` (or its alias + ``universal_newlines=True``) is passed, preventing charmap decode errors + on Windows. + + Note: ``errors='replace'`` substitutes U+FFFD for invalid bytes in + stdout/stderr rather than raising. This is intentional - subprocess + output is used for status display and diagnostics, not for security + analysis (parser results are read from JSON files separately). + Callers can override with ``errors='strict'`` if needed. + """ + if kwargs.get("text") or kwargs.get("universal_newlines"): + kwargs.setdefault("encoding", "utf-8") + kwargs.setdefault("errors", "replace") + return subprocess.run(*args, **kwargs) diff --git a/libs/openant-core/validate_dataset_schema.py b/libs/openant-core/validate_dataset_schema.py index 1312bceb..7f65a7c0 100755 --- a/libs/openant-core/validate_dataset_schema.py +++ b/libs/openant-core/validate_dataset_schema.py @@ -8,6 +8,7 @@ import json import sys +from utilities.file_io import read_json def validate_unit(unit, index): @@ -61,9 +62,7 @@ def validate_unit(unit, index): def validate_dataset(path): - with open(path) as f: - data = json.load(f) - + data = read_json(path) all_errors = [] units = data.get("units", [])