Skip to content

Replace TAP filtering with split-output NDJSON pipeline #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
friedenberg wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Replace TAP filtering with split-output NDJSON pipeline #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 46 additions & 19 deletions nix/packages/batman.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,9 @@ let
sandbox=true
allow_local_binding=false
no_tempdir_cleanup=false
hide_passing=false
split=true
pass_out=""
config=""

bats_args=()
while (( $# > 0 )); do
Expand All @@ -180,8 +182,20 @@ let
no_tempdir_cleanup=true
shift
;;
--hide-passing)
hide_passing=true
--no-split|--full-output)
split=false
shift
;;
--pass-out)
if (( $# < 2 )); then
echo "bats wrapper: --pass-out requires a path argument" >&2
exit 2
fi
pass_out="$2"
shift 2
;;
--pass-out=*)
pass_out="''${1#--pass-out=}"
shift
;;
--)
Expand All @@ -197,6 +211,27 @@ let
done
set -- "''${bats_args[@]}"

# Per-run tmpdir for the passing-test NDJSON when split is on and
# no caller-supplied path was given. Reported on stderr so an
# interactive user can find it. Cleaned with the wrapper's EXIT
# trap unless --no-tempdir-cleanup is set.
pass_tmp=""
if $split && [[ -z "$pass_out" ]]; then
pass_tmp="$(mktemp -d --suffix=.batman)"
pass_out="$pass_tmp/passes.ndjson"
echo "bats wrapper: passing-test NDJSON -> $pass_out" >&2
fi

cleanup() {
if [[ -n "$config" ]]; then
rm -f "$config"
fi
if [[ -n "$pass_tmp" ]] && ! $no_tempdir_cleanup; then
rm -rf "$pass_tmp"
fi
}
trap cleanup EXIT

# Append batman's bats-libs to BATS_LIB_PATH (caller paths take precedence)
export BATS_LIB_PATH="''${BATS_LIB_PATH:+$BATS_LIB_PATH:}${bats-libs}/share/bats"

Expand All @@ -213,32 +248,24 @@ let
use_tap14=true
fi

filter_tap() {
if $hide_passing; then
awk '
/^ ---$/ { in_yaml = 1; if (show) print; next }
/^ \.\.\.$/ { in_yaml = 0; if (show) print; next }
in_yaml { if (show) print; next }
/^ok / { show = ($0 ~ /# [Ss][Kk][Ii][Pp]/ || $0 ~ /# [Tt][Oo][Dd][Oo]/); if (show) print; next }
/^not ok / { show = 1; print; next }
{ show = 1; print }
'
reformat_tap() {
if $use_tap14; then
tap-dancer reformat
else
cat
fi
}

reformat_tap() {
if $use_tap14; then
tap-dancer reformat
split_or_passthrough() {
if $split; then
tap-dancer format-ndjson --split --pass-out "$pass_out"
else
cat
fi
}

if $sandbox; then
config="$(mktemp --suffix=.json)"
trap 'rm -f "$config"' EXIT

# fence config: denyRead blocks credential dirs; allowWrite
# restricts writes to /tmp; empty allowedDomains denies all
Expand Down Expand Up @@ -290,12 +317,12 @@ let
set -- --no-tempdir-cleanup "$@"
fi

fence --settings "$config" -- bats "$@" | filter_tap | reformat_tap
fence --settings "$config" -- bats "$@" | reformat_tap | split_or_passthrough
else
if $no_tempdir_cleanup; then
set -- --no-tempdir-cleanup "$@"
fi
bats "$@" | filter_tap | reformat_tap
bats "$@" | reformat_tap | split_or_passthrough
fi
'';
};
Expand Down
79 changes: 65 additions & 14 deletions nix/packages/bats-lane.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,13 @@ let

# Opt-in: capture the bats TAP-14 stream and convert it to NDJSON
# alongside the run. When true, `$out` becomes a DIRECTORY containing
# `run.tap`, `run.ndjson`, and `exit_code` (the bats exit status as a
# one-line decimal). When false (default), `$out` remains a single
# stamp file as before. Existing consumers that `stat result` as a
# file are unaffected unless they opt in.
# `run.raw.tap`, `run.tap`, `exit_code`, and one of:
# - `run.failures.ndjson` + `run.passes.ndjson` when `splitNdjson`
# is true (the default), or
# - `run.ndjson` (combined records) when `splitNdjson` is false.
# When `emitNdjson` itself is false (the original default), `$out`
# remains a single stamp file as before. Existing consumers that
# `stat result` as a file are unaffected unless they opt in.
#
# Requires `tap-dancer-go` either as a top-level arg of this file
# (the default plumbed by flake.nix) or as a per-call override
Expand All @@ -158,6 +161,15 @@ let
# (`docs/rfcs/0001-test-result-ndjson-schema.md`).
emitNdjson ? false,

# When `emitNdjson` is true, split NDJSON records into
# `$out/run.failures.ndjson` (failures + bail-outs) and
# `$out/run.passes.ndjson` (passing records) via
# `tap-dancer format-ndjson --split`. Defaults to true so build
# logs (and the inline stderr echo) aren't drowned in pass
# records. Set false to keep the combined `run.ndjson` and full
# inline echo of every record.
splitNdjson ? true,

# Per-call override of the `tap-dancer-go` derivation used when
# `emitNdjson = true`. Falls back to the top-level `tap-dancer-go`
# arg of this file. Ignored when `emitNdjson = false`.
Expand Down Expand Up @@ -265,18 +277,60 @@ let
'';

# NDJSON form: $out is a directory carrying `run.raw.tap`,
# `run.tap`, `run.ndjson`, and `exit_code`. The pipeline is:
# `run.tap`, NDJSON records (split or combined), and `exit_code`.
# The pipeline is:
#
# bats --formatter tap13 ... → run.raw.tap (TAP-13 + YAML)
# tap-dancer reformat → run.tap (TAP-14 header prepended)
# tap-dancer format-ndjson → run.ndjson (per-test NDJSON records)
# tap-dancer format-ndjson → split or combined NDJSON
#
# When `splitNdjson` is true (the default), `format-ndjson --split`
# writes failure records to `run.failures.ndjson` (stdout) and
# passing records to `run.passes.ndjson` (--pass-out). The inline
# stderr echo then carries only the failure records, keeping the
# build log focused on what went wrong.
#
# When `splitNdjson` is false, the original combined behavior is
# preserved: every record lands in `run.ndjson` and is echoed
# inline.
#
# We disable errexit around the bats call so a failed test run
# still gets converted to NDJSON before the derivation exits
# with the bats status. On bats failure the directory is only
# preserved via `nix build --keep-failed`. The bats exit status
# is the gating signal; reformat/format-ndjson exit codes are
# ignored so we don't double-fail or mask the bats outcome.
ndjsonRenderStep =
if splitNdjson then
''
${tapDancerGo}/bin/tap-dancer format-ndjson --split \
--pass-out "$out/run.passes.ndjson" \
< "$out/run.tap" > "$out/run.failures.ndjson" || true
# Ensure both files exist even on all-pass / all-fail runs,
# so downstream consumers can unconditionally read them.
[ -e "$out/run.failures.ndjson" ] || : > "$out/run.failures.ndjson"
[ -e "$out/run.passes.ndjson" ] || : > "$out/run.passes.ndjson"
''
else
''
${tapDancerGo}/bin/tap-dancer format-ndjson \
< "$out/run.tap" > "$out/run.ndjson" || true
'';
ndjsonEchoStep =
if splitNdjson then
''
printf '%s\n' '>>> BATSLANE NDJSON BEGIN <<<' >&2
cat "$out/run.failures.ndjson" >&2
pass_count=$(wc -l < "$out/run.passes.ndjson" | tr -d ' ')
printf 'passes: %s record(s) at %s\n' "$pass_count" "$out/run.passes.ndjson" >&2
printf '%s\n' '>>> BATSLANE NDJSON END <<<' >&2
''
else
''
printf '%s\n' '>>> BATSLANE NDJSON BEGIN <<<' >&2
cat "$out/run.ndjson" >&2
printf '%s\n' '>>> BATSLANE NDJSON END <<<' >&2
'';
ndjsonInvocation = ''
mkdir -p "$out"
cd stage/zz-tests_bats
Expand All @@ -292,18 +346,15 @@ let
set -o errexit
${tapDancerGo}/bin/tap-dancer reformat \
< "$out/run.raw.tap" > "$out/run.tap" || cp "$out/run.raw.tap" "$out/run.tap"
${tapDancerGo}/bin/tap-dancer format-ndjson \
< "$out/run.tap" > "$out/run.ndjson" || true
${ndjsonRenderStep}
echo "$bats_status" > "$out/exit_code"
# Echo the NDJSON to stderr between sentinel markers so the
# nix builder log carries the captured records inline. On
# failure, `nix build` prints the build-log tail to the user
# Echo NDJSON to stderr between sentinel markers so the nix
# builder log carries the captured records inline. On failure,
# `nix build` prints the build-log tail to the user
# automatically; for any run, `nix log <drv>` retrieves the
# full log including this block. Agents extracting the NDJSON
# programmatically can sed/awk between the BEGIN/END markers.
printf '%s\n' '>>> BATSLANE NDJSON BEGIN <<<' >&2
cat "$out/run.ndjson" >&2
printf '%s\n' '>>> BATSLANE NDJSON END <<<' >&2
${ndjsonEchoStep}
exit "$bats_status"
'';

Expand Down
34 changes: 30 additions & 4 deletions packages/batman/doc/bats-lane.7.scd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,15 +192,30 @@ testFiles = [ "current_version/*.bats" "previous_versions/main.bats" ];
The pipeline when emitNdjson is on:

```
bats --formatter tap13 ... → run.raw.tap (TAP-13 + YAML diagnostics)
tap-dancer reformat → run.tap (TAP-14 with version header)
tap-dancer format-ndjson → run.ndjson (per-test NDJSON records)
bats --formatter tap13 ... → run.raw.tap (TAP-13 + YAML)
tap-dancer reformat → run.tap (TAP-14 + header)
tap-dancer format-ndjson [--split] → split or combined NDJSON
```

With *splitNdjson = true* (the default), the final stage runs
*tap-dancer format-ndjson --split --pass-out run.passes.ndjson*
so failure records land in *run.failures.ndjson* and passing
records land in *run.passes.ndjson*. The inline build-log echo
then carries only the failure records.

With *splitNdjson = false*, the original combined behavior is
preserved: every record lands in *run.ndjson*.

The TAP-13 formatter is selected automatically when no
formatter is supplied via *extraBatsArgs*; callers passing
their own *--formatter* / *--tap* are honored as-is.

*splitNdjson*
A bool (default true). Ignored unless *emitNdjson = true*. Set
false to opt out of the split-output default and produce a
single combined *run.ndjson* file plus a full inline build-log
echo.

*tapDancerGo*
A derivation override (default: the top-level *tap-dancer-go*
passed when this file is imported). Ignored unless
Expand All @@ -213,7 +228,18 @@ When *emitNdjson = false* (the default), *$out* is a single empty
regular file — a stamp signaling that bats succeeded. Consumers can
treat *result* as a file and *stat* / hash it normally.

When *emitNdjson = true*, *$out* is a *directory* containing:
When *emitNdjson = true*, *$out* is a *directory*. Under the default
*splitNdjson = true*:

```
run.raw.tap # bats's raw TAP-13 output (pre-reformat)
run.tap # TAP-14 with version header (tap-dancer reformat)
run.failures.ndjson # NDJSON failure records (stdout of --split)
run.passes.ndjson # NDJSON passing records (--pass-out target)
exit_code # bats's exit status as a one-line decimal
```

Under *splitNdjson = false*:

```
run.raw.tap # bats's raw TAP-13 output (pre-reformat)
Expand Down
19 changes: 16 additions & 3 deletions packages/batman/doc/bats-testing.7.scd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ bats-testing - BATS integration test conventions for amarbel-llc projects

Integration tests in amarbel-llc projects use BATS (Bash Automated Testing
System) via the *batman* wrapper. Batman bundles BATS with six support
libraries, fence-based sandbox isolation, and TAP-14 output formatting.
libraries, fence-based sandbox isolation, and a split-NDJSON output
pipeline (failures on stdout, passes to a per-run tmpdir file) backed
by *tap-dancer format-ndjson --split*. Pass *--no-split* / *--full-output*
to restore the historical TAP-14 stream.

This manpage covers the conventions for writing, organizing, and running
BATS tests.
Expand Down Expand Up @@ -288,8 +291,18 @@ arguments:
*--no-tempdir-cleanup*
Keep BATS temp directories after test run.

*--hide-passing*
Filter TAP output to show only failures and skips.
*--no-split*, *--full-output*
Bypass the default split-output pipeline and emit the bats
TAP-14 stream verbatim on stdout (plan, version, and every
*ok*/*not ok* line). Equivalent to the pre-split behavior.

*--pass-out* _PATH_
Override the path for the passing-test NDJSON file produced
by the default split pipeline. Implies the split mode is
active; the file is written when at least one test passes
and is otherwise an empty file. The wrapper still emits a
stderr diagnostic naming the chosen path. Ignored under
*--no-split*.

*--diagnostics-stderr*
Write batman's wrapper diagnostics (parse errors, missing
Expand Down
Loading