Skip to content

Commit fcce612

Browse files
mwlog1cMike Wilson
andauthored
chore: sanity-check sweep — registry stub, CI badges, benchmarks (#5)
Six items, one PR: 1. Python source: ruff, mypy --strict, pytest 15/15 — clean. No formatting changes needed. 2. Registry packages: replace `npm install`/`pip install` blocks with from-source instructions. PyPI and npm are 404 for vella-sdk; the one-liners were inviting users into a dead end. Note v1.0.1 will land registry packages. 3. Clean-clone validation: documented Python venv pin to 3.10+ in the install instructions to surface the requires-python floor (system python3 is 3.9 on many macOS installs and silently picks the wrong interpreter). 4. CI badges: add Test SDK / Verify Test Vectors / Lint Schemas badges under the DOI badge. 5. Benchmarks: new benchmarks/ harness for both SDKs. - 11 independent runs, warmup phase, median-of-medians headline - GC quiesced (Node --expose-gc, Python gc.disable in inner loop) - Ephemeral ECDSA P-256 key for with-proof scenario - 100K samples decision-only / 10K with-proof per run - Reference results checked in for Apple M2 Pro (Node 22 / Py 3.12) - Linux x86_64 row left open for community contribution 6. Move premature commercial-component paragraph: drop the line-26 "separate commercial components" sentence so it doesn't precede Install/Quickstart. The dedicated "Where this SDK fits" section later in the README remains the single place commercial scope is discussed. Co-authored-by: Mike Wilson <mwlog1c1-netizen@users.noreply.github.com>
1 parent 4f45cd6 commit fcce612

6 files changed

Lines changed: 643 additions & 14 deletions

File tree

README.md

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19738377.svg)](https://doi.org/10.5281/zenodo.19738377)
2+
[![Test SDK](https://github.com/vellacognitive/vella-substrate/actions/workflows/test-sdk.yml/badge.svg)](https://github.com/vellacognitive/vella-substrate/actions/workflows/test-sdk.yml)
3+
[![Verify Test Vectors](https://github.com/vellacognitive/vella-substrate/actions/workflows/verify-test-vectors.yml/badge.svg)](https://github.com/vellacognitive/vella-substrate/actions/workflows/verify-test-vectors.yml)
4+
[![Lint Schemas](https://github.com/vellacognitive/vella-substrate/actions/workflows/lint-schemas.yml/badge.svg)](https://github.com/vellacognitive/vella-substrate/actions/workflows/lint-schemas.yml)
25

36
# VELLA — Governance Substrate for AI Agents & Autonomous Systems
47

@@ -23,8 +26,6 @@ VELLA decides whether an action is authorized before it runs. The calling system
2326

2427
The SDK is designed for in-process, low-latency adjudication; the authority decision can sit directly on the action path rather than being deferred to post-hoc logging.
2528

26-
This public SDK is a focused reference implementation of the VELLA decision/proof primitive. Production deployment patterns that require a runtime service, sidecar adapter, Helm packaging, or management plane are supported through separate commercial components.
27-
2829
**Start here:**
2930
- [An Inspectable Substrate for AI Governance](https://vellacognitive.com/research/an-inspectable-substrate-for-ai-governance) — the conceptual argument (~14 min read)
3031
- [Quickstart](#quick-example) — Node or Python SDK, working example in 2 minutes
@@ -33,29 +34,31 @@ This public SDK is a focused reference implementation of the VELLA decision/proo
3334

3435
## Install
3536

36-
```bash
37-
npm install @vellacognitive/vella-sdk
38-
```
39-
40-
```bash
41-
pip install vella-sdk
42-
```
43-
44-
## Local development
37+
> **Registry packages are not yet published.** `npm install @vellacognitive/vella-sdk` and `pip install vella-sdk` will be live with v1.0.1. Until then, install from source against this repository — the SDK has no external runtime dependencies on Node and one (`cryptography`) on Python.
4538
4639
### Node SDK from source
4740

4841
```bash
49-
cd sdk/node
42+
git clone https://github.com/vellacognitive/vella-substrate.git
43+
cd vella-substrate/sdk/node
5044
npm ci
5145
npm test
5246
```
5347

48+
Then import directly from the checkout, or `npm pack` to produce a tarball:
49+
50+
```bash
51+
npm pack # produces vellacognitive-vella-sdk-1.0.0.tgz
52+
# in your project:
53+
npm install /path/to/vellacognitive-vella-sdk-1.0.0.tgz
54+
```
55+
5456
### Python SDK from source
5557

5658
```bash
57-
cd sdk/python
58-
python3 -m venv .venv
59+
git clone https://github.com/vellacognitive/vella-substrate.git
60+
cd vella-substrate/sdk/python
61+
python3.10 -m venv .venv # 3.10+ required; replace with python3.11/3.12 as available
5962
. .venv/bin/activate
6063
python -m pip install --upgrade pip
6164
pip install -e ".[dev]"
@@ -107,6 +110,8 @@ print(result["proof_bundle"]["envelope_id"])
107110
- `verify/verify.js`, `verify/verify.py`, `verify/verify.sh` standalone proof-bundle verifiers
108111
- Test vectors:
109112
- `test-vectors/valid/` and `test-vectors/tampered/` for verifier CI and audit workflows
113+
- Benchmarks:
114+
- `benchmarks/` reproducible latency harness for both SDKs — see [`benchmarks/README.md`](benchmarks/README.md) for methodology and reference results
110115

111116
## Where this SDK fits
112117

benchmarks/README.md

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# VELLA SDK — Latency Benchmarks
2+
3+
Reproducible benchmarks for the in-process SDK. Backs the latency claims in
4+
[`DEPLOYMENT.md`](../DEPLOYMENT.md).
5+
6+
The harness measures two scenarios:
7+
8+
- **decision-only**`govern({intent, evidenceMask})`. Hot path, no proof bundle.
9+
- **with-proof**`govern({intent, evidenceMask, proof: {signingKey}})`. Includes
10+
envelope construction, SHA-256 hashing, and ECDSA P-256 signing.
11+
12+
## Methodology
13+
14+
- **Warmup.** Each scenario runs a warmup phase (10K iterations decision-only,
15+
2K with-proof) before any measurement, so JIT and import caches are stable.
16+
- **Independent runs.** Each scenario then runs 11 independent measurement
17+
passes. Per-pass percentiles are computed from the full sample distribution;
18+
the headline metric is the **median of the per-run medians** (resilient to a
19+
single noisy run).
20+
- **GC quiesced.** Node runs with `--expose-gc` and calls `global.gc()`
21+
between runs. Python calls `gc.collect()` then `gc.disable()` for the timed
22+
inner loop, re-enabled afterward.
23+
- **Ephemeral signing key.** A fresh ECDSA P-256 key is generated at startup
24+
for the with-proof scenario — no fixture coupling.
25+
- **Timer.** `process.hrtime.bigint()` (Node) and `time.perf_counter_ns()`
26+
(Python). Monotonic, nanosecond resolution. Timer overhead (~50 ns/sample)
27+
applies equally to all scenarios.
28+
- **Sample sizes.** Decision-only: 100K iterations × 11 runs = 1.1M samples
29+
per runtime. With-proof: 10K × 11 = 110K samples per runtime.
30+
31+
The harness emits both the median-of-medians and the full quantile breakdown
32+
of run #1 (so you can see what tail latency on a representative pass looks
33+
like, not just a smoothed central tendency).
34+
35+
## Running
36+
37+
```bash
38+
# full run, both runtimes, human-readable
39+
./benchmarks/run.sh
40+
41+
# JSON output for archival or comparison
42+
./benchmarks/run.sh --json > benchmarks/results/my-machine.json
43+
44+
# ~10s sanity check at reduced N
45+
./benchmarks/run.sh --quick
46+
```
47+
48+
Or per-runtime:
49+
50+
```bash
51+
node --expose-gc benchmarks/bench.js
52+
python3 benchmarks/bench.py
53+
```
54+
55+
The Python script adds `sdk/python` to `sys.path` directly, so `pip install`
56+
is not required — only `cryptography` needs to be available in the active
57+
environment.
58+
59+
## Reference results
60+
61+
### Apple M2 Pro · macOS 26.4 · Node 22.21 / Python 3.12.13
62+
63+
Source JSON: [`results/macos-arm64-m2pro.json`](results/macos-arm64-m2pro.json)
64+
65+
| Runtime | Scenario | p50 | p95 | p99 |
66+
| ---------------- | ------------- | ------ | ------ | ------ |
67+
| Node 22 / M2 Pro | decision-only | 0.13µs | 0.13µs | 0.17µs |
68+
| Node 22 / M2 Pro | with-proof | 153µs | 179µs | 247µs |
69+
| Python 3.12 / M2 Pro | decision-only | 0.96µs | 1.04µs | 1.17µs |
70+
| Python 3.12 / M2 Pro | with-proof | 158µs | 180µs | 227µs |
71+
72+
(All values are median-of-medians across 11 runs.)
73+
74+
### Linux x86_64 · *(pending — submit a PR with your `results/<machine>.json`)*
75+
76+
The harness is intentionally architecture-agnostic. Run it on your hardware
77+
and either commit the JSON to `benchmarks/results/` or paste the table into
78+
an issue. We will keep the reference table here in sync as more numbers come
79+
in.
80+
81+
## Notes on interpretation
82+
83+
- **Decision-only is dominated by function-call overhead** at this scale.
84+
The actual policy evaluation is a small constant-time check; the measurement
85+
largely reflects the runtime's own dispatch cost. Node's V8 inlines the hot
86+
path aggressively, hence the ~130 ns floor.
87+
- **With-proof is dominated by ECDSA signing** (~150 µs is roughly what a
88+
P-256 sign costs on Apple Silicon). Hashing and envelope serialization are
89+
comparable noise.
90+
- **The wide p99→max gap on a single run** (visible in the JSON's
91+
`sample_run_ns` field) is the OS scheduler, not the SDK. The
92+
median-of-medians strips most of it; if your SLO is tail-bound, plan for
93+
~500 µs decision-only worst case on a typical desktop and budget the
94+
difference against your scheduler's QoS guarantees.
95+
96+
## Reproducibility
97+
98+
The harness is deterministic given an unloaded machine. To compare numbers
99+
between runs:
100+
101+
1. Quiesce background processes (close browsers, dev servers, etc.).
102+
2. Plug into wall power if on a laptop.
103+
3. Run several times — the median-of-medians should settle within a few
104+
percent across consecutive invocations.
105+
106+
If your numbers diverge significantly from the reference table on similar
107+
hardware, please file an issue with the JSON output attached.

benchmarks/bench.js

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
#!/usr/bin/env node
2+
/* VELLA SDK — MIT License — Copyright (c) 2026 Vella Cognitive, LLC
3+
*
4+
* Latency benchmark for the Node SDK.
5+
*
6+
* Methodology:
7+
* - Warmup phase to stabilize JIT and resolve hot paths.
8+
* - K independent runs of N samples each. Per-run percentiles, then
9+
* median-of-medians across runs as the headline metric.
10+
* - GC quiesced between runs (requires --expose-gc; passed via run.sh).
11+
* - Ephemeral ECDSA P-256 signing key for the with-proof scenario.
12+
* - Timer is process.hrtime.bigint() (nanoseconds, monotonic).
13+
*
14+
* Usage:
15+
* node --expose-gc benchmarks/bench.js # human-readable
16+
* node --expose-gc benchmarks/bench.js --json # JSON only
17+
* node --expose-gc benchmarks/bench.js --quick # smaller N for sanity check
18+
*/
19+
20+
import crypto from "node:crypto";
21+
import os from "node:os";
22+
import { govern } from "../sdk/node/index.js";
23+
24+
const args = new Set(process.argv.slice(2));
25+
const QUICK = args.has("--quick");
26+
const JSON_ONLY = args.has("--json");
27+
28+
const RUNS = 11;
29+
const DECISION_INNER = QUICK ? 10_000 : 100_000;
30+
const PROOF_INNER = QUICK ? 1_000 : 10_000;
31+
const WARMUP_DECISION = QUICK ? 1_000 : 10_000;
32+
const WARMUP_PROOF = QUICK ? 200 : 2_000;
33+
34+
function ephemeralSigningKeyPem() {
35+
const { privateKey } = crypto.generateKeyPairSync("ec", { namedCurve: "P-256" });
36+
return privateKey.export({ format: "pem", type: "pkcs8" });
37+
}
38+
39+
function quantiles(samples) {
40+
const sorted = Float64Array.from(samples);
41+
sorted.sort();
42+
const at = (q) => sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * q))];
43+
let sum = 0;
44+
for (let i = 0; i < sorted.length; i++) sum += sorted[i];
45+
return {
46+
min: sorted[0],
47+
p50: at(0.5),
48+
p95: at(0.95),
49+
p99: at(0.99),
50+
max: sorted[sorted.length - 1],
51+
mean: sum / sorted.length,
52+
};
53+
}
54+
55+
function median(arr) {
56+
const s = [...arr].sort((a, b) => a - b);
57+
const m = Math.floor(s.length / 2);
58+
return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;
59+
}
60+
61+
function runScenario(label, callable, { warmup, inner }) {
62+
for (let i = 0; i < warmup; i++) callable();
63+
64+
const p50s = [];
65+
const p95s = [];
66+
const p99s = [];
67+
let firstRunQuantiles = null;
68+
69+
for (let r = 0; r < RUNS; r++) {
70+
if (typeof global.gc === "function") global.gc();
71+
72+
const samples = new Float64Array(inner);
73+
for (let i = 0; i < inner; i++) {
74+
const t0 = process.hrtime.bigint();
75+
callable();
76+
const t1 = process.hrtime.bigint();
77+
samples[i] = Number(t1 - t0); // nanoseconds
78+
}
79+
80+
const q = quantiles(samples);
81+
p50s.push(q.p50);
82+
p95s.push(q.p95);
83+
p99s.push(q.p99);
84+
if (r === 0) firstRunQuantiles = q;
85+
}
86+
87+
return {
88+
label,
89+
runs: RUNS,
90+
warmup,
91+
iterations_per_run: inner,
92+
median_of_medians_ns: {
93+
p50: median(p50s),
94+
p95: median(p95s),
95+
p99: median(p99s),
96+
},
97+
sample_run_ns: firstRunQuantiles,
98+
};
99+
}
100+
101+
function fingerprint() {
102+
const cpu = os.cpus()[0];
103+
return {
104+
cpu_model: cpu?.model ?? "unknown",
105+
cpu_count: os.cpus().length,
106+
arch: os.arch(),
107+
platform: os.platform(),
108+
release: os.release(),
109+
node_version: process.version,
110+
gc_exposed: typeof global.gc === "function",
111+
quick: QUICK,
112+
timestamp: new Date().toISOString(),
113+
};
114+
}
115+
116+
function fmtUs(ns) {
117+
return (ns / 1000).toFixed(2) + "µs";
118+
}
119+
120+
function main() {
121+
const fp = fingerprint();
122+
if (!fp.gc_exposed && !JSON_ONLY) {
123+
process.stderr.write(
124+
"warning: GC not exposed. Re-run with `node --expose-gc` for cleaner numbers.\n",
125+
);
126+
}
127+
128+
const signingKey = ephemeralSigningKeyPem();
129+
130+
const decisionOnly = runScenario(
131+
"decision-only",
132+
() => govern({ intent: "EXECUTE_CHANGE", evidenceMask: 1 }),
133+
{ warmup: WARMUP_DECISION, inner: DECISION_INNER },
134+
);
135+
136+
const withProof = runScenario(
137+
"with-proof",
138+
() => govern({ intent: "EXECUTE_CHANGE", evidenceMask: 1, proof: { signingKey } }),
139+
{ warmup: WARMUP_PROOF, inner: PROOF_INNER },
140+
);
141+
142+
const output = {
143+
runtime: "node",
144+
fingerprint: fp,
145+
results: [decisionOnly, withProof],
146+
};
147+
148+
if (JSON_ONLY) {
149+
console.log(JSON.stringify(output, null, 2));
150+
return;
151+
}
152+
153+
console.log("VELLA Node SDK — latency benchmark");
154+
console.log("───────────────────────────────────");
155+
console.log(`CPU : ${fp.cpu_model} (${fp.cpu_count} threads, ${fp.arch})`);
156+
console.log(`Platform : ${fp.platform} ${fp.release}`);
157+
console.log(`Node : ${fp.node_version}`);
158+
console.log(`GC : ${fp.gc_exposed ? "exposed (quiesced between runs)" : "NOT exposed"}`);
159+
console.log("");
160+
for (const r of output.results) {
161+
console.log(
162+
`[${r.label}] runs=${r.runs} N=${r.iterations_per_run} warmup=${r.warmup}`,
163+
);
164+
console.log(
165+
` median-of-medians: p50=${fmtUs(r.median_of_medians_ns.p50)} p95=${fmtUs(r.median_of_medians_ns.p95)} p99=${fmtUs(r.median_of_medians_ns.p99)}`,
166+
);
167+
console.log(
168+
` first run sample : min=${fmtUs(r.sample_run_ns.min)} p50=${fmtUs(r.sample_run_ns.p50)} p99=${fmtUs(r.sample_run_ns.p99)} max=${fmtUs(r.sample_run_ns.max)}`,
169+
);
170+
console.log("");
171+
}
172+
console.log("(JSON: re-run with --json)");
173+
}
174+
175+
main();

0 commit comments

Comments
 (0)