Skip to content

Commit b784816

Browse files
committed
Reframe benchmark as TwinBench for public launch
1 parent 737c721 commit b784816

27 files changed

Lines changed: 901 additions & 252 deletions

.github/ISSUE_TEMPLATE/submit-results.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: Submit Benchmark Results
3-
about: Submit DTaaS-Bench results for your runtime
3+
about: Submit TwinBench results for your runtime
44
title: "[Results] <Runtime Name> v<Version>"
55
labels: results
66
---
@@ -16,7 +16,8 @@ labels: results
1616

1717
## Benchmark Run
1818

19-
- **DTaaS-Bench Version**: 0.2
19+
- **TwinBench Version**: 0.2
20+
- **Run Profile**: local reference / SaaS runtime / multi-tenant-ready
2021
- **Date**:
2122
- **Platform**:
2223
- **Harness Command Used**:
@@ -33,6 +34,12 @@ python -m harness.runner --url <URL> --token <TOKEN> --user-id <ID> --name "<Nam
3334
**Coverage-Adjusted Verified Score**: /100
3435
**Rating**:
3536

37+
### Dimension Availability
38+
39+
- **Any partially measured dimensions**:
40+
- **Any unavailable dimensions**:
41+
- **Reason codes seen in artifact**:
42+
3643
### Dimension Scores
3744

3845
| Dimension | Verified | Projected | Coverage |
@@ -53,6 +60,7 @@ python -m harness.runner --url <URL> --token <TOKEN> --user-id <ID> --name "<Nam
5360
- **Diagnostics attached**: yes/no
5461
- **Metrics attached**: yes/no
5562
- **Run manifest attached**: yes/no
63+
- **Preflight completed**: yes/no
5664
- **Any runtime/provider incident during run**:
5765
- **Incident attribution**: runtime / upstream dependency / network / unknown
5866
- **Notes on projected components**:

CHANGELOG.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Changelog
22

3-
All notable changes to DTaaS-Bench should be documented in this file.
3+
All notable changes to TwinBench should be documented in this file.
44

55
The format is intentionally simple and release-oriented.
66

@@ -28,3 +28,20 @@ The format is intentionally simple and release-oriented.
2828
### Notes
2929

3030
- This release is focused on benchmark trust, open-source readiness, and compatibility with the current Nullalis gateway contract.
31+
32+
## [0.3.0] - 2026-03-25
33+
34+
### Added
35+
36+
- Public-facing TwinBench branding for personal AI assistant runtimes
37+
- New newcomer docs: getting started, run profiles, preflight, compatibility checklist, glossary, integration paths, and lightweight submission flow
38+
- Public results index and press kit
39+
- Outreach packet and outreach target list for competitor and lab wave
40+
- Dimension-level artifact status and reason code fields
41+
42+
### Changed
43+
44+
- Reframed README and specification around personal AI assistant runtime language
45+
- Clarified scale fairness: same-session contention is diagnostic, multi-user fanout should provision users first
46+
- Updated trust model and issue template to distinguish unavailable dimensions from product failure
47+
- Added provisioning-aware scale benchmark behavior and Nullalis local token discovery flow

CONTRIBUTING.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
# Contributing to DTaaS-Bench
1+
# Contributing to TwinBench
22

3-
We welcome contributions from the community. Here's how you can help.
3+
We welcome contributions from the community. TwinBench is meant to be approachable to runtime builders, researchers, and advanced indie teams, not just internal operators.
44

55
## Submit Benchmark Results
66

7-
The most impactful contribution is running DTaaS-Bench against your runtime and submitting verified results.
7+
The most impactful contribution is running TwinBench against your runtime and submitting verified results.
88

99
1. Run the harness: `python3.10 -m harness.runner --url YOUR_URL --token YOUR_TOKEN --user-id 1 --name "Your Runtime" --output results/your-runtime.json --markdown results/your-runtime.md --html results/your-runtime.html`
1010
2. Open an issue using the **Submit Results** template
@@ -29,12 +29,18 @@ Recommended submission attachments:
2929

3030
Please also read `docs/TRUST_MODEL.md` before submitting. The goal is not just a high score; it is a result other builders will trust.
3131

32+
If you are new to the repo, start with:
33+
34+
- `docs/GETTING_STARTED.md`
35+
- `docs/PREFLIGHT.md`
36+
- `docs/HOW_TO_SUBMIT.md`
37+
3238
## Propose New Dimensions
3339

34-
If you believe an important DTaaS capability is not covered by the current 10 dimensions, open an issue with:
40+
If you believe an important personal AI assistant runtime capability is not covered by the current 10 dimensions, open an issue with:
3541

3642
- What the dimension measures
37-
- Why it matters for DTaaS
43+
- Why it matters for persistent personal AI assistant runtimes
3844
- A proposed test protocol
3945
- How it would be scored
4046

PRESSKIT.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# TwinBench Press Kit
2+
3+
## One-Line Description
4+
5+
TwinBench is the open benchmark for personal AI assistant runtimes.
6+
7+
## Category Definition
8+
9+
TwinBench evaluates the runtime category behind persistent personal AI assistants: systems that remember across sessions, act autonomously, stay safe during background turns, and operate over time rather than only answering a single prompt.
10+
11+
## Benchmark Principles
12+
13+
- open and vendor-neutral
14+
- evidence-first
15+
- verified and projected scores separated clearly
16+
- unsupported behavior reported honestly
17+
- reference runtimes welcomed, not privileged
18+
19+
## Why This Matters
20+
21+
The market has benchmarks for coding agents, memory recall, and task completion, but not for the full runtime behavior expected from a real personal AI assistant. TwinBench is an attempt to define that category publicly and make it measurable.
22+
23+
## Reference Runtime
24+
25+
Nullalis is the current reference runtime because it demonstrates the full-stack behavior TwinBench is trying to name. It is not the owner of the benchmark and should be beatable in public.
26+
27+
## Where to Start
28+
29+
- [README.md](README.md)
30+
- [docs/GETTING_STARTED.md](docs/GETTING_STARTED.md)
31+
- [docs/RESULTS_INDEX.md](docs/RESULTS_INDEX.md)
32+
- [docs/OUTREACH_PACKET.md](docs/OUTREACH_PACKET.md)

0 commit comments

Comments
 (0)