diff --git a/docs/E2E_TEST_ARCHITECTURE_SPEC.md b/docs/E2E_TEST_ARCHITECTURE_SPEC.md new file mode 100644 index 00000000000..33708db64e1 --- /dev/null +++ b/docs/E2E_TEST_ARCHITECTURE_SPEC.md @@ -0,0 +1,978 @@ +# One end-to-end test system for Dash Platform: spec + +**Status:** revision 3. Revision 2 folded in a five-lens review (feasibility, coverage against the old suite, CI cost, maintainability across SDKs, simplicity). Revision 3 re-assesses draft #3549 as the base of the protocol-e2e framework, module by module, and makes its wallet-free core the base (§2.3, §3.4); §15 lists both rounds. Re-checked against `origin/v5.1-dev` at `03945d3aa7` (2026-10-29): the six newer commits change none of the cited facts. Spec only: no code, PRs or issues come from this document. +**Base:** `origin/v5.1-dev` at `b9abe42e03`. File:line references are at that commit. CI timings come from read-only `gh run view` on the nightly runs of 2026-10-26 (v5.0-dev) and 2026-10-27 (v5.1-dev). +**Supersedes:** the e2e parts of `docs/LEGACY_JS_RETIREMENT_SPEC.md` rev 2.1: all of §5, PRs 4-8, D4, the e2e rows of §10-§13 there, and §11.1's "one JS e2e harness" and grpc-js decisions. D2 stays, attached to the optional PR 4. The rest of that spec stays valid as one track of this plan (§10). +**Working artifact:** never commit this file (`AGENTS.md`, "Specs are working artifacts"). + +## 1. Problem + +Platform has many suites that talk to a network, each with its own harness, network selection, funding and data: + +- the legacy JS `platform-test-suite`; +- the wasm-sdk functional suite (Node and Chrome); +- rs-sdk's recorded-vector tests; +- the Swift integration tests; +- the Kotlin nightly; +- dashmate's e2e specs; +- the unmerged platform-wallet e2e framework (draft #3549). + +The legacy suite is the only one that covers Platform behaviour broadly over a network, and it does that through the legacy JS SDK, which is being retired. + +The retirement spec proposed moving those cases into the JS SDK's e2e suite. Ivan's objection: the old suite tested both Platform and the SDK. Moving its cases into an SDK suite makes that SDK's e2e carry Platform's functional coverage. + +**The answer here is no, that is not the right home.** Platform behaviour is the same for every client. Every SDK we ship is a binding over rs-sdk: +- wasm-sdk compiles rs-sdk to wasm; +- Swift goes through rs-sdk-ffi and rs-platform-wallet-ffi; +- Kotlin goes through rs-unified-sdk-jni. + +So protocol behaviour is tested once, in Rust, against a real network. Each SDK then needs only proof that its own binding and transport work, offline from recorded responses and live from a thin smoke. + +**Goal:** one structure, with +- a few levels with a clear purpose and owner, and a rule for where a new test goes; +- tiers (smoke, full) and domains (identities, documents, tokens, …); +- one way to select the network; +- one funding model; +- one fixture pipeline; +- a mapping of every old-suite case. + +The legacy suite is deleted at the end. + +## 2. Inventory + +### 2.1 What exists today + +| Suite | What it proves | Runs against | How it runs in CI | Size | +|---|---|---|---|---| +| **drive-abci unit and in-process tests** | Consensus rules per transition, queries and provers | In-process `Platform` on a temp GroveDB (`rs-drive-abci/src/test/helpers/setup.rs:24-117`) | Every Rust PR, nextest (`tests-rs-workspace.yml:445-486`) | About 4,800 test attributes; 2,531 under `state_transitions/`. Also 76 signed-answer and proof-compatibility tests in `tests/` | +| **Strategy tests** (`rs-drive-abci/tests/strategy_tests`) | Multi-block behaviour: epochs, quorum rotation, upgrades, withdrawals over time, voting, tokens, addresses | `FullAbciApplication` through `mimic_execute_block` (`src/mimic/mod.rs:147-419`), mocked Core, BLS-signed test locks | PRs run only `comprehensive_mixed_operations` and 5 collision tests (`tests-rs-workspace.yml:482`). The full binary runs on push and nightly (`:503-522`). 4 upgrade tests are in `tests-rs-nightly-long-running.yml` | 125 tests. `shielded_tests.rs` (6) is not compiled (`test_cases/mod.rs:14-15`) | +| **System-contract JS tests** | Contract JSON schemas, through wasm-dpp | In-process | `js-packages`, only when the contract package changes | 228 cases, mostly v1 schemas (retirement spec §3.4) | +| **rs-sdk `tests/fetch`** | Request building plus proof verification of real responses, read-only | **Offline:** 108 recorded directories (`tests/vectors`, 16 MB). **Live:** `network-testing` against the node in `tests/.env`, which needs Core RPC (`fetch/config.rs:101-110`) and SDK_TEST_DATA genesis data | Offline only, in the workspace nextest run (`--all-features`, where offline wins, `fetch/config.rs:201-252`). No job records or runs live | 130 test fns (+35 parametrised rows). **No writes.** 11 are ignored offline for missing vectors; 15 are ignored live because they need manual setup | +| **rs-sdk-ffi integration tests** | The C FFI over the same vectors | `dash_sdk_create_handle_with_mock` (`rs-sdk-ffi/src/sdk.rs:798`) | Workspace nextest | 34 integration tests. Test name → directory is a hand table (`tests/integration_tests/ffi_utils.rs:42-53`) | +| **Swift unit tests** (`SwiftDashSDKTests`) | Swift wrappers. 2 vector directories are replayed through the FFI mock (`SDKMethodTests.swift:107-138`); one of them, `test_identity_read_v1`, is an orphan recorded on testnet that no rs-sdk test regenerates | Offline | `swift-sdk-build.yml` → `run_tests.sh`, self-hosted macOS, about 5 min, on swift/rs-sdk/ffi paths (`tests.yml:333-350,517-538`) | About 861 | +| **Swift integration tests** (`SwiftDashSDKIntegrationTests`) | platform-wallet through FFI on a live network: SPV sync and restart, Core send, Core → Platform funding, identity registration | A local dashmate network (`Support/LocalDevnet.swift`, `IntegrationTestEnv.swift:33-41`); 1 testnet read | **Not in CI.** `run_integration_tests.sh` is manual (`RUN_INTEGRATION_TESTS=1`) | 15 | +| **SwiftExampleApp tests, UAT** | App logic. Manual or agent-driven UAT on testnet (`simulator-control` skill, `TEST_PLAN.md`) | Unit; UAT on testnet | Unit tests in `run_tests.sh`. The UI smoke workflow is manual and was last run 2026-05-26 (failed) | 228 + 5 UI | +| **Kotlin SDK tests** | JVM unit, instrumented; 3 testnet reads (`TestnetQueriesTest.kt:24-70`) | Emulator; testnet for reads | `kotlin-sdk-build.yml` on PRs (testnet tests skip). `kotlin-sdk-nightly.yml` daily with `-Ptestnet=true`, 12-19 min, no secrets | 538 + 58 + app 242 + 6 | +| **wasm-sdk functional** | wasm-sdk API end to end: 15 query specs (91 tests) and 6 transition specs (39) | **Hardcoded local:** `WasmSdkBuilder.local()` (`wasm-sdk/src/sdk.rs:41-43,452-466`), quorum sidecar `:22444`. Signs with the SDK_TEST_DATA genesis identities | `test-functional` (`tests.yml:673-687`) → `tests-packges-functional.yml`, 6-7 min, Node then karma. It runs **after** the wallet-lib and `dash` steps, so a legacy failure skips it (`:37-44`) | 130. 3 skipped: `identityCreate`/`identityTopUp` (no asset lock, `transitions/identity.spec.ts:201,211`) and a proof error | +| **js-evo-sdk** | Unit only (sinon stubs) | — | `js-packages` | 278. Its functional specs were deleted in #2868; `karma.functional.conf.cjs:9` points at the empty folder | +| **Legacy `platform-test-suite`** | Platform **and** the legacy SDK | Local; devnet/testnet via `configure_test_suite_network.sh`, which no workflow calls | `test-suite` (`tests.yml:636-671`): Node 17 min, Chrome batch 1 19-22 min, batch 2 7-8 min | 95 network tests + 5 unit (§8) | +| **wallet-lib, `dash` functional** | The legacy packages | Local | Inside `tests-packges-functional.yml` | 12 + 4 | +| **dashmate e2e** | Node lifecycle: setup, start, restart, stop, reset | Local (3 nodes); testnet fullnode; testnet "evonode", which is really a fullnode with `isHP` (`testnetEvonode.spec.js:61-64`) | `dashmate-e2e-tests` (`tests.yml:603-634`): 5, 3 and 3.5 min | 3 specs | +| **bench-suite** | Nothing; its setup is broken | — | None | Deleted by retirement PR 1 | +| **Draft #3549** (platform-wallet e2e) | Wallet and protocol flows, live | Testnet or devnets | **Never ran in CI** | §2.3 | + +### 2.2 Cross-cutting findings + +1. **No network test runs on Rust-only PRs.** `build-images` and every e2e job require one of `version-changed`, `e2e-tests-changed`, the schedule or a manual dispatch (`tests.yml:441,634,649,686`); `build-js` also runs on any `js-packages` change (`:429`). The `e2e-tests-changed` path list (`tests.yml:123-154`) has no Rust crates. An rs-dpp or rs-drive-abci change merges without any network test; for example, the 2026-10-28 run for "fix(dpp): state the low-S rule…" skipped the image builds. +2. **Every suite picks its network differently:** + - rs-sdk: `DASH_SDK_*`; + - wasm-sdk: hardcoded; + - legacy: `DAPI_SEED`/`NETWORK`/`EVO_SDK_DEVNET_NAME`; + - #3549: `PLATFORM_WALLET_E2E_*`; + - Swift: `RUN_INTEGRATION_TESTS` + `dashmate config get`; + - Kotlin: `-Ptestnet`. +3. **Every suite funds differently:** + - legacy: a wallet-lib faucet over DAPI streams; + - wasm-sdk: genesis identities, no Core; + - Swift: Core RPC `sendtoaddress` plus mining (`IntegrationTestEnv.swift:394-432`); + - #3549: a hand-funded mnemonic; + - Kotlin and UAT: a testnet faucet by hand. +4. **The vectors are stale, partly hand-edited, and hard to refresh.** + - **Old protocols:** they were recorded at protocol versions 4 to 14, none at 15 (`rs-platform-version/src/version/mod.rs:37`). 87 of 163 successful responses are from versions 4 to 12. + - **Hand edits:** 75 of 188 `msg_*.json` filenames don't match the hash of their request, and #4701 edited 70 files in place. + - **Hand-pinned values break on a re-record:** + - the recorded protocol version (`fetch/document.rs:15-19`, PV12); + - the proTxHash, pinned twice (`fetch/config.rs:279-284`, `rs-sdk-ffi/tests/integration_tests/config.rs:133-134`); + - the FFI directory table. + - **Never checked live:** CI never records and never runs the read suite against a network. +5. **Proof checking over a real network** is covered only by the legacy suite and wasm-sdk functional. + - Strategy tests check GroveDB proofs against the app hash. Only two places check the quorum signature (`strategy_tests/query.rs:80-165`, `address_tests.rs:2077-2240`). + - There is no proof check at all for shielded, moderation, fee-claim and key-limits transitions (`verify_state_transitions.rs:1359-1378`). +6. **The nightly is not trusted.** `NIGHTLY_STATUS.md` (2026-07-05, still about v4.1-dev) lists the legacy suite and the functional tests as failing or flaky. + +### 2.3 Draft #3549 + +References in this section are at #3549's head `0eff5ebc62`, under `packages/rs-platform-wallet/tests/e2e/` unless a path says otherwise. Revision 3 read every framework module, the 342 commits that touch `tests/e2e`, `TEST_SPEC.md` and the PR thread. 338 of those commits are authored by lklimek; the PR was opened by an agent account acting for lklimek. + +- **Size and stack:** +66,210 / −1,083 lines, 295 files, draft (`postponed`). 67 commits behind `origin/v5.1-dev` on 2026-10-30. + - It carries #5150 by merge ("follows its fate", PR body), and with it #5207, #5210 and #5528: about 21k lines. + - It also carries older copies of three more open PRs (#5564, #5565, #5570) and a change to the default runtime of dash-platform-macros' `#[stack_size]`. + - The e2e part is `framework/` (27 files, 17.6k lines), `cases/` (102 files, 21.2k) and `TEST_SPEC.md`. +- **What the framework depends on.** It uses **no API from the four stacked PRs**. It needs #3549's own platform-wallet hooks: + - `PlatformWalletManager::with_handlers`; + - the `test-utils` accessors `inject_address_balance` and `platform_payment_account_max_derived_index`; + - the `e2e-seams` shielded builders; + - `try_sync` from #5565, which is open and stacked on #5564. + + This was checked by names and signatures, not by compiling. The stack matters only at run time. The asset-lock accounting cases (CR-003/005/006, ID-002b, AL-001) assert #5150's fix, and the restore and drain cases need #5210's `WALLET_RESTORE`. +- **Coverage:** 105 tests. + - About 55 files are protocol-level: platform-address transfers, identities from addresses, top-ups and transfers, 17 token cases, DPNS register, a funded asset lock, and shielded transitions and abuse cases. + - About 47 are wallet-level: SPV, sync, gap limit, sweep, persistence, the shielded wallet. + - Every case, protocol-level included, funds and signs through `PlatformWallet`. + - **Not implemented:** documents, contracts, DashPay, withdrawals, DAPI Core. +- **Running it:** + - Testnet by default; it has also run on the porter and paloma devnets. + - `local` maps to regtest (`framework/config.rs:854`), but it has never run against a dashmate network. SPV has no regtest port default (`framework/spv.rs:507-519`), and nothing in it uses Core RPC. + - 30-50 min per full run. The last full run (testnet, 2026-09-09) was 158 passed and 35 failed of 193; it has not been re-run since the 2026-10-23 merge. +- **It would turn CI red as it stands.** + - `tests-rs-wallet.yml:241-250` and the workspace job run nextest on platform-wallet with `--all-features` and a 4 MiB stack. + - That turns on `e2e`, which satisfies the target's `required-features`, so every case whose name lacks "shield" runs and panics at the stack guard. + - Draft PRs skip CI, so this has never shown up. +- **Funding:** + - a mnemonic funded by hand: about 6.27 DASH net per full run on paloma, with a 550B-credit floor (`framework/config.rs:153`); + - per-test seeds registered before use and swept back afterwards; + - a cost-ordered planner and a Platform→Core refill. + + Core money is visible to it only through SPV. +- **What it found:** 33 Found-* entries, which are bugs it found in the wallet, rs-sdk and rust-dashcore. None of them is in Drive. Two were fixed in the SDK: #3650, and Found-033 by #5562. The "fail-closed persist" in its title was dropped at the 2026-09-30 merge, in favour of upstream #4586's retry of a failed cache write. +- **Fixtures:** none. +- **Comments:** they carry about 300 tracker references, such as QA-*, Found-*, Wave and Task numbers, and case IDs. About 55 of them are in the modules lifted in E2. + +**Its framework, module by module.** "Core" means it needs no platform-wallet and no SPV. + +| Module | Lines | What it is | Verdict | +|---|---|---|---| +| `sdk` | 265 | SDK build: DAPI list, short ban, protocol-version probe, trusted context provider with a contract cache | Core, once its SPV branch is dropped (`config.rs:320-324` picks SPV for devnet and local when no quorum URL is set) | +| `config` | 1,187 | `PLATFORM_WALLET_E2E_*` keys; testnet, devnet, local→regtest | About 350 lines core; the rest are bank and SPV knobs | +| `wait` | 1,358 | Waits | About 1,000 lines core: proof-verified waits for addresses, nonces, identities, contracts, DPNS names and tokens. About 340 wallet/SPV | +| `harness`, `mod` | 1,165 + 634 | Shared context, stack guard, `setup()` | About 280 lines core: the `OnceCell` context, the stack guard, SDK init. The rest builds the wallet manager, the bank and SPV | +| `shielded` | 1,024 | Shielded binding, prover, raw broadcast | `broadcast_raw`, `wait_commit_raw` and `assert_consensus_rejection` (`:618-760`) are core. The rest is wallet, partly behind `e2e-seams` | +| `tokens` | 1,386 | Token contract publish, mint, reads | About 70% core (dash-sdk only). Identity setup goes through the wallet | +| `signer`, `signer_derivation` | 332 + 201 | Identity and address signers derived from a seed | No wallet types beyond one call (`signer.rs:200`), but needed only by a seed-based bank. Protocol e2e uses `SimpleSigner` with random keys | +| `context_provider` | 174 | Quorum source plus contract cache; an SPV provider | Useful only with SPV | +| `funding_ledger`, `workdir`, `registry` | 762 + 125 + 286 | Per-test cost ledger, `flock` workdir slots, crash-safe wallet registry | No wallet types, but useful only with a persistent bank | +| The other 13 modules | About 11.8k | The bank wallet and its identity, the planner, the Core↔Platform refill, sweep-back, test wallets, gap limit, transaction accounting, SPV, the event hub, the SQLite persister | Wallet/SPV | + +**In short,** about 2k lines need nothing from the wallet and are worth lifting now, plus about 1k lines of token helpers later. The other ~14.5k lines are a wallet test harness. + +**Problems it already solved.** Tags: **L** = also hits protocol e2e on the local network; **R** = on a devnet or testnet. + +1. **A process-shared SDK needs one shared runtime** (L). + - rs-dapi-client pools lazily connected tonic channels (`rs-dapi-client/src/transport/tonic_channel.rs:79`, `connection_pool.rs:119-140`). Their workers live on the runtime that created them. + - Under plain `#[tokio::test]`, those workers die with the first test. The later tests get dead channels, the node gets banned, and nothing recovers. + - #3549 uses `#[tokio_shared_rt::test(shared)]` and a `OnceCell` context (`framework/harness.rs:86`). +2. **The test stack** (L). Debug-build futures plus proof verification overflow the default 2 MiB stack. A startup guard names the command to run (`framework/harness.rs:88-117`). +3. **Concurrent funding races the nonce** (L). + - #3549's bank spends from a platform address. It holds a mutex up to broadcast and retries nonce rejections (`framework/bank.rs:62,491-600,961`). + - An identity bank needs less (§5.2). +4. **Reading your own write across nodes** (R; L only with more than one DAPI address). + - One proof-verified read proves only that one node has the data. The next transition then lands on a lagging node ("Address does not exist"). + - The fix waits for 2 (or 4) proof-verified reads in a row (`framework/wait.rs:196-237`). The upstream issue, #3611, is open. +5. **Contracts deployed during a run** (L) must be registered with the context provider, or proofs over them don't verify (`framework/sdk.rs:58-73`). +6. **A small node pool and the 60 s ban** (R). A node one block behind gets banned; #3549 uses a 5 s base ban (`framework/sdk.rs:56,236-238`). With the local network's single address, any ban leaves no node at all, so banning is off there (§3.3). +7. **Accepted by `check_tx` is not committed** (L). `broadcast_raw`, `wait_commit_raw` and `assert_consensus_rejection` assert the typed consensus code (`framework/shielded.rs:618-760`). That is smoke item 7 (§6). +8. **Client and network versions** (R). + - A pinned SDK version decoded proofs in the wrong format: 83 of 88 failures on 2026-07-23. + - #3549 probes and refreshes the protocol version (`framework/sdk.rs:47,120`, `framework/harness.rs:380`), and prices fees at `sdk.version()`, not `latest()` (`dc0e47844c`). +9. **Fee estimates below the fee charged on chain** (L): #3040, still open. The harness adds headroom, and PA-3040 pins the bug. +10. **Key levels** (L): contract create signs with HIGH, token batches with CRITICAL (`framework/tokens.rs:20-33`). +11. **Timeouts that say why** (L): they report the last observed value and the streak (`framework/wait.rs:103-110`). Fetch errors are logged only at debug level, which is a gap to close. +12. **Wallet- or testnet-only** (sweep-back, the planner, cold SPV scans, devnet InstantSend under concurrency, memory above about 4 threads): these matter only for the wallet domains (E9). + +## 3. The target structure + +### 3.1 Levels + +| Level | Proves | Runs against | Lives in | Owner | +|---|---|---|---|---| +| **In-process** | Consensus rules, fees, state, multi-block behaviour, upgrades, provers and verifiers | No network | drive-abci unit and strategy tests, rs-dpp (gaining the system-contract schema tests, retirement PR 3), rs-drive, and platform-wallet's own tests for wallet logic that needs no network (`test-utils`, `lib.rs:24-25`) | Whoever changes the rule or the wallet logic | +| **Protocol e2e** | A real network accepts or rejects each transition and returns proofs that verify with quorum signatures. Also that the components work together: Core ↔ Platform (asset locks, withdrawals paid on Core), rs-dapi's Core endpoints and streams, Tenderdash timing | A live network: local in CI; a devnet or testnet on demand | **One Rust test crate**, `packages/rs-platform-e2e` (§3.3). Companion: rs-sdk `tests/fetch` in live mode (local only) | Platform team. A protocol change that is visible over the network adds its case here | +| **SDK checks** | Each shipped SDK correctly marshals real responses (**offline replay**) and works end to end over its own transport, signing and error mapping (**live smoke**). **No protocol rules** | Replay: recorded vectors. Smoke: a live network, selected as above | Replay: rs-sdk, rs-sdk-ffi, Swift (wasm later, §7.3). Smoke: wasm-sdk functional (Node + Chrome), Swift integration, Kotlin instrumented | Each SDK's maintainers. The SDK team owns the recording workflow; the author of a protocol bump re-records (§7.2) | +| **Node operations** | dashmate lifecycle | Local; public testnet (Core sync only) | dashmate `test/e2e` | dashmate maintainers | + +**Where a new test goes** + +| You are adding | Tests to add | +|---|---| +| A rule (validation, fee, state change) | In-process | +| Anything over several blocks | In-process strategy test | +| A new transition | In-process tests for its rules; one protocol-e2e happy path; a binding unit test in each SDK that exposes it | +| A new query or proof shape | A prover/verifier round trip in-process. One rs-sdk `tests/fetch` test, which runs live and records the vector. A replay row in each binding that replays it. This is the book's checklist, `coding-conventions.md:559-569` | +| Anything that needs real Core, Tenderdash, rs-dapi, quorum signatures, network timing, or a flow across components | Protocol e2e | +| A binding change | The binding's unit tests plus a replay row. Add a smoke case only for a new transport, signing or wait path | +| Node lifecycle | Node operations | + +**SDK smoke does not re-check protocol rules.** It keeps one rejected transition per SDK, asserting the error *code*, to prove codes reach the caller (`wasm-sdk/src/error.rs:413-416`). +- wasm-sdk functional already has a few protocol-flavoured cases, such as the token-payment rejections at `transitions/documents.spec.ts:458,477,497`. They stay until protocol e2e has equivalents. +- New ones are refused in review, by the book's rule. + +### 3.2 Tiers and domains + +**Domains** (protocol e2e): + +| Domain | Covers | +|---|---| +| `identities` | create from addresses, IS and CL asset locks; top-up; transfer; keys; masternode identities | +| `contracts` | create, update, history | +| `documents` | CRUD, transfer, purchase, index-only, queries | +| `dpns` | | +| `dashpay` | | +| `withdrawals` | identity withdrawals; the payout on Core | +| `dapi_core` | rs-dapi's Core endpoints and its three streams | +| `tokens`, `groups`, `voting` | after parity (§9) | +| `addresses`, `wallet`, `shielded` | after parity (§9) | + +**Tiers:** +- **smoke:** about a dozen cases across domains, under 10 minutes of test time. It runs on PRs whose paths can change network behaviour (trigger: E1; protocol-e2e smoke from E2). +- **full:** everything. It runs nightly, on manual dispatch, and before each release. + +**Selection:** +- Each domain is a module. The tier is a submodule: `tests/e2e//smoke.rs` and `full.rs`, or `smoke/` and `full/` directories with one file per case once a domain grows (#3549's cases are one file each, about 200 lines). The filters below work either way. +- Test names keep the repo's `should_…` convention (`coding-conventions.md:403`). Moving a test between tiers moves it between files; its name doesn't change. +- **Filters:** + - `cargo test -p platform-e2e --features e2e --test e2e -- ::smoke::` (smoke tier); + - `-- tokens::` (one domain); + - `-- tokens::smoke::` (both); + - no filter (full). +- Plain `cargo test`, not nextest: one process shares one context on one shared runtime (§3.3). nextest's one process per test would redo setup for every test. +- Every run sets `RUST_MIN_STACK=8388608`; the harness refuses to start without it (§2.3, item 2). +- **CI:** the workflow takes `workflow_dispatch` inputs `tier` and `domains`. Schedule means full; a PR means smoke. + - No label trigger. Adding `labeled` to `pull_request.types` (`tests.yml:17`) would restart and cancel the whole Tests run on every label change, because of `cancel-in-progress` (`:25-27`). + +**SDK checks have no tiers.** Each SDK's smoke suite is small by design, and replay always runs in full. + +### 3.3 The protocol-e2e crate + +``` +packages/rs-platform-e2e/ crate `platform-e2e`, publish = false + Cargo.toml [[test]] name = "e2e", required-features = ["e2e"] + .env written by the local-network setup (§4), gitignored + tests/e2e.rs + tests/e2e/framework/ sdk, config, wait, harness, broadcast, bank, faucet, asset_lock + tests/e2e//{smoke,full} .rs files, or directories with one file per case +``` + +**Dependencies for parity** (§9, E2-E6): `dash-sdk`, `dpp` (with `core_key_wallet` for `TransactionBuilder`), `dashcore-rpc`, `dapi-grpc`, `simple-signer`, `rs-sdk-trusted-context-provider` (with its `dpns-contract`, `dashpay-contract` and `withdrawals-contract` features), `tokio`, `tokio-shared-rt` (new to the workspace), `dotenvy` and `tracing`. Each one is used directly, or `cargo machete` flags it. **No platform-wallet and no dash-spv** until the wallet domains (E9). + +**Why a separate crate, not rs-sdk's `tests/`:** +- **CI safety.** `dash-sdk` is in the nextest package list, which runs with `--all-features` (`tests-rs-workspace.yml:445-482`). A live target there would run in every Rust PR's job. +- **Runner.** It runs under plain `cargo test`, not nextest. +- **Ownership.** Protocol coverage is owned by the Platform team, not the SDK. +- **Later dependencies.** It can depend on platform-wallet later without rs-sdk taking a dev-dependency on its own dependent. + +**Guards:** +- **No library target.** The framework lives under `tests/`, so there is nothing to add to the nextest list. If the core ever becomes a library crate (§3.4), that crate stays out of the nextest list too. +- **Not in the nextest list:** the crate is not added to `tests-rs-workspace.yml:445-482`, so nextest never builds or runs it. +- **`required-features = ["e2e"]`:** `cargo test --workspace` skips it. +- **Clippy still compiles it.** Workspace clippy runs `--workspace --all-targets --all-features` (`:227-232`), which keeps the code compiling and linted. +- **Workspace wiring:** add the crate to the root `Cargo.toml` members and to `.github/package-filters/rs-packages-no-workflows.yml`. + +**The framework: #3549's wallet-free core, plus local funding** (§3.4). + +**Lifted from #3549** (about 2k lines, §2.3): +- **`sdk`:** the DAPI list, the protocol version, and the trusted context provider with its contract cache. Contracts a test deploys are registered with it. +- **`config`:** the network part. +- **`wait`:** the proof-verified waits with their streaks, and a generic `wait_for`. +- **`harness`:** the `OnceCell` context on a shared runtime, and the stack guard. +- **`broadcast`:** `broadcast_raw`, `wait_commit_raw` and `assert_consensus_rejection`, for rejected transitions. + +**Changed while lifting.** The API stays the one #3549's wallet layer calls, so that layer can rebase onto it cheaply (§3.4). The only changes: +- **Keys:** the keys become `DASH_E2E_*` (§4). + - Secrets are redacted in every print, as #3549 already does for its mnemonic. + - Today `fetch/config.rs:106` and `rs-sdk-ffi/tests/integration_tests/config.rs:86` print whole configs, Core password included. +- **Context provider:** the SPV branch is dropped. `QUORUMS` is required for devnet and local networks. +- **System contracts:** they come from the trusted provider's contract features, not hand registration. +- **Protocol version:** on the local network it starts at `latest()`. The probe from version 10 stays for remote networks. +- **Banning:** it is off on the local network, which has one DAPI address. +- **Broadcast helpers:** they lose the `adversarial_enabled()` gate (`shielded.rs:71`). With the gate, smoke item 7 would fail with a configuration error. +- **Timeouts:** `wait_for` reports the last error in its timeout. This is additive; signatures don't change. +- **Comments are rewritten to be self-contained.** QA-*, Found-*, Task and Wave numbers, persona names and case IDs are replaced by the failure they describe. The mapping from old IDs goes in the PR description. The same rule applies in E7 and E9. +- **`.env` lookup:** the one that walks up to a `.claude` directory is dropped. + +**Not lifted for parity:** +- `signer`: `SimpleSigner` with random keys, plus the bank key from the environment, is enough. +- `context_provider`: it is SPV-only. +- Everything wallet-bound. That moves with the wallet layer (§3.4). + +**New:** `bank` (§5.1, §5.2), `faucet` (§5.3) and `asset_lock` (§5.3). Neither #3549 nor anything else in the repo has them for a local network. + +**Later from #3549:** the token helpers (E7) and the wallet layer (E9). + +**Rules for the lifted code:** +- Retries and streak waits wrap only setup steps: funding, and waiting until something is visible. The transition under test is never retried, so they can't hide a regression. +- The transition under test is built and signed by the protocol framework, never by platform-wallet, so a wallet bug can't fail a protocol test. On remote networks the E9 bank wallet may hand a case its Core coins; the case still builds and signs the asset lock itself. + +At startup the framework prints the network, its protocol version (from `getStatus`) and the bank identity. + +### 3.4 Why #3549's core is the base + +Three ways to get the framework, compared on what #3549 actually contains (§2.3): + +| | **A. Fresh framework** (revision 2) | **B. #3549's core as the base** (chosen) | **C. Land #3549 as is, grow it** | +|---|---|---|---| +| What happens | Write about 1.5k lines and read #3549 for ideas | E2 lifts its ~2k wallet-free lines into `rs-platform-e2e`. #3549 rebases onto them soon after | Merge it inside platform-wallet, then add the parity domains there | +| Reaches legacy deletion | Yes | Yes | No. It has no documents, contracts, DashPay, withdrawals or DAPI Core, and its protocol cases fund and sign through the wallet | +| Local-network funding | New code: genesis bank, Core RPC faucet, asset-lock builder | The same new code | None: a hand-funded testnet mnemonic, with Core money seen only through SPV. It has never run on a local network | +| Problems already solved (§2.3) | Re-implemented from #3549's code and notes, without its run history | Most come with the lifted code. The funding ones are re-applied in the new bank (§5.2); key levels come in E7 | Come with the code | +| Wallet domains later (E9) | Port about 14k lines of wallet harness and 47 cases onto a core with a different API, or keep two frameworks | Already rebased onto the API they were written against | Already there, in a second framework | +| Needs first | Nothing | lklimek's agreement to the copy | Rebasing it off the stack it carries by merge (its framework uses no stack API), or waiting for the stack. Then a nextest exclusion, because `--all-features` would run it | +| CI | e2e crate outside nextest | The same | Red until platform-wallet's two nextest jobs exclude the `e2e` binary | +| Weeks to deletion (§9) | About 5.5-7 | About 5.25-6.75 | Not comparable: protocol e2e still has to be built | +| Ownership | Platform team; #3549 stays separate | Proposed: the core and the protocol cases with the Platform team, lklimek reviewing; the wallet layer with lklimek | lklimek; protocol e2e is separate again | + +**B, because:** +- most of #3549's value for protocol e2e is in problems it already hit; +- its wallet layer is the most expensive part to write again; +- neither needs the stack. + +**B saves little time before deletion.** About as many lines get adapted as would be written, and the local funding is new either way. What it saves: +- debugging cycles in E2 and E8 (the lifted core already handles testnet); +- most of E7; +- most of E9. + +**How the lift works** (a proposal to lklimek; nothing here is settled without them): +- **Before E2:** lklimek agrees to the copy, and to the copy then being the maintained one. lklimek also reviews E2. +- **The copy:** + - It is taken at a pinned #3549 commit, recorded in the E2 PR. + - The PR body links #3549 at that commit and lists the lifted files, because `git blame` over its 342 commits is lost. + - The squash commits of E2 and E7 carry a `Co-authored-by:` trailer for lklimek, added by hand. +- **Soon after E2 merges, #3549 rebases onto it:** + - its wallet layer moves under `rs-platform-e2e/tests/e2e/framework/wallet/`, and its own copy of the core is deleted; + - from then on, fixes to the core are PRs to `rs-platform-e2e`; + - E9 then lands that already-rebased branch as lklimek's PRs, and they supersede #3549. + + This keeps the two copies from drifting for the months between E2 and E9. #3549 had 26 e2e commits in October alone. +- **If lklimek declines the copy,** the choice goes back to Ivan: lift with credit anyway, or option A. + +**Where the wallet layer lives** is lklimek's choice, needed by the time E2 merges, because the rebase depends on it: +- **Recommended: the same crate.** + - A `wallet` feature, never on by default, enables platform-wallet with its seams, platform-wallet-storage and dash-spv. They are optional normal `[dependencies]`, because Cargo has no optional dev-dependencies. + - One runner, one job, one set of keys. + - The `--all-features` problem goes away, because the crate is not in the nextest list. + - The early rebase needs no extra crate. +- **Alternative: it stays in platform-wallet's `tests/e2e`.** + - The core then becomes a small `publish = false` library crate that both use. + - That costs a crate, two key sets, and a `not binary_id(platform-wallet::e2e)` filter in both nextest jobs. + +## 4. Network selection: one set of keys + +**Every new live harness (protocol e2e, wasm-sdk functional, Swift, Kotlin) reads the same `DASH_E2E_*` keys from the process environment.** rs-sdk's own test configs keep `DASH_SDK_*`, written by the same setup. +- **Local:** the local-network setup writes them into a package `.env`. That happens in the cache-miss step of `.github/actions/local-network/action.yaml:44-58`, so `**/.env` caches them (`:64-71`). +- **Remote:** for a devnet or testnet, CI inputs and secrets set the environment directly. +- **No new file format and no new script.** `configure_test_suite.sh`/`configure_dotenv.sh` gain the lines. Writing on a cache miss matters: + - on a cache hit `logs/setup.log` doesn't exist; + - `dashmate wallet mint` can't run while the network is up (`DockerCompose.js:91-93`). + +| Key (`DASH_E2E_`) | local (written by setup) | devnet / testnet (env, CI secrets) | +|---|---|---| +| `NETWORK`, `DEVNET_NAME` | `local` | `devnet` + name, or `testnet` | +| `DAPI_ADDRESSES` | `http://127.0.0.1:` for native clients, as rs-sdk tests and Swift do (`fetch/config.rs:131-142`, `IntegrationTestEnv.swift:131`). JS uses `https://127.0.0.1:2443` with verification off | From inputs; the SDK default list for testnet | +| `QUORUMS` | sidecar `http://127.0.0.1:22444` | `https://quorums..networks.dash.org`, or testnet's trusted context | +| `CORE_RPC_URL`, `_USER`, `_PASSWORD` | local_seed, user `dashmate` (no whitelist, `getBaseConfigFactory.js:69-73`) | Absent: Core cases skip loudly | +| `BANK_IDENTITY_ID`, `BANK_TRANSFER_KEY`, `BANK_TRANSFER_KEY_ID` | The genesis identity of the runner's slot (§5.1) | A funded identity (secret) | +| `CORE_FAUCET_WIF`, `CORE_FAUCET_ADDRESS` | A key minted during setup, before start | Absent | + +**Adoption:** +- **Protocol e2e:** from day one. +- **rs-sdk and rs-sdk-ffi test configs:** keep `DASH_SDK_*` for now, written by the same setup. +- **wasm-sdk functional (S1):** + - Node reads `process.env`; + - karma gets an explicit allowlist through webpack `EnvironmentPlugin`, as the legacy suite does (`platform-test-suite/karma.conf.js:81-83`). The allowlist never includes Core RPC credentials or faucet keys. +- **Swift (S2):** reads the keys if they are set, and otherwise uses `dashmate config get` as today. +- **Kotlin:** when its write smoke comes (after E8), it forwards the keys as instrumentation arguments next to `sdk/build.gradle.kts:45-49`, rewriting `127.0.0.1` to `10.0.2.2`. + +**Which targets CI uses** +- **Local:** on PRs and nightly. +- **Testnet, read-only:** the Kotlin nightly, as today. + - rs-sdk `tests/fetch` can't do testnet reads: it needs Core RPC and genesis data. + - A testnet read smoke inside protocol e2e is possible only once remote runs exist (E8). +- **Remote write runs:** a manual dispatch with a bank-identity secret. These are Platform-only cases (E8). Core cases need a persistent bank with sweep-back, which arrives with #3549's wallet layer in E9 (§13). That bank only hands a case its Core coins; the case builds the asset lock itself (§3.3). + +## 5. Funding and test data: one model + +### 5.1 Roles + +- **Platform bank:** an identity with a CRITICAL TRANSFER key. + - **Local:** the SDK_TEST_DATA genesis identities `[1;32]`, `[2;32]`, `[3;32]`. Each has 10^13 credits and deterministic keys; key 3 is TRANSFER/CRITICAL (`create_genesis_state/test/tokens.rs:57,198-203`; `generateTestIdentityKeys`, `wasm-sdk/src/wallet/key_generation.rs:230-258`). + - The setup writes one identity per concurrent runner ("slot"). In CI each job has its own local network, so jobs never share a bank. A developer running two suites against one local network gives them different slots. + - **Remote:** a funded identity, given as a secret. +- **Core faucet:** protocol e2e only, local only. + - A key minted during setup (`configure_test_suite.sh` already mints `FAUCET_1`/`FAUCET_2` before start, `:24-32`). The script writes the e2e faucet straight into the crate's `.env`, avoiding the `mint.log` overwrite that `configure_dotenv.sh:35-36` depends on. + - Plus Core RPC. + - `mint` refuses non-local networks (`mint.js:51-53`), so devnets get no faucet. +- **Test wallets:** fresh random keys per test. Local networks are disposable, so nothing is swept back. + +### 5.2 One recipe in every SDK + +**A funded platform address:** transfer credits from the bank identity to a fresh address. +- rs-sdk: `TransferToAddresses::transfer_credits_to_addresses` (`rs-sdk/src/platform/transition/transfer_to_addresses.rs:36`). +- evo-sdk: `addresses.transferFromIdentity` (`js-evo-sdk/src/addresses/facade.ts:184`). + +**A funded identity:** create it from that address. +- rs-sdk: `put_with_address_funding_fetching_nonces` (`put_identity.rs:139`). +- evo-sdk: `addresses.createIdentity` (`:264`). +- Swift: `Address/Addresses.swift`. +- Kotlin: `registerFromAddresses`/`transferCreditsToAddresses`. + +**Minimums:** 100k credits per input, 500k per output, 200k to fund an identity (`dpp_state_transition_versions/v4.rs:46-48`). + +**SDK smoke uses only this recipe; none of it touches Core.** The genesis *platform addresses* (`test/addresses.rs:11-16`) have no known keys. They are query fixtures, not a funding source. + +**Bank concurrency:** +- **No funding mutex.** #3549 needs one because its bank spends from a platform address (§2.3, item 3). An identity bank doesn't: + - the SDK hands out distinct identity nonces under its own lock (`rs-sdk/src/internal_cache/mod.rs:280-301`); + - Drive accepts them out of order within 24 (`rs-dpp/src/identity/identity_nonce.rs:17`). + + `transfer_credits_to_addresses` fetches the nonce, broadcasts and waits in one call (`transfer_to_addresses.rs:98,123`), so a mutex around it would serialise every funding call through its wait. +- **Retries.** Only `InvalidIdentityNonceError` is retried, up to 3 times with backoff, after `sdk.refresh_identity_nonce`. The SDK refreshes it by itself only after a failed broadcast, not after a failed wait. +- **No retry on a timeout,** or the test address could be funded twice. + +### 5.3 Core-side funding (protocol e2e, local) + +1. **Coins.** The framework spends the faucet's UTXOs (`getaddressutxos`; local_seed has `addressindex`). + - It keeps a spent-outpoint set within its single process. + - It sends each funding transaction to the masternodes' mempools as well, as Swift does (`IntegrationTestEnv.swift:405-414`), so the InstantSend quorum signs at once. +2. **The asset-lock transaction** is built in test code with key-wallet's `TransactionBuilder`: + - inputs at `transaction_builder.rs:239`; + - payload at `:303`; + - the burn output at `:537,605-610`; + - signing through `TransactionSigner::sig_and_pubkey` (`:902-962`) or the blanket `impl` (`:987`). + + This duplicates part of platform-wallet's production builder (`rs-platform-wallet/src/wallet/asset_lock/build.rs:94-175`). That is accepted for parity. E9 adds one wallet-domain case that goes through the production builder. +3. **The proof.** rs-sdk's own `start_instant_send_lock_stream` + `wait_for_asset_lock_proof_for_transaction` (`rs-sdk/src/core/transaction.rs:223-308`) returns an instant proof, or a chain proof as the fallback. That also exercises rs-dapi's transaction stream. + - **ChainLock-proof cases** wait for Platform's `core_chain_locked_height` (`:402-420`). + - **InstantSend works on the local network:** SPORK_2 on (`configureCoreTaskFactory.js:237`), `llmqtestinstantsenddip0024` (`dash.conf.dot:127`), and Swift waits for `instantlock=true` (`IntegrationTestEnv.swift:417-432`). +4. **Withdrawals** check the payout on Core: `getaddressutxos` on the destination. If asset-unlock outputs turn out not to be address-indexed, the check falls back to `getrawtransaction` on the unlock txid (E4). + +**Binding coverage of asset-lock transitions:** +- wasm-sdk's skipped `identityCreate`/`identityTopUp` become wasm-sdk **unit** tests on a fixed asset-lock proof vector, comparing the transition bytes with rs-sdk's. +- The retirement spec's PR 4 (an asset-lock builder in rs-sdk/wasm-sdk/evo-sdk for JS tests) is **off the critical path**. It stays as an optional product feature for apps like dash-bridge. + +## 6. The protocol-e2e smoke set + +The same list is the minimum for each SDK's live smoke (§7.4). + +1. Status and protocol version. +2. Fetch the bank identity with proof. +3. Bank → fresh platform address, waiting for the proven result. +4. Identity from that address. +5. Contract publish, document create, query it back. +6. DPNS register and resolve. +7. One rejected transition, asserting its consensus code. +8. *Protocol e2e only:* an identity from an InstantSend asset lock, a top-up from one, and an identity → identity credit transfer. + +## 7. Fixtures + +### 7.1 What is recorded, and from what + +- **Source:** rs-sdk's `tests/fetch` read suite, run live against a fresh local network with SDK_TEST_DATA genesis data (`generated_data.rs:17-62`). +- **Not recorded:** protocol-e2e writes. Their state depends on fresh keys, entropy and timestamps, so every recording would churn every file. Replaying writes would need a fixed clock and seeded randomness through the SDK (§14). +- **Missing data** (contested names, groups) comes first from **extending the SDK_TEST_DATA genesis** (`create_genesis_state/test/`, Regtest/Devnet only, `test/mod.rs:33-40`). A write step is used only for what genesis can't hold, such as masternode votes. Recording never depends on protocol-e2e code. + +### 7.2 Recording, versioning, refresh + +**Recording** +- A `workflow_dispatch` job, "Record SDK vectors", with a `tests` filter input: + 1. builds the images at the commit; + 2. starts the local network; + 3. runs `packages/rs-sdk/scripts/generate_test_vectors.sh [filter]`; + 4. uploads `packages/rs-sdk/tests/vectors` as an artifact, with a diff summary. + + A developer commits the result. +- Local recording stays allowed (rs-sdk `README.md:183-191`, and contributors from forks), with the guards below as the provenance check. +- **A full regeneration happens only at a protocol bump or a release.** A new query records only its own directory, so a PR doesn't churn all 16 MB. + +**`vectors/manifest.json`** is written by every recording: +- protocol version; +- chain id; +- commit; +- the voter proTxHash. + +rs-sdk, rs-sdk-ffi and the binding replay tests read pinned values from it, instead of `document.rs:15-19` and the two proTxHash constants. + +**Guards (rs-sdk offline tests)** +- **The filename matches the request:** `rs-dapi-client/src/dump.rs:116-128`. The hash includes `type_name::>()`, whose output may change between compiler versions; replace it with a fixed tag in the same re-record. +- **No orphans:** every vector directory maps to an rs-sdk test namespace. Swift's testnet-recorded `test_identity_read_v1` gets replaced by a locally recorded directory. +- **A protocol floor:** every response was recorded at a protocol ≥ `PlatformVersion::latest() - 1`. The floor is computed, so nobody raises it by hand. The PR that bumps the latest protocol still passes; the next bump forces a re-record. + +**Ownership** +- The author of the protocol-bump PR owns the re-record. +- The stable-release checklist (`.claude/skills/release/SKILL.md`) gets a line. +- **If the network can't produce a vector,** the test is ignored with a reason and its directory deleted. A stale vector is never kept. + +**The live run** +- Nightly, rs-sdk `tests/fetch` runs live (`--no-default-features --features network-testing`) on the nightly local network. +- That proves the read suite passes against a real network at the commit. +- Freshness of the committed files comes from the manifest and the floor. + +**Format:** unchanged (Rust serde of prost types, `dump.rs:47-64`). Every binding replays through the Rust mock. + +### 7.3 Replay in each SDK, through Rust + +Every SDK sends its requests through rs-sdk and verifies proofs in Rust. "Replay the same vectors" therefore means exposing rs-sdk's mock through the binding. A native JS or Kotlin replay would have to rebuild rs-sdk's requests byte for byte and re-implement proof verification. + +| SDK | Today | Plan | +|---|---|---| +| rs-sdk | `SdkBuilder::new_mock().with_dump_dir()` (`src/sdk.rs:1645-1687`) | — | +| rs-sdk-ffi | `dash_sdk_create_handle_with_mock`, 34 tests | Read pinned values from the manifest | +| Swift | `SDK(mockVectorsDirectory:)` (`SDK.swift:371-385`), 2 directories | **R2:** a replay table, one row per query family | +| wasm-sdk | `mocks` is a default feature (`Cargo.toml:11-22`); the mock is used only in Rust unit tests | **Deferred** (§13). It needs more than byte loaders: an async in-memory expectation loader, because `load_expectations_sync` uses `block_on`, which always fails on wasm32 (`mock/sdk.rs:157`, `rs-dash-async/src/block_on.rs:121-130`). It also needs `DumpData::from_bytes`, an in-memory `MockContextProvider` (`rs-context-provider/src/provider.rs:314-408`), a mock build without `with_dump_dir`, and a test-only export under `./raw`. It would make `mocks` load-bearing, against the TODO at `rs-sdk/Cargo.toml:82`. wasm-sdk functional already exercises the same marshalling live | +| Kotlin | No mock over JNI | Not planned | + +**A replay row:** +- names the rs-sdk namespace it replays and the exact request parameters, because the mock matches the serialized request exactly (`rs-dapi-client/src/mock.rs:187-207`); +- calls the binding; +- asserts only fields that a re-record does not change (ids, key data, counts, genesis token balances), or values read from the manifest. Proofs, heights, quorum keys and fee-dependent balances change on every fresh network. + +Rows can be tracked per binding in the CI-checked `docs/sdk/sdk-parity-manifest.json` (`scripts/check_sdk_parity_manifest.py`). Then a new request type without a row, or without an explicit opt-out, fails CI. + +### 7.4 SDK live smoke + +- **JS: wasm-sdk functional** (Node + Chrome). It covers smoke items 1, 2, 5 and 6 today (`status.spec.ts:23`, `identities.spec.ts:37`, `transitions/contracts.spec.ts:39`, `documents.spec.ts:126,238`, `dpns.spec.ts:44,87`). **S1** adds: + - items 3 and 4 (no JS test covers transfer-then-create yet); + - an `err.code` assertion on one rejection (item 7). Today's rejections match the message only. + - It also removes the silent early return in `transitions/dpns.spec.ts:89-93`. +- **Swift: `SwiftDashSDKIntegrationTests`** against a local network. S2 adds smoke items 3-7 (bank → address, identity from it, contract and document, DPNS, an error code). Its existing wallet, SPV and Core-funding cases stay. + - New wallet-logic cases go to platform-wallet's own tests when they need no network, and to the protocol-e2e `wallet` domain when they do. + - Swift keeps FFI, persistence and keychain cases. +- **Kotlin:** the nightly's 3 testnet reads stay. A write smoke waits for the remote bank-identity secret (E8). + +## 8. Mapping the legacy `platform-test-suite` + +Counts are active cases; "smoke" and "full" are protocol-e2e tiers. The "Covered in-process by" column names the test each drop relies on. + +| Old file (active) | Goes to | Covered in-process by (drops) | +|---|---|---| +| `Identity.spec.js` (19) | `identities`. **smoke:** create from an IS asset lock, top-up, credit transfer. **full:** asset-lock amount too low; invalid IS lock (10513, no in-process test, since Core is mocked); reused output; no key proof (one negative); ChainLock create and fetch; documents failing without credits and working after a top-up; top-up with an invalid IS lock or reused output; transfer above balance; add and disable a key. **Masternode identities:** walk the masternode list (the rs-dapi stream, or `protx list valid`), then fetch owner, operator and voting identities. That needs no secret, so it runs on every network (`Identity.spec.js:664-731`). Get-by-id and by-key run inside the flows | — | +| `DataContract.spec.js` (7) → 4 | `contracts`. **smoke:** create and fetch. **full:** update; unknown owner (one negative) | Wrong version: `validate_update/v0/mod.rs:137` and drive-abci `data_contract_update/state/v0/mod.rs:830`. Incompatible schema: `document_type/methods/validate_update/common/mod.rs:1792`, `data_contract_update/mod.rs:1369`. Non-contiguous positions (10411): only generations 0/1 are tested (`try_from_schema/v0/mod.rs:800`, `v1/mod.rs:809`), so **add a `PlatformVersion::latest()` test** (L1-tests) before dropping | +| `Document.spec.js` (8) → 7 | `documents`. **smoke:** create and query. **full:** fetch by `$createdAt`; replace; delete; unique-index violation; **unknown type** (no in-process test asserts `InvalidDocumentTypeError` on the batch path; built at `batch/transformer/v0/mod.rs:559`). The retirement spec §5.6 was wrong here | Unknown owner: a legacy-client error, not a Platform rule | +| `IndexOnlyDocument.spec.js` (17) → 13 | `documents`. Keep: the contract; the agreeing like; **nonexistent post** (in-process coverage is on a stored type only, `creation.rs:7246`); covering-index synthesis; chained query with verified proofs; composite feed query; delete by values; second identity's like/delete; skipIfAbsent create, out-of-index and delete. **Subset projection without proofs and fetch-by-id refused** need a raw unproved `getDocuments` request asserting the server's error (`rs-drive/src/query/mod.rs:3927,2708`); rs-sdk always asks for proofs | Hashtag mismatch: `batch/tests/document/index_only.rs:542`. Both skipIfAbsent refusals: `:620`. Duplicate like: `:266` | +| `e2e/contacts.spec.js` (10) | `dashpay` (full). The test publishes **its own** contract with `requiresIdentityEncryptionBoundedKey`/`DecryptionBoundedKey` (`contacts.spec.js:46-48`). It adds contract-bound MEDIUM keys and checks `getIdentitiesContractKeys` (`:167-222`); rs-sdk `tests/fetch` has no test for that. Then profile create and update, contact request, approve, remove approval | — | +| `e2e/dpns.spec.js` (8) → 7 | `dpns`. **smoke:** register and resolve. **full:** TLD refused, homograph refused, subdomain without parent refused, search, resolve by record | Contract existence: rs-sdk `fetch/data_contract.rs:26` | +| `e2e/withdrawals.spec.js` (6) → 5 | `withdrawals` (full): withdraw and see the payout on Core; recent withdrawal updates; a completed withdrawal document can be deleted (`withdrawals.spec.js:115-118`); delete of an incomplete one and update refused. "More than balance" was a js-dash-sdk pre-check (`creditWithdrawal.ts:78`). It is **replaced** by a real consensus negative that broadcasts and asserts the code | Authentication-key purpose: a legacy client-side error. The Platform rule (`validate_state_transition_identity_signed/v0/mod.rs:171-174`) has no test anywhere, so **add an in-process test** (L1-tests) | +| `functional/core/*` (6) | `dapi_core` (full): broadcastTransaction, getTransaction ×2, getBlockchainStatus through rs-dapi-client's transports (`transport/grpc.rs:887-912`); getBlockHash ×2 over JSON-RPC | — | +| `functional/dapi/subscribeToBlockHeadersWithChainLocks…` (2) | `dapi_core` (full), with dapi-grpc's tonic client. Compare the header bytes with `getblockheader false`, and assert a ChainLock arrives (the old test checks `bestChainLock.height`) | — | +| Epoch, ProtocolVersion ×2, getStatus (4) | rs-sdk `tests/fetch` (live and replay); getStatus is also smoke item 1 | rs-sdk `fetch/epoch.rs`, `protocol_version_*` | +| `proofs.spec.js` (3) | **Dropped:** they check proof shape only; every fetch verifies proofs with quorum signatures | — | +| `e2e/wallet.spec.js` (5) | **wallet-lib behaviour dropped.** The **rs-dapi transaction-stream** coverage it gave (historical bloom-filter sync plus a new transaction) moves to `dapi_core`: subscribe, send to a faucet address, compare with Core RPC. The **masternode-list stream** (used by the masternode-identities case) also gets its own `dapi_core` case, checked against `protx list valid`. platform-wallet syncs over P2P and never calls either stream; `masternode_list_stream.rs` has no unit tests | — | +| `unit/createPlatformProofVerifier` (5) | **Dropped:** legacy adapter | — | +| Chrome runs (81 cases) | **Reduced on purpose** (open question). Kept: 130 wasm-sdk functional tests in Chrome, including writes. **Lost in the browser:** protocol flows, gRPC-web *server streaming* through Envoy (wallet-lib synced over the transaction and header streams in Chrome), and asset-lock identity create/top-up broadcast from a browser | — | + +**Rules** +- A case is dropped only when the test in the last column exists and covers the same rule. The L1-tests PR adds the two missing ones. +- Each family keeps at least one negative case in protocol e2e. +- **Totals.** Of the 95 network cases: + - about 73 are ported (one replaced by a real consensus negative); + - 5 are covered by rs-sdk's read tests; + - about 17 are dropped with a reason; + - 2 new `dapi_core` stream cases are added. +- **Parity** is every row merged. It gates retirement PR 8. + +## 9. Phased plan + +**Sizes** are rough lines changed, excluding lockfiles and vectors. "→" means "must merge after". + +### Phase 0: close the trigger gap + +**E1. A consensus PR runs a network smoke** (about 50 lines of YAML). +- **A new filter output, `protocol-e2e-changed`:** rs-dpp, rs-drive, rs-drive-abci, rs-dapi, rs-dapi-client, rs-sdk, rs-platform-version, rs-drive-proof-verifier, rs-context-provider, wasm-sdk, wasm-dpp2, js-evo-sdk, the setup scripts, and the e2e crate once E2 creates it. +- **Its OR goes only into `build-js`, `build-images` and `test-functional`** (later the protocol-e2e step). The legacy `test-suite` and `dashmate-e2e-tests` keep `e2e-tests-changed`. Without this split, every Rust PR would also run about 47 legacy runner-minutes and 3 dashmate jobs. +- **All three images are rebuilt on such a PR:** Drive, rs-dapi and dashmate-helper. `local-network` pulls all three, and the helper is built from the Dockerfile `build-js` stage, which copies the Rust crates (`Dockerfile:673-749`). +- **rs-sdk-only PRs reuse images.** Neither drive-abci, rs-dapi nor dashmate depends on rs-sdk. For a PR that touches only rs-sdk (7 of the 20 newly triggered PRs in the last 3 weeks), pull the base commit's images, falling back to the nightly SHA. That saves the 32-43 runner-minutes of image builds, and takes the helper image (13.9-19.1 min) off the critical path. +- **wasm-sdk functional stops depending on the legacy steps.** It is already a separate step (`tests-packges-functional.yml:37-44`); give it `if: ${{ !cancelled() }}` so a wallet-lib or `dash` failure can't skip it. +- **Interim:** wasm-sdk functional is the protocol smoke until E2. +- **Cost:** §11; this is an open question. + +### Phase 1: protocol e2e to parity, then delete the legacy suite + +**E2. The crate, framework, smoke set and CI** (about 3k lines, about 2k of them lifted from #3549). +- **Before it starts:** lklimek agrees to the copy (§3.4, §13 Q5) and reviews the PR. +- §3.3, §3.4 and §6, plus the setup lines that write the crate's `.env` and mint its faucet. +- **CI wiring:** + - **A separate compile job** builds the test binary (`cargo test --no-run`) in parallel with the image builds and uploads it as an artifact (stripped, `retention-days: 1`). Nothing is shared with the Rust job: that runs on self-hosted `rust-ci` runners with a local `target/` and no S3 sccache, while the e2e jobs are GitHub-hosted. Estimate 6-12 min warm, 15-25 min cold on `ubuntu-24.04`. + - **Smoke runs inside the functional job, on the same network, after wasm-sdk functional.** That is one network start per run, not two. It has to run after, not alongside: wasm-sdk functional signs with all three genesis identities. Running them in parallel would need more genesis bank identities. + - **Full** runs in its own nightly job, and on dispatch. + - The crate must be added to the Dockerfile's three `COPY --parents` lists, because the Cargo workspace must be complete in the image. As a side effect, an edit to e2e tests alone rebuilds the images. +- **Book chapter `book/src/testing/e2e-tests.md`:** levels, the placement table, tiers, the keys, funding and the smoke list. Short now, completed in PR 8. + +**E3 (→ E2). `identities`** (about 1.5k): Identity.spec. This is the Core-touching work. + +**E4 (→ E2). `withdrawals` + `dapi_core`** (about 1.2k), including the transaction, masternode-list and header streams. Also Core-touching. + +**E5 (→ E2). `contracts` + `documents`** (about 2k): DataContract, Document, IndexOnlyDocument, including the raw unproved request. + +**E6 (→ E2). `dpns` + `dashpay`** (about 1k). + +**L1-tests (independent; small). Two in-process tests:** +- a withdrawal signed with an AUTHENTICATION key; +- non-contiguous positions at `PlatformVersion::latest()`. + +**→ Retirement PR 8 (→ E3-E6, L1-tests). Delete `platform-test-suite`:** +- its job and image, `configure_test_suite_network.sh`, and the platform-test-suite part of `configure_test_suite.sh`, keeping the faucet mint and the `DASH_E2E_*` lines; +- `NIGHTLY_STATUS.md`; +- the `AGENTS.md:8,63,67,90` references. + +The book chapter is completed in the same PR. + +### Phase 2: fixtures (in parallel with Phase 1) + +**R1a. Recording and guards** (about 400 lines + vector churn). +- The workflow, `manifest.json`, the filename (fixed tag), orphan and floor guards. +- A re-record of everything that records on a fresh network. +- The nightly live run of `tests/fetch`. +- Secret redaction in the two test configs. +- Fixes to `rs-sdk/README.md:100,134` and the header of `generate_test_vectors.sh`. +- The floor applies only to directories that record on a fresh network until R1b; the rest stay on an explicit list. + +**R1b (→ R1a). Genesis data for groups and contested documents, and a vote step.** +- Re-record the rest and un-ignore the 11 offline-ignored tests. +- Remove the list. + +**R2 (→ R1a). The Swift replay table** (about 500). + +### Phase 3: SDK smoke alignment (after E2) + +**S1. wasm-sdk functional.** +- Reads `DASH_E2E_*`, with local as the default. +- Adds smoke items 3, 4 and 7. +- Moves `identityCreate`/`identityTopUp` to unit tests on a fixed proof. +- Deletes evo-sdk's dangling functional karma config. +- Updates `wasm-sdk/README.md:147-150`. + +**S2. Swift integration tests.** +- Run nightly on the self-hosted macOS runner against a local network, if that runner can run Docker. Otherwise they stay manual. +- Add the missing smoke items. +- Replace `swift-sdk/TESTING.md` (stale) with a pointer to the book. + +### Phase 4: beyond parity + +**E7. `tokens`, `groups`, `voting`.** +- **Minimum:** token transfer, mint, burn, freeze, pause, price/purchase, a group action, a contested DPNS name with masternode votes. +- **Network-only behaviour:** a replayed broadcast; nonce skew and `check_tx` refusals (#5562); masternode owner-key withdrawals. +- **From #3549:** its `tokens` helpers and its 17 TK cases move over. + - Their setup switches from the wallet to the genesis bank. Their bodies already call dash-sdk. + - They are renamed to `should_…`, and their comments are cleaned (§3.3). + - `tk_001` moves to `tokio_shared_rt::test(shared)`. Its `#[stack_size(…, worker_threads)]` runs the shared context on a runtime of its own, the hazard of §2.3 item 1, and needs #3549's macros change. + +**E8. Remote runs.** +- A manual dispatch with network inputs and a bank-identity secret, for Platform-only cases and a testnet read smoke. +- The lifted core already handles testnet and devnets that publish a quorums host: seed lists, the trusted quorum provider, the version probe, the ban period and the streak waits. E8 adds the dispatch, the secret and `requires_protocol(n)`. Devnets without a quorums host, like porter, need SPV and wait for E9. +- `requires_protocol(n)`: a loud skip on a remote network that runs an older protocol, a failure on local. + +**E9. Wallet domains: #3549's wallet layer lands.** + +By now #3549 has been rebased onto the core (§3.4), so E9 lands that branch as lklimek's PRs rather than moving code. +- **Domains:** + - `addresses`; + - `wallet`: live-network cases only. It folds in `rs-platform-wallet/tests/spv_sync.rs`; + - `shielded`: nightly, in its own shard. +- **What lands:** #3549's bank wallet, planner, refill, sweep-back, registry, workdir slots, funding ledger, SPV runtime, persister, the wallet waits, and its ~47 wallet cases. +- **Its protocol-level PA, ID and SH cases** move into the protocol domains (`addresses`, `identities`, `shielded`), funded from the genesis bank wherever they don't test the wallet. +- **Ownership** (proposed, §3.4): + - lklimek owns `framework/wallet/` and the `wallet` domain; + - the protocol-level cases sit with the Platform team, with lklimek reviewing. +- **Review routing:** the path rules for both go into the PR Hygiene policy that generates `.github/CODEOWNERS` (`dashpay/stale_prs_are_bad`, `policies/platform.json`). +- **Clean-up on the way in:** + - names become `should_…`; + - comments follow §3.3; + - wallet settings move from `PLATFORM_WALLET_E2E_*` to `DASH_E2E_WALLET_*`, with no aliases, since the old names were only ever in a draft. +- **Prerequisites:** + - **#3549's own platform-wallet hooks** land first, as a small PR: the `test-utils` accessors and the `e2e-seams` shielded builders. `with_handlers` can be replaced by one fan-out handler passed to `new`. + - **`try_sync`:** either land #5564 → #5565, or switch the harness to `coordinator.sync(force)`, which is on v5.1-dev. + - **The four stacked PRs are not prerequisites.** The cases that assert their fixes (asset-lock accounting, restore, drain) are `#[ignore = "needs #5150"]` (or `#5210`) until those PRs merge, so the nightly stays green. +- **Funding:** + - **Local network:** Core RPC sends to the bank wallet's Core address, and a genesis identity tops up its Platform address. No hand funding in CI. + - **Testnet and devnet:** they keep the hand-funded mnemonic and sweep-back. +- **CI:** a nightly shard. + - The runner has 16 GB and also hosts the dashmate network. #3549 ran out of memory above 4 threads on 19 GB with no network on the machine, so the shard starts at `--test-threads=2` and is measured. + - SPV runs against the local network, as Swift does, with an explicit regtest P2P port. +- **Housekeeping:** + - `check-wallet-closure.py` `EXPECTED_DEPENDENTS` gets `platform-e2e`; + - `tests-rs-wallet.yml`'s scoped clippy (`:216-227`) gets `--package platform-e2e`, so that a wallet-only PR can't break the wallet layer unseen. It does not go into the nextest list. +- **#3549** is then superseded by these PRs. + +### CI at each step + +| After | On a PR (paths permitting) | Nightly | +|---|---|---| +| Today | Legacy suite, 3 jobs (17 + 22 + 8 min); functional 7; dashmate 5 + 3 + 3. Only JS/e2e paths or a version bump; **nothing on Rust-only PRs** | The same; Rust long-running; Kotlin testnet reads | +| E1 | + build-js, all three images and functional on `protocol-e2e-changed` (about 135 more runs a week) | Unchanged | +| E2 | + compile job; smoke appended to the functional job | + protocol e2e full | +| Retirement PR 8 | − legacy suite | − legacy suite | +| R1a | Replay unchanged, inside the Rust job | + live `tests/fetch` on the nightly protocol-e2e network (a second binary to compile) | +| E7-E9 | Unchanged (smoke stays small) | Protocol e2e full split into domain shards, each with its own network, once it passes 30 min; the wallet shard (SPV, starting at `--test-threads=2`); `shielded` alone | +| S2 | Unchanged | + Swift integration (self-hosted) | + +### Effort (rough, engineer-weeks) + +| Work | Weeks | +|---|---| +| E1 | 0.25 | +| E2 | 2-2.5 (lift and adapt about 2k lines; about 1k new for the bank, faucet, asset lock and the smoke set; CI and the book chapter). Option A: 2.25-3 | +| E3-E6 + L1-tests | 3-4 | +| R1a | 1 | +| R1b | 1 | +| R2 | 0.5-1 | +| S1-S2 | 1 | +| E7-E9 | 4.5-6 (local SPV, which #3549 never ran; the hooks PR; memory tuning). #3549's token helpers, token cases and wallet layer move rather than being rewritten | + +**To legacy deletion (E1-E6, L1-tests): about 5.25-6.75 weeks** (option A: about 5.5-7), against 4-6 for the retirement spec's JS port. The difference buys two things: +- one protocol suite that no binding owns; +- a Core path with no dashcore-lib V24 problem. + +The fixtures, remote runs and wallet domains are separate work, listed separately. + +## 10. The legacy JS retirement track inside this plan + +**Unchanged from `LEGACY_JS_RETIREMENT_SPEC.md` rev 2.1:** +- U1, PR 0 and PR 0s (dashcore-lib vs Core 24); +- PR 1 (delete JS DAPI and bench-suite); +- PRs 2a/2b/2c (dashmate); +- PR S (stop publishing); +- PR 3 (contract tests to rs-dpp, now part of the in-process level); +- PRs 9 and 10; +- the npm deprecation path; +- the wasm32 wallet findings (its §6). + +**Changed:** + +| Retirement spec | Now | +|---|---| +| §5.2 option A: port into wasm-sdk functional | Port into protocol e2e (E3-E6). wasm-sdk functional stays the JS smoke | +| PR 4: asset-lock builder in rs-sdk/wasm-sdk/evo-sdk, on the critical path | An optional product feature, with D2 (`waitForProof`). The test builder lives in protocol e2e (§5.3) | +| PR 5: a JS Core faucet, karma middleware, a grpc-js Core client, a Node/browser split | Dropped. Replaced by E2 (a Rust faucet; dapi-grpc's tonic client) | +| PRs 6-7: port into wasm-sdk | E3-E6 | +| PR 8: delete after the JS ports | Delete after E3-E6 and L1-tests | +| §5.6: "unknown type covered by `batch/tests`" | Wrong; the case stays in protocol e2e (§8) | +| D4: remote runs dropped | Platform-only remote runs come back in E8. Core-side remote runs wait for #3549's bank wallet (E9, §13) | + +**Deadline:** +- The legacy suite breaks at bootstrap once CI's local network runs dashd:24 (retirement spec §4). Protocol e2e is unaffected: rust-dashcore parses V24. +- Parity takes slightly longer than option A, so U1/PR 0 (or PR 0s) is what keeps the legacy suite alive until E6. That is the retirement spec's first decision (D1). + +## 11. CI cost and time + +### Measured + +Nightly runs 40417401516 and 40599546110; PR runs 40826562377, 40823175547, 40827343941 and 40722904818. + +| Job | On a PR | Nightly | +|---|---|---| +| build-js | 9.3-16.2 min | 7-9 | +| Drive image | 10.0-12.8 | 11 cold / 1 cached. PR builds never write the layer cache (`cache_to_name` is set only on push), so "cached" never happens on a PR | +| rs-dapi image | 8.5-10.9 | 7 / 1 | +| dashmate-helper image | 13.9-19.1 (the slowest, and on the critical path) | 1 | +| Functional job | 7.5-9.3 (network start 2.7-5.1, wasm-sdk tests 2.4-2.7) | 6-7 | +| Legacy suite (3 jobs) | 45-51 runner-min | 17 + 19-22 + 7-8 min | +| dashmate e2e | — | 5 / 3 / 3 | +| Rust workspace | 10.7-26.8 (self-hosted `rust-ci`, warm `target/`) | 38-74 (with coverage) | +| Swift unit | about 5 | — | +| Kotlin nightly | — | 12-19 | + +### Volume (v5.1-dev, last 3 weeks) + +- **Merged PRs:** 54, with forward merges excluded. + - 32 (59%) match `protocol-e2e-changed`. + - 16 (30%) match today's `e2e-tests-changed`. + - **20 (37%) would trigger network jobs for the first time**, about 7 merged PRs a week. +- **Runs** matter more than merges: each push to a PR reruns. In the last 7 days, 364 non-draft Tests runs: about **135 newly triggered runs a week**, and about **215 protocol-smoke runs a week**. + +### Cost per run and per week (runner-minutes) + +| Item | Per run | Per week | +|---|---|---| +| E1, newly triggered runs: three images 32-43, functional 7.5-9.3, build-js where not already run (it runs today on 17 of the 20 PRs, through the JS filters) | 42-54 | About 5.7-7.3k (upper bound; about 320 if each PR ran once). The rs-sdk-only image reuse removes about a third | +| E2: compile 6-12 + smoke under 10, sharing the functional network | 16-22 | About 3.5-4.7k (×215) | +| PR 8: legacy suite removed | −45-51 | About −5k (about 109 runs) | + +- **Net:** roughly **+4-7k runner-minutes a week after PR 8**, and **+9-12k a week during the 5-6 weeks before it**, when the legacy suite and the new jobs overlap. +- **What it actually costs.** The repo is public, so GitHub-hosted `ubuntu-24.04` minutes are free. The real costs are: + - **Concurrency:** queueing behind other jobs. + - **PR latency:** the slowest path becomes the helper image plus functional, about 22-28 min under E1 and 28-35 min under E2. That runs in parallel with the 11-27 min Rust job. + - **AWS:** ECR pushes rise about 2.2× (from about 109 to about 245 runs a week, three images each), plus pull egress. The ECR lifecycle policy is unverified. +- **Not required to merge.** On v5.1-dev only `PR Hygiene` is a required check, so a red smoke is advisory unless it is made required (§13). + +### Rejected or deferred ways to cut it + +- **Skipping test- or doc-only changes:** rejected. + - Only 2-3 of the 20 newly triggered PRs are test-only, and one of those (`79cb05685a`) edits an inline `#[cfg(test)]` that no path rule can see. + - Negation also needs the `predicate-quantifier: every` workaround (`js-packages-no-workflows.yml`). +- **Choosing smoke or functional by crate:** rejected. The images are about 75% of the cost. +- **A merge queue:** one run per merge, about 5× fewer runs. But it needs required checks and a change in how the team works (§13). + +### Other costs + +- **The recording workflow** can reuse the nightly SHA images instead of building its own. +- **Swift nightly (S2):** the test images are built `linux/amd64` only (`tests-build-image.yml`), and the Swift runner is `[self-hosted, macOS, ARM64]`, shared with Swift PR builds. S2 needs either emulation or nightly arm64 builds of all three images (§12). +- **Savings that come later:** + - PR 8 also drops `+dash`/evo-sdk from the Dockerfile `build-js` stage (`:747-749`, "the platform test suite's proof verifier"), which shortens the helper build. + - The wasm-dpp saving in build-js waits for dashmate PRs 2a-2c, because dashmate needs the legacy stack to start the network. + +## 12. Risks + +| Risk | Effect | Mitigation | +|---|---|---| +| E1/E2 make most Rust PRs pay for images and a network | About +4-7k runner-minutes a week after PR 8; PR feedback at 28-35 min on those PRs; more ECR pushes | A dedicated filter; image reuse for rs-sdk-only PRs; smoke inside the functional job; Ivan decides (§13) | +| The local network is slow to produce ChainLocks (60 s miner, `setup_local_network.sh:5`) | ChainLock-proof cases are slow | Keep them in full; smoke uses InstantSend | +| A mistaken drop | Lost coverage | The "covered by" column must name an existing test; the L1-tests PR adds the two missing ones | +| Asset-unlock outputs aren't address-indexed | The withdrawal payout check fails | Verify in E4; fall back to `getrawtransaction` | +| The legacy suite dies on dashd:24 before E6 | A coverage gap | U1/PR 0 or PR 0s (§10) | +| The floor guard is red on day one | rs-sdk tests red | It lands with the R1a re-record, and an explicit list holds the directories R1b re-records | +| Genesis has only 3 bank slots | Contention when running locally | One suite per local network, or more genesis bank identities (test data only) | +| The self-hosted macOS runner can't run Docker, or the amd64-only test images | S2 blocked or slow | Build arm64 images nightly, keep the Swift integration tests manual, or reach a Linux-hosted network over the tailnet | +| lklimek declines the copy, or #3549 keeps changing after it | Two copies drift for months; E9 becomes a merge of two forks | Agree before E2 (§13). Pin the commit in the E2 PR. Keep the lifted API unchanged (§3.3). #3549 rebases onto the core soon after E2, deletes its own copy, and sends core fixes to `rs-platform-e2e` (§3.4). If lklimek declines, Ivan chooses between lifting with credit and option A | +| The lifted retries and streak waits hide a regression | A protocol bug passes | They wrap only setup steps; the transition under test is never retried (§3.3) | +| Lifted code carries testnet-tuned complexity the local network doesn't need | More code to review in E2 | Lift only the modules in §3.3; signers, the SPV provider, the planner, refill, sweep-back and registry come with the wallet layer. Remote-only code is not refactored before E8 | +| Lifted comments carry tracker IDs | Comments a reader can't follow | Rewrite them as the failure they describe (§3.3); the ID mapping goes in the PR description | + +## 13. Open questions (for Ivan) + +1. **A network smoke on every consensus PR.** Should PRs that change Rust consensus or SDK crates build the three images and run a network smoke? Today none of them run a network test before merge. + - **Cost (§11):** about 37% of merged PRs would newly trigger it, about 135 runs a week. That adds roughly 4-7k GitHub-hosted runner-minutes a week after the legacy suite goes (free on a public repo), about 28-35 min of PR feedback on those PRs, and about 2.2× the ECR pushes. + - **Two follow-ups:** make it a required check (otherwise it is advisory), and consider a merge queue to cut runs about 5×. + - **Recommended:** yes, through the dedicated filter, with image reuse for rs-sdk-only PRs; advisory at first. +2. **The browser reduction.** The old suite ran about 81 protocol cases in Chrome. The new structure keeps wasm-sdk functional's 130 Chrome tests (including writes), but loses three things: protocol flows in a browser, gRPC-web server streaming through Envoy, and asset-lock identity creation from a browser. Accept? Recommended: yes. The protocol code is the same Rust code, and the browser-specific transport stays covered. +3. **Remote runs.** At deletion time, protocol e2e runs only on the local network. Platform-only remote runs (a devnet or testnet, with a funded identity as a secret) come in E8. Core-side remote runs need a persistent bank with sweep-back, which #3549's wallet layer brings in E9; they are not planned before that. The legacy suite's remote mode is unused by CI today. Accept? Recommended: yes. +4. **Replay in JS and Kotlin.** You asked whether every SDK can replay the same recordings. They can, but only through the Rust layer. Swift and the FFI already do; this plan widens Swift. JS needs more work than it looks (§7.3), and it would duplicate what the live JS suite already checks. Kotlin would need a JNI addition. Recommended: JS later, if a marshalling bug slips past the live suite; Kotlin not planned. +5. **#3549 as the base.** You asked whether its framework, not a fresh one, should be the base. Mostly yes (§2.3, §3.4). + - **E2:** lift its ~2k wallet-free lines into `rs-platform-e2e`, credited to lklimek. + - **Soon after E2:** #3549 rebases onto that core. + - **E7:** its token helpers and 17 token cases move over. + - **E9:** its wallet layer lands as lklimek's PRs, superseding #3549. + - **Not landed as is:** it doesn't reach parity, and its protocol cases run through the wallet (§3.4). + - **What lklimek is asked to agree to,** before E2: + - the copy, and the copy then being the maintained one; + - reviewing E2; + - rebasing #3549 onto the core soon after; + - owning the wallet layer, while the protocol-level cases sit with the Platform team; + - the clean-up of names, comments and keys. + + By the time E2 merges, lklimek also chooses where the wallet layer lives. The recommendation is the same crate, behind a `wallet` feature. + - **If lklimek declines,** you choose between lifting with credit anyway and option A. + - Recommended: yes. + +**Already decided elsewhere:** what keeps the legacy suite alive if Core 24 reaches CI before parity is the retirement spec's D1 (fix dashcore-lib one last time). This plan assumes that answer. + +## 14. Alternatives rejected + +- **Protocol e2e in the JS SDK suite** (retirement option A). Protocol coverage would belong to one binding. It would also need a JS Core faucet and an asset-lock builder for tests only. +- **A hybrid:** Core cases in Rust, Platform-only cases (about 50) in wasm-sdk functional. It saves about a week. But "where does a protocol test go" gets two answers, split by an accidental criterion, and that is exactly the objection. +- **Protocol e2e in each SDK.** Writes the same cases three or four times. +- **A fresh framework** (revision 2). It would re-implement what #3549 already solved (§2.3, items 1-11) without its run history. It would also leave E9 a choice between porting about 14k lines of wallet harness onto a different API and keeping two frameworks. +- **Landing #3549 whole** (option C, §3.4). It doesn't reach parity: no documents, contracts, DashPay, withdrawals or DAPI Core. Its protocol cases fund and sign through the wallet. It has never run on a local network and can't fund itself there. Smaller points: it carries the stacked PRs, and `--all-features` would run it in the wallet CI. +- **Protocol cases funded through #3549's bank wallet.** Every protocol case would need SPV and the wallet, and a wallet bug would fail protocol tests. On every fresh local network the bank would also have to be funded through Core and SPV. +- **A funding mutex for the identity bank.** The SDK's nonce cache and Drive's out-of-order window already handle concurrent identity nonces. A mutex around the one-call transfer would serialise all funding through its wait (§5.2). +- **Protocol e2e inside rs-sdk's `tests/`.** The `--all-features` nextest run would run the live target in every Rust job, and it needs a different runner. +- **A repo-root profile file and a new configure script.** The same keys in the environment, written by the existing setup, do the job without a second source of truth. +- **A language-neutral vector format.** Every SDK replays through Rust. +- **Recording write flows.** That needs a seeded RNG and a fixed clock through the SDK, and would churn every vector. +- **nextest for protocol e2e.** It runs one process per test. +- **A PR label for the full tier.** It would cancel and restart the whole Tests workflow on every label change. +- **Path exclusions for test- or doc-only edits in consensus crates.** They catch few PRs and miss inline `#[cfg(test)]` edits. +- **A Rust funding CLI for the other SDKs.** No SDK smoke needs Core funding. + +## 15. Review log + +### Revision 2 → 3: #3549 as the base + +**Why.** Ivan pushed back on "write a fresh framework and harvest #3549's cases": its author has probably already solved many real problems, and writing from scratch would repeat that work. + +**What was done.** Four read-only passes over #3549 at `0eff5ebc62`: +- the core modules; +- the wallet and funding modules; +- its commit history, `TEST_SPEC.md` and the Found-* entries; +- its stacked PRs, its non-test diff, CI and the PR thread. + +**What changed:** +- §2.3 now has the module table and the list of solved problems. Corrected facts: + - 67 commits behind; + - the last full run was 158/35 on 2026-09-09, not 148/41; + - "fail-closed persist" is gone; + - the framework uses no API from the stack; + - the `--all-features` hazard is concrete. +- §3.3 lifts #3549's wallet-free core instead of writing a fresh framework; §3.4 compares the three options. +- §3.2 and §3.3 add the shared runtime and the 8 MiB stack, and allow per-case files in a tier. +- E2 needs lklimek's agreement first; #3549 rebases onto the core soon after E2. E7 and E9 move #3549's code instead of rewriting it. E8 notes what the core already handles. +- The effort, CI table, risks, Q3, Q5 and §14 follow from those changes. + +**Three-lens review of the revised parts:** +- **Feasibility:** + - §5.2 no longer copies #3549's funding mutex. An identity bank needs none, retries only nonce errors after a refresh, and never retries a timeout. + - The lifted `sdk` drops its SPV branch, so `QUORUMS` is required on devnet and local, and E8 covers only devnets with a quorums host. + - E9 lists `try_sync`. + - Pinned cases are `#[ignore]`d, not red, in the nightly. + - Banning is off on the single-address local network. + - The broadcast helpers lose their adversarial gate. + - System contracts come from the provider's features. + - `tk_001` moves to the shared runtime. + - The wallet shard starts at 2 threads. + - Remote Core cases build their own asset lock. + - Effort raised to 2-2.5 weeks (E2) and 4.5-6 (E7-E9), with option A's figures alongside. + - The shared-runtime claim was checked and holds. +- **Simplicity:** + - The signers and the SPV provider are not lifted, so the lift stays about 2k lines. + - No refactor while lifting, so the wallet layer rebases cheaply. + - §2.3 is trimmed and its duplicates removed. + - Option C is rejected for its deciding reasons first. + - Option A is described fairly. + - The walkthrough states where the read-lag problem applies. +- **Maintainability:** + - Lifted comments lose their tracker IDs. + - The drift between two copies is closed by rebasing #3549 soon after E2, not at E9. That is a second reason for keeping the wallet layer in the same crate. + - Ownership is a proposal to lklimek, named, with a fallback if declined. + - Credit is given through `Co-authored-by` and a PR-body link. + - Review routing goes through the PR Hygiene policy. + - Tiers can be directories. + - Wallet cases are renamed `should_…`, and the wallet keys move to `DASH_E2E_WALLET_*`. + - The `wallet` feature is optional and off by default. + - The wallet fast path's scoped clippy gets the crate. + +### Revision 1 → 2 + +- **Feasibility:** + - E1 needs its own filter output; the shared one would start the legacy suite and dashmate. + - The guards are made consistent (no library target; clippy compiles the e2e target). + - The local values are written in the cache-miss setup step, because `mint` can't run on a live network and `setup.log` doesn't exist on a cache hit. + - rs-sdk `tests/fetch` can't do testnet reads. + - wasm replay's real scope (`block_on` on wasm32). + - The recording/seeding order (now R1a, then R1b). + - Native clients use http on the local network. + - InstantSend already works locally. + - Use rs-sdk's proof wait. + - The `labeled` side effect. + - Tier filters that compose (`smoke.rs` modules). + - Corrected line refs. + - Devnets can't mint. + - Workspace and filter wiring; platform-wallet seams are a prerequisite for E9. +- **Coverage:** + - "Unknown type" has no in-process test, so it is kept. + - The rs-dapi transaction and masternode-list streams get `dapi_core` cases. + - "More than balance" never reached Platform; it is replaced with a real negative. + - The auth-key-purpose and positions rules get in-process tests. + - Correct "covered by" references for IndexOnly and DataContract. + - Masternode identities need no secret. + - The contacts row is described accurately. + - Raw unproved requests. + - The smoke list vs wasm-sdk today. + - The browser losses are named. + - Missing withdrawal and header-stream assertions. + - Minimum token/group/voting sets. +- **Maintainability:** + - The manifest replaces hand-pinned values; the orphan guard; Swift's testnet orphan vector. + - Replay rows assert only stable fields and name their request. + - Targeted recording with local recording allowed. + - A computed protocol floor and named ownership. + - Rs-sdk live mode is not a protocol-e2e domain. + - Wallet logic without a network stays in platform-wallet's tests. + - The same keys in the environment instead of a profile file; the karma allowlist; the Kotlin host rewrite; secret redaction. + - Seeding through genesis data. + - `should_` names kept. + - The protocol-flavoured cases in wasm-sdk functional are named. + - The doc updates are listed. +- **Simplicity:** + - The parity crate needs no platform-wallet or SPV. + - The framework is written fresh. + - The profile file and script are dropped. + - Four levels instead of five (replay and smoke are one "SDK checks" level). + - wasm replay deferred; Kotlin replay dropped. + - Deferred: the persistent bank, remote Core runs, `requires_protocol` (until E8), the label, and sharding until needed. + - An honest effort comparison. + - Porting PRs grouped by Core versus Platform-only. + - Open questions reduced; the browser and remote-run questions added. +- **CI cost** (the first attempt hit a rate limit and was re-run on revision 2): + - PR-side timings replace nightly ones; the dashmate-helper image (the slowest) is added. + - Weekly volume and net runner-minutes; the real costs are concurrency, latency and ECR. + - No compile-cache sharing with the Rust job; a separate compile job with an artifact. + - Smoke shares the functional job's network; image reuse for rs-sdk-only PRs. + - The "not a required check" caveat; the merge-queue option. + - The Swift arm64 image gap; the Dockerfile `COPY --parents` side effect; the transition-window cost. diff --git a/docs/E2E_TEST_ARCHITECTURE_WALKTHROUGH.md b/docs/E2E_TEST_ARCHITECTURE_WALKTHROUGH.md new file mode 100644 index 00000000000..2c0a05b187b --- /dev/null +++ b/docs/E2E_TEST_ARCHITECTURE_WALKTHROUGH.md @@ -0,0 +1,218 @@ +# One e2e test system: the plan in plain words + +This is the short version of `docs/E2E_TEST_ARCHITECTURE_SPEC.md` (revision 3), which has the evidence and file references. Five independent reviewers checked the first draft, each from one angle: does it fit the real code and CI, does it lose coverage from the old suite, what does it cost in CI, can it be maintained across the SDKs, and is it too big. Revision 3 answers your question about the draft PR #3549 (below). The spec's last section lists every change. + +It replaces the e2e part of the legacy-JS retirement plan. The rest of that plan still stands, as one track inside this one. None of these files is meant to be committed. + +## Your question first + +> The old test suite tested both Platform and the SDK. If its cases move into the new JS SDK's e2e suite, the SDK's e2e would carry Platform's functional coverage. Is that right? + +**No, and this plan doesn't do it.** Every SDK we ship is a thin layer over the same Rust SDK: +- the JS SDK is the Rust SDK compiled to wasm; +- Swift and Kotlin call it through FFI, a C-level bridge from their language into the Rust code. + +So Platform behaviour is tested **once**, in Rust, against a real network. Each SDK then only has to show that its own layer works, in two ways: +- **offline:** by reading recorded real responses correctly; +- **live:** with a small smoke test. + +## Your second question: build on #3549 instead of from scratch? + +> Its author probably already solved many real problems. Should its framework be the base rather than a fresh one? + +**Mostly yes. Revision 3 changes the plan to do that.** I read its framework module by module, its 342 test commits and its test spec. Almost all of those commits are lklimek's; the PR was opened by an agent account acting for lklimek. The framework is 17.6k lines, in two parts. + +**A small part that needs no wallet, about 2k lines.** It builds the SDK, picks the network, and waits for results with proofs. It already solves problems a fresh framework would run into again: +- one SDK shared by all tests dies with the first test unless all tests share one async runtime; +- the tests need an 8 MB stack, not the default 2 MB; +- a contract deployed during the run is unknown to the proof checker until it is registered; +- a rejected transition has to be checked after the block, with its exact error code, not just at mempool entry. + +It also solves problems that show up on testnet, which matters once we run there: +- after a write, one node can have the result while the next request lands on a node that is still behind; +- a node one block behind gets banned for a minute; +- a client and a network on different protocol versions misread proofs. + +**A large part that is a wallet test harness, about 14k lines:** +- the hand-funded bank wallet; +- moving money between Core and Platform; +- sweeping leftover funds back; +- SPV, persistence and shielded. + +It is good work, but protocol tests on our local network don't need it. That network is thrown away after each run, and genesis already gives us funded identities. + +**Three options:** +- **A. Fresh framework.** This is what revision 2 said. We would re-implement those fixes from its code without its months of runs, and later either port its wallet harness onto a different base or keep two frameworks. +- **B. Its wallet-free part becomes the base. Chosen.** + - The new protocol crate starts by copying that part, credited to lklimek, with the internal ticket references in its comments replaced by plain explanations. + - Soon after, #3549 rebases onto that copy and deletes its own, so the two can't drift apart. + - When we do tokens, after the old suite is deleted, its token helpers and 17 token tests move over. + - Its wallet harness and wallet tests land later as the wallet domains, owned by lklimek. + - It needs nothing from the four open wallet PRs #3549 is stacked on. +- **C. Merge #3549 as it is and grow it.** No: + - it covers none of what the old suite covers: documents, contracts, DashPay, withdrawals, DAPI Core; + - its protocol tests run through the wallet, so a wallet bug fails a protocol test; + - it has never run on our local network and has no way to fund itself there. + + Smaller points: it pulls in the four PRs it is stacked on, and our wallet CI would run its live tests and go red. + +**Honest cost.** B saves little time before the old suite is deleted: about as much code gets adapted as would have been written, and local-network funding is new either way. What it saves is debugging, the token work, and most of the wallet work later. + +**One thing I did not copy:** #3549 serialises funding with a lock. Our bank is a genesis identity, and the Rust SDK and Drive already handle concurrent identity nonces, so a lock would only slow every test down. + +**It needs lklimek's agreement before work starts** (decision 5). + +## The structure + +| Level | What it proves | Runs against | Who owns it | +|---|---|---|---| +| **In-process** | The rules: consensus, fees, state, multi-block behaviour, upgrades. Also wallet logic that needs no network | No network | Whoever changes the rule | +| **Protocol e2e** (one Rust test crate) | A real network accepts and rejects transitions and proves results with quorum signatures. Core ↔ Platform (asset locks, withdrawals paid on Core), DAPI's Core endpoints and streams | A live network: local in CI; devnet or testnet on demand | Platform team | +| **SDK checks** | Each SDK reads real responses correctly (offline replay), and works end to end: transport, signing, broadcast, error codes (live smoke). **No protocol rules** | Recorded files; a live network | Each SDK's maintainers | +| **Node operations** | dashmate setup, start, stop, reset | Local; testnet | dashmate maintainers | + +**Where a new test goes:** +- A rule goes in-process. +- Something that needs the real network goes to protocol e2e. +- A new query is one Rust SDK test: it runs live and records its file. The SDKs that replay files (the FFI layer and Swift) add a replay of it. +- A new transition gets in-process tests for its rules, one protocol-e2e happy path, and a unit test in each SDK that exposes it. +- A change to a binding (the thin per-language layer over the Rust SDK) gets that binding's unit tests and, where the SDK replays files, a replay. + +**Tiers.** Every protocol-e2e test is *smoke* or *full*, and belongs to a *domain*: identities, contracts, documents, DPNS, DashPay, withdrawals, DAPI Core, and later tokens, groups, voting, addresses, wallet and shielded. +- **Smoke** is about a dozen cases, under 10 minutes. It runs on PRs that can change network behaviour. +- **Full** runs nightly, by hand, and before releases. +- **Choosing** is a name filter: `::smoke::` for smoke, `tokens::` for one domain. +- The SDK checks have no tiers. They are small by design. + +**Network.** Every live suite reads the same environment variables (`DASH_E2E_…`): node addresses, the quorum-key source, Core RPC access, and the "bank" identity. +- **Local network:** our existing setup scripts write them, during the step whose result CI already caches. +- **Devnet or testnet:** CI inputs and secrets set them. +- There is no new settings file and no new script. + +**Money.** One model everywhere: +- **The bank.** On our local network, the three test identities that genesis already creates with known keys. On a devnet or testnet, a funded identity stored as a secret. +- **Funding a test.** Move credits from the bank to a fresh Platform address, then create identities from it. Every SDK already has both calls, and none of this touches Core. +- **Core.** Of the new tests, only the Rust e2e touches Core. Swift's existing live tests keep their own Core funding: + - a key minted during local setup pays for asset locks; + - the InstantSend lock arrives through the Rust SDK's own wait. InstantSend already works on the local network; the Swift tests rely on it; + - withdrawals check the payout on Core. + +## What we have today (the short inventory) + +- **In-process tests:** about 4,800 in Drive. On PRs only one big multi-block "strategy" test and 5 short ones run; the full set runs after merge and nightly. +- **The old JS test suite:** 95 network cases plus 5 unit, in Node and Chrome. About 47 runner-minutes per run. +- **The new JS SDK's functional suite:** 130 tests in Node and Chrome against our CI network, which it has hardcoded. +- **The Rust SDK's recorded tests:** 130 read-only tests replaying 108 recorded directories. Swift and the FFI layer already replay some of the same files through Rust. But: + - none of the recordings come from the current protocol (15); 87 of 163 come from protocols 4 to 12; + - 75 of 188 file names don't match their contents, a sign of hand edits; + - CI never re-records them, and never checks them against a live network; + - re-recording needs hand edits in four places. +- **Swift:** about 860 unit tests in CI. 15 live tests against a local network, mostly wallet and Core, which no CI job runs. +- **Kotlin:** unit and emulator tests on PRs. A nightly with 3 read-only testnet queries. +- **dashmate:** 3 lifecycle tests. +- **The draft Rust e2e PR (#3549):** + - About 66,000 lines, about 21,000 of them from four other open PRs it is stacked on. + - Strong on wallet, addresses, tokens and shielded; nothing on documents, contracts, DashPay or withdrawals. + - Testnet-first and funded by hand. It has never run in CI or on our local network. + - About 2,000 of its 17,600 framework lines need no wallet. They become the base of the new protocol suite (see above). + +**A real gap.** A PR that changes only Rust consensus code runs **no network test at all** before merge. The trigger for the Docker images and e2e jobs lists no Rust crates. + +## Where the old suite's cases go + +| Old area | New home | +|---|---| +| Identities (19): asset locks, top-ups, transfers, keys, masternode identities | Protocol e2e, identities. The masternode case reads the masternode list, so it needs no secret and runs on any network | +| Contracts (7), documents (8), index-only documents (17) | Protocol e2e, contracts/documents. 8 cases are dropped: 6 rules already tested in Drive (each drop names the test), 1 rule that first gets a test at the latest version, and 1 old-SDK error | +| DashPay contacts (10), DPNS (8) | Protocol e2e, dashpay/dpns | +| Withdrawals (6) | Protocol e2e, withdrawals (5), now also checking the payout on Core. The "wrong key type" case is dropped in favour of a new Drive test. "More than balance" only ever hit the old SDK's own check; it becomes a real Platform rejection | +| DAPI Core endpoints and the header stream (8) | Protocol e2e, DAPI Core. **Plus two new stream tests** (transactions, masternode list), because the old wallet test was their only coverage | +| Epoch, protocol version, status (4); DPNS contract (1) | Already in the Rust SDK's recorded tests | +| Proof "shape" checks (3), wallet-lib sync (5), plus the 5 unit tests of the old proof adapter | Dropped. They test the old libraries | + +About 73 cases are ported, 5 are already covered, about 17 are dropped with a reason, and 2 stream tests are new. + +**What the reviewers caught:** +- Three "already covered by Drive" claims were wrong: + - document of an unknown type; + - an index-only like on a missing post; + - one withdrawal case that never reached Platform. + + Those are now kept, or replaced with a real test. +- One rule had no test anywhere (a withdrawal signed with the wrong key type), and one was tested only at older versions (a contract-schema position rule). A small PR adds both tests before anything is dropped. + +**One deliberate reduction.** Protocol flows no longer run in a browser. What we keep in Chrome is the JS SDK's 130 functional tests, writes included. What we lose in the browser: +- the old protocol flows; +- browser streaming of the transaction and header streams; +- creating an identity from a Core asset lock in the browser. + +## Order of work + +1. **Now, if you agree to the first decision below:** consensus PRs start running a network smoke. That means building the three Docker images, reused when only the Rust SDK changed. Until the Rust suite exists, that smoke is the JS SDK's functional suite. It gets its own trigger, so the old suite doesn't start running on every Rust PR. A legacy-package failure no longer skips it. +2. **The Rust e2e crate.** Its framework starts from the wallet-free part of #3549's. Then it adds what our local network needs and #3549 never had: the bank, a Core faucet and asset locks. The step also brings the first smoke set, a CI job and a short book chapter. lklimek agrees first (decision 5). +3. **Port the old suite** in four PRs: + 1. identities; + 2. withdrawals and DAPI Core; + 3. contracts and documents; + 4. DPNS and DashPay. + + Plus the small PR that adds the two missing Drive tests. **Then delete the old suite.** +4. **In parallel, fix the recordings.** + - Record them in CI (or locally) through one script. A manifest file holds the values that are hand-pinned today. + - Add guards against hand edits, against orphan files, and against recordings older than the previous protocol version. + - Run the read tests live every night. + - Later, add test data to genesis so contested names and groups can be recorded too. + - Swift gets a fuller replay table. +5. **Align the SDK smokes.** + - The JS suite reads the same variables and adds the three missing smoke steps: fund an address from the bank, create an identity from it, check an error code. + - Swift's live tests run nightly on the self-hosted Mac runner, if it can run Docker. A catch: our test images are built only for Intel/AMD Linux, and that runner is an ARM Mac. It needs ARM images or emulation. +6. **After deletion,** in order of value: + - tokens, groups and voting, starting from #3549's token helpers and 17 token tests; + - runs against a devnet or testnet; + - the wallet, addresses and shielded domains. #3549's wallet harness and wallet tests, already rebased onto the same base, land as lklimek's PRs and replace #3549. + +**Effort:** about 5.25 to 6.75 engineer-weeks to delete the old suite. A fresh framework would be about 5.5 to 7. The previous plan (port into the JS SDK) was 4 to 6. The difference buys: +- one protocol suite that no single SDK owns; +- a Core path with no Core 24 parsing problem, because Rust already handles it. + +Recordings, remote runs and the wallet domains are separate work, listed separately. + +## Where this differs from the earlier plan and from your framing + +- **The port target changed.** The old suite's cases go to Rust, not into the JS SDK's suite. The JS suite stays as the JS smoke and gets no new protocol cases. Its few existing protocol-flavoured cases stay until Rust has them. +- **The JS SDK no longer needs an asset-lock builder for tests.** It becomes an optional product feature. Its two skipped JS tests become offline unit tests on a fixed asset lock. +- **The JS Core faucet, browser middleware and Node gRPC client** from the earlier plan are dropped. +- **#3549's framework is now the base, not just a source of ideas.** Its wallet-free part starts the new crate, and its wallet harness lands later on the same base. It is still not merged as is, because it doesn't cover what the old suite covers and its protocol tests run through the wallet. +- **"Real e2e and/or fixtures":** both, but only *read* requests are recorded. Recording writes would churn every file and would need a fake clock and fixed randomness in the SDK. +- **"Replay the same files in every SDK":** possible only through the Rust layer each SDK wraps. Swift and the FFI layer do it today, and Swift gets more. For JS it is bigger than it looks, and it overlaps the live JS suite, so I propose later. Kotlin is not planned. +- **"The same network selection everywhere":** the same variable names in every suite, written by the setup we already have, instead of a new settings file. +- **Testnet and devnet runs come back after deletion,** for Platform-only cases with a funded identity as a secret. Core-side runs on testnet need the draft PR's "bank with sweep-back" (a long-lived funded wallet that takes leftover test funds back), which arrives with its wallet domains. +- **The legacy JS retirement is otherwise unchanged:** the dashcore-lib fix, deleting the JS DAPI and bench suite, dashmate's cleanup, moving the contract tests to Rust, stopping publishing, deleting the packages, and npm deprecation. + +## Decisions I need from you + +1. **Should every PR that changes Rust consensus or SDK code run a network smoke test?** Each such PR would build the three Docker images and start a local network. Today none of these PRs run any network test before merge. + - **Who it hits:** about a third of merged PRs (37%) would newly trigger it, about 135 CI runs a week. + - **Cost:** + - roughly 4,000 to 7,000 extra runner-minutes a week once the old suite is gone, and 9,000 to 12,000 a week in the weeks before, while both run. That is free on a public repo, but it means queueing; + - those PRs wait about 28 to 35 minutes for this check; + - about twice as many image pushes to AWS. + - **Two follow-ups for you:** whether the check should block merging (today only "PR Hygiene" does), and whether a merge queue is worth it. A merge queue would cut the runs about five times. + + I recommend yes, through its own trigger, reusing images when only the Rust SDK changed, and not blocking merges at first. +2. **Is it OK that protocol flows stop running in a browser?** We keep the JS SDK's 130 functional tests in Chrome, writes included. We lose three things in the browser: the old protocol flows, browser streaming of transactions and headers, and creating an identity from a Core asset lock. I recommend accepting it: the protocol code in the browser is the same Rust code, and the browser-specific transport stays covered. +3. **Is it OK that, when the old suite is deleted, the new suite runs only on our local network for a while?** Runs against a devnet or testnet come back later (after tokens, groups and voting), for Platform-only cases, using a funded identity kept as a secret. Core-side runs on testnet need the draft PR's bank with sweep-back, which arrives with its wallet domains. The old suite's remote mode isn't used by CI today. I recommend accepting it. +4. **Should JS and Kotlin also replay the recorded files?** It is possible only through the Rust layer. Swift and the FFI layer already do it, and Swift gets more. For JS it is more work than it looks, and it mostly repeats what the live JS tests already check. Kotlin would need a small addition to its Java-to-Rust bridge (JNI). I recommend JS later, only if a bug in how the JS layer converts responses slips past the live tests, and no Kotlin replay. +5. **Build on #3549 (option B above)?** I recommend yes. lklimek would be asked to agree, before work starts, to: + - our copy of its wallet-free part becoming the maintained one, with lklimek reviewing that PR and credited in it; + - rebasing #3549 onto it soon after, and deleting #3549's own copy; + - owning the wallet harness and wallet tests, while the protocol-level tests sit with the Platform team; + - the clean-up: test names, comments without internal ticket references, and the shared variable names. + + lklimek also picks where the wallet part lives, by the time the first PR merges: + - **in the same new crate,** behind a switch that is off by default. I recommend this: one runner and one CI job, and our wallet CI can't pick it up by accident; + - **or in the wallet crate,** with the shared part split into a small crate of its own. + + If lklimek says no, you choose between copying with credit anyway and writing a fresh framework. + +**Already decided elsewhere.** If Core 24 reaches our CI before the Rust suite covers the old cases, the old suite stays alive through the dashcore-lib fix. That is the first decision of the retirement plan, and this plan assumes it. diff --git a/docs/LEGACY_JS_RETIREMENT_SPEC.md b/docs/LEGACY_JS_RETIREMENT_SPEC.md new file mode 100644 index 00000000000..07ac91d1aaa --- /dev/null +++ b/docs/LEGACY_JS_RETIREMENT_SPEC.md @@ -0,0 +1,728 @@ +# Retiring the legacy JavaScript stack: spec + +**Status:** revision 2.1. It folds in a five-lens review (feasibility, scope/simplicity, CI and release, dashmate, external users) and a consistency check of the result; §14 lists what changed. Spec only: no code, PRs or issues come from this document. +**Base:** `origin/v5.1-dev` at `60cc5f4508`. File:line references are at that commit. +**Working artifact:** never commit this file (`AGENTS.md`, "Specs are working artifacts"). + +> **Superseded in part (2026-10-29) by `docs/E2E_TEST_ARCHITECTURE_SPEC.md`.** That spec makes this plan one track of a single e2e test system. Superseded here: +> - **All of §5** (options, recommendation, funding in §5.3, the asset-lock builder in §5.4, the grpc-js Core client in §5.5). The old suite's protocol cases move to one Rust protocol-e2e suite, not into wasm-sdk's functional suite, and wasm-sdk functional stays as the JS SDK smoke. §5.6's rule survives, but its claim that "unknown document type" is covered by `batch/tests` is wrong; that case stays in the Rust suite. +> - **PR 4:** off the critical path; an optional product feature. D2 stays attached to it. +> - **PR 5:** dropped. What survives: its CI filter change becomes E1, and its `identityCreate`/`identityTopUp` un-skip becomes wasm-sdk unit tests (S1). +> - **PRs 6-7:** replaced by the Rust domain PRs E3-E6. +> - **PR 8:** deletion only, gated on E3-E6 and the in-process test PR L1-tests. Its ports of the read-only and Core-endpoint specs go to `dapi_core` (E4) and rs-sdk `tests/fetch`. +> - **§11.1:** "one JS e2e harness" and "header-stream test with grpc-js". +> - **§10, §12 and §13:** the e2e rows (port risks, verification for PRs 4-8, and the rejection of the Rust option B and of hand-written asset-lock serialization). +> - **D4:** the new spec recommends Platform-only remote runs after deletion (E8). That still awaits Ivan. +> +> Everything else here is unchanged: U1, PR 0/0s, PR 1, PRs 2a-2c, PR S, PR 3, PRs 9-10, D1, D3, deprecation, and the wasm32 findings (§6). + +## 1. Problem + +Platform still builds, tests and publishes a JavaScript stack that predates the Rust SDK: + +- **`dash`** (`packages/js-dash-sdk`) +- **`@dashevo/wallet-lib`** +- **`@dashevo/dapi-client`** (`packages/js-dapi-client`) +- **`@dashevo/wasm-dpp`** +- **`@dashevo/dash-spv`** (JS) +- **`@dashevo/grpc-common`** +- the JS side of `@dashevo/dapi-grpc` +- the private JS DAPI server (`packages/dapi`) +- `packages/bench-suite` + +All of it rests on the external **`@dashevo/dashcore-lib` 0.22**. + +Ivan wants it gone because: +- **It is insecure.** The `.yarnrc.yml` audit excludes for `elliptic`, `bn.js` and `crypto-js` exist for it. elliptic has an advisory with no patched release (§8.3). +- **It lacks features:** no tokens, platform addresses, shielded pool, groups or voting. +- **It costs maintenance on every protocol change.** + +The replacement exists: `@dashevo/evo-sdk` over `@dashevo/wasm-sdk` (rs-sdk compiled to wasm), with `wasm-dpp2`. + +What we need to keep is the end-to-end coverage `packages/platform-test-suite` gives today: a real local network driven from JavaScript, in Node and in Chrome. + +**Deadline: Dash Core V24.** dashcore-lib 0.22 cannot parse V24's formats (§4). Legacy client discovery has been broken on testnet since V24 activated on 2026-10-23 (draft PR #5609). The whole JS e2e suite stops working once CI's local network runs Core 24. + +## 2. Goals and non-goals + +**Goals** +1. Delete the legacy JS packages and their CI, Docker and release plumbing. +2. Keep e2e coverage of protocol behaviour over a real network, of the JS bindings, and of the browser (wasm + gRPC-web) path, with **one** JS e2e harness, not two. +3. Give external npm users a deprecation path, with notice and a final version. +4. Answer whether a wallet with Core sync can run in JS/wasm, and what it would take. + +**Non-goals** +- A browser SPV wallet (§6 sizes it as a separate initiative). +- Replacing dashmate's non-monorepo npm dependencies: `@dashevo/bls`, `@dashevo/dashd-rpc` (partly, see PR 2b), and the `@dashevo/docker-compose` fork. +- Retiring `wasm-drive-verify` (current-era code, with no legacy dependency). +- Any consensus change. +- Deprecating `@dashevo/dashcore-lib` itself. It has real non-Platform users (§8.1). + +## 3. Inventory + +### 3.1 Package by package + +| Package | npm | Who uses it in the repo | Verdict | +|---|---|---|---| +| `packages/dapi` (JS DAPI) | private | **Nobody at runtime.** Replaced by rs-dapi (#2716). Its image, Dockerfile stage and release job were removed in `d7c3536939` (#2827). dashmate's compose file has only `rs_dapi` (`packages/dashmate/docker-compose.yml:138-180`). Still tested on most rs-dpp PRs through the `*wasm-dpp` filter alias (`.github/package-filters/js-packages-no-workflows.yml:77-81`). | Delete now (PR 1) | +| `packages/bench-suite` | private | Not in any CI filter. Its setup is likely broken: nothing writes `DRIVE_LOG_PATH` (`scripts/configure_test_suite.sh:76`). | Delete now (PR 1) | +| `dash` 8.0.1 | public | platform-test-suite, bench. Its README already says "no longer recommended" (`packages/js-dash-sdk/README.md:10-12`). | Delete after the port | +| `@dashevo/wallet-lib` 12.0.1 | public | platform-test-suite (funding, asset locks). dashmate's copy is **dead code** (§3.3). Functional tests: `tests-packges-functional.yml:37-38`. | Delete after the port | +| `@dashevo/dapi-client` 5.0.1 | public | platform-test-suite; dashmate's readiness probe (§3.3); wallet-lib; `dash` | Delete after the port and PR 2a | +| `@dashevo/dash-spv` 6.0.1 | public | js-dapi-client (`lib/BlockHeadersProvider/BlockHeadersProvider.js:2`). Its npm dependencies `@dashevo/dark-gravity-wave` and `@dashevo/dash-util` have no other users. | Delete with dapi-client | +| `@dashevo/grpc-common` 5.0.1 | public | js-dapi-client, dapi, `dash` (`GrpcError`, `broadcastStateTransition.ts:13`), wallet-lib's workers, one platform-test-suite spec, **and dapi-grpc's npm package** (`packages/dapi-grpc/package.json`, `workspace:*`) | Delete together with dapi-grpc's npm package (PR 9) | +| `@dashevo/dapi-grpc` (JS side) | public | JS clients used only by js-dapi-client and dapi; `dash` declares it but never imports it. The Rust crate reads only `protos/` (`build.rs:43-76`) plus two examples, `clients/core/v0/rust` and `clients/platform/v0/rust` (`Cargo.toml:90-97`). | Keep the crate, protos and both Rust example directories. Remove the npm package and the multi-language codegen (PR 9) | +| `@dashevo/wasm-dpp` 5.0.1 | public | `dash`, dapi-client, wallet-lib, dapi, bench, test-suite; the ten contract packages (devDependency, tests only). No Rust crate depends on it. | Delete after PR 3 and the port | +| `@dashevo/dashcore-lib` 0.22 | public, **external repo** | Everything above, plus dashmate (§3.3) | Remove every in-repo use. Do not deprecate it (§8) | +| Ten `*-contract` packages | public | npm: `dash` (contract ids at runtime), test-suite, dashmate (withdrawals id), bench. Rust: `data-contracts`, dpp, drive and the SDKs. | Keep the crates. Move the JS tests to Rust (PR 3). Stop publishing to npm in the same PR as `dash` (PR S), because the published `dash` depends on them at runtime | + +These stay: js-evo-sdk, wasm-sdk, wasm-dpp2, wasm-drive-verify, dashmate, all Rust crates, `@dashevo/bls`, `@dashevo/dashd-rpc`, `@dashevo/docker-compose`. + +### 3.2 platform-test-suite + +Abbreviation: `PTS` = `packages/platform-test-suite`. + +**Size.** About 95 active network cases in Node, plus 5 unit cases. About 81 also run in Chrome through karma, in two batches (`PTS/karma.conf.js:21,30-52`; `.github/workflows/tests.yml:636-671`). + +**Every write goes through the legacy stack.** evo-sdk is used in two places only: +- as the proof verifier injected into `Dash.Client` (`PTS/lib/test/createPlatformProofVerifier.js:148-275`); +- for two queries (`PTS/test/functional/platform/IndexOnlyDocument.spec.js:321-411`). + +**Funding today** +1. `dashmate wallet mint` uses Core RPC to mine to two fresh keys, `FAUCET_1_*` and `FAUCET_2_*` (`scripts/configure_test_suite.sh:24-32`). +2. A wallet-lib wallet holds one of those keys and syncs over DAPI streams (`createFaucetClient.js:43-57`). dashcore-lib parses every transaction in that sync (`packages/wallet-lib/src/plugins/Workers/TransactionsSyncWorker/utils.js:54-56`). +3. `fundWallet` pays each test client's HD wallet (`packages/wallet-lib/src/utils/fundWallet.js:32-46`). +4. js-dash-sdk builds the asset locks with dashcore-lib (`packages/js-dash-sdk/src/SDK/Client/Platform/createAssetLockTransaction.ts`). +5. wallet-lib never accepts InstantSend locks (`TransactionsSyncWorker.js:70`), so every registration waits for a ChainLock. + +**Cases by feature area** + +| Area | File (active cases) | Port? | +|---|---|---| +| Identity | `Identity.spec.js` (19): asset-lock consensus errors 10530/10513/10504/20009, top-up, double spend, credit transfer, key add/disable, masternode identities | Yes. Needs the asset-lock builder (PR 4) and Core RPC (`protx info` for masternode keys) | +| Data contracts | `DataContract.spec.js` (7) | Yes, minus cases already covered in drive-abci (§5.6) | +| Documents | `Document.spec.js` (8), `IndexOnlyDocument.spec.js` (17) | Yes, minus cases already covered in drive-abci (§5.6) | +| DashPay | `contacts.spec.js` (10) | Yes | +| DPNS | `dpns.spec.js` (8) | Yes | +| Withdrawals | `e2e/withdrawals.spec.js` (6; already skipped in the browser, `:21-25`) | Yes; Core side via Core RPC | +| Read-only queries | Epoch, ProtocolVersion, getStatus (4) | Yes | +| DAPI Core endpoints | broadcastTransaction, getTransaction, getBlockHash, getBlockchainStatus, `subscribeToBlockHeadersWithChainLocks` (8) | Yes, with a Node gRPC client on `dapi-grpc/protos` (§5.5) | +| Proof shape | `proofs.spec.js` (3): checks the proof *shape* only | Drop | +| Wallet sync | `e2e/wallet.spec.js` (5): tests wallet-lib itself | Drop | + +**Also dropped**, because they test the legacy stack rather than Platform: +- the proof-verifier adapter unit test; +- client-side-only checks: `creditWithdrawal.ts:78`, the wasm-dpp signing-error string at `withdrawals.spec.js:188-202`, and wallet-lib's error at `Document.spec.js:126-155`; +- the skipped `waitForStateTransitionResult`, `getBlock` and `getMasternodeStatus` specs; +- dead helpers and unused dependencies. + +**The new SDK already has an e2e suite in CI.** `@dashevo/wasm-sdk test:functional` runs against the CI local network in Node and in karma (`.github/workflows/tests-packges-functional.yml:43-44`; `packages/wasm-sdk/package.json:35`): 15 query specs and 6 transition specs. They cover: +- credit transfer, key add/disable, Platform-side withdrawal; +- contract publish and update; +- document create, replace, delete and transfer; +- DPNS register and resolve; +- epoch, protocol and status queries. + +They sign with the identities `SDK_TEST_DATA=true` seeds at genesis. Those identities have deterministic keys (`generateTestIdentityKeys`, `packages/wasm-sdk/src/wallet/key_generation.rs:251-280`, which match the genesis keys in `.../create_genesis_state/test/tokens.rs:179-209`) and 10^13 credits each. The suite skips identity create and top-up for lack of a Core funding path (`tests/functional/transitions/identity.spec.ts:201-211`). + +### 3.3 dashmate + +Paths are under `packages/dashmate`. Dashmate does not use `dash`, wasm-dpp, evo-sdk or wasm-sdk. The inventory below was re-derived independently and is complete: there are no hits in `test/`, `scripts/`, `bin/` or templates beyond these. + +| Dependency | Live use | Replacement | +|---|---|---| +| wallet-lib | **None.** `src/util/generateHDPrivateKeys.js` is registered in DI (`src/createDIContainer.js:93,212`). With CLASSIC injection it is a parameter of `setupLocalPresetTaskFactory.js:28`, but it is never called. | Delete the file, the registration and the parameter | +| dapi-client + withdrawals-contract | `--wait-for-readiness`: `getDataContract(, {retries:0, prove:false})` against the gateway with a hard-coded `no-ssl` (`waitForNodeToBeReadyTaskFactory.js:29-52`). It swallows every error, retries every 500 ms with no timeout, and runs in CI through `yarn start` → `group start -w`. **Bug, found by reading the code (two reviewers agree; not executed):** the gateway offers a plaintext listener only for the `self-signed` provider (`templates/platform/gateway/envoy.yaml.dot:268-303`), while the default is `zerossl` (`configs/defaults/getBaseConfigFactory.js:270`) and `file` is also TLS-only. So `start -w` loops forever on every default evonode. `start.js:49-53` also runs the probe when `platform.enable=false`, where no gateway exists. | Keep the same query and meaning (gateway → rs-dapi → Drive): a unary `Platform/getDataContract` for the constant withdrawals id with `prove:false`, as native gRPC over `node:http2`. **Always use TLS**; the self-signed listener accepts TLS too. Use no SNI, as `src/ssl/probeServedCertificate.js:97-107` does. On verification, see "TLS in the probe" below. Add a timeout (per network, or a flag) that reports the last error. Skip the probe when Platform is disabled. Do not use rs-dapi JSON-RPC `getStatus`: it returns success with defaults when Drive errors, and caches the result for 10 s (`packages/rs-dapi/.../get_status.rs:62,84,118`) | +| dashcore-lib `PrivateKey` → address/WIF | Spork key (`configureCoreTaskFactory.js:49-76`), miner address (`startGroupNodesTaskFactory.js:116`), throwaway mining addresses (`generateBlocks.js:19`, `waitForBalanceToConfirm.js:22`) | A small helper: `randomBytes(32)` plus `ECDH.setPrivateKey`, compressed public key, WIF with the 0x01 suffix, hash160, base58check. Version bytes: local/devnet/testnet `0x8c`/`0x13`/`0xef`; mainnet `0x4c`/`0x10`/`0xcc`. Generate test vectors with dashcore-lib before removing it | +| dashcore-lib `Address` | Prompt validation in regular setup (`src/listr/prompts/validators/validateAddress.js:11`); no tests today | base58check plus version check; must accept P2SH as dashcore-lib does | +| dashcore-lib `Transaction` | Collateral and reward funding in local setup (`src/core/wallet/sendToAddress.js:28-58`) | See below | + +**TLS in the probe.** Skipping verification here is defensible: +- the probe dials the node's own gateway on a local address; +- it sends no credentials or secrets; +- it acts on nothing it receives except "the call succeeded". + +The certificate names the node's external address, so a verifying client fails against 127.0.0.1. The stricter option, which PR 2a should take unless it proves awkward: verify the chain (public CAs for zerossl and Let's Encrypt; dashmate's generated CA for self-signed and `file`), with `checkServerIdentity` against the configured external IP instead of the dialled address. Either way, a verification failure must be logged, not silently retried. + +**Collateral funding without dashcore-lib.** dashmate's RPC client cannot make these calls. `@dashevo/dashd-rpc` 19.1.0 types `createRawTransaction` as `'str str int'`, so arrays and objects are sent as `"[object Object]"`, and it has no `signrawtransactionwith*` at all. + +Use a small JSON-RPC helper over `jayson`, which is already a dependency (`src/tenderdash/createTenderdashRpcClient.js`), with the `dashmate` user: +1. Take inputs from `getaddressutxos`. +2. `createrawtransaction` with outputs `{collateral: amount, fundingAddress: change}`, a size-based fee of at least 1000 duff/kB, and no change output if the change would be dust. +3. `signrawtransactionwithkey(hex, [fundingWif])`. +4. `sendrawtransaction`. + +**Not `sendtoaddress`:** its change goes to a fresh wallet address. That breaks the next node's balance check (`registerMasternodeTaskFactory.js:95`), and its coin selection can spend the 0.1 DASH reward balance or a not-yet-registered collateral, all of which sit in wallet `main`. + +**Packaging and images** +- The dashmate-helper image copies the legacy packages at `Dockerfile:851-867` (line 850 is dashmate itself), and the stage is built `FROM build-js` (`:821`), which compiles wasm-dpp and wasm-sdk. Once dashmate has no legacy dependencies, the helper needs neither. The Node 26/cbor comment at `:833-835` goes stale with it. +- `scripts/pack_dashmate.sh:54-74` is generic and needs no change. +- dashmate's filter entry aliases `*wallet-lib`, `*dapi-client` and contract anchors (`js-packages-no-workflows.yml:99-107`). +- `packages/dashmate/Cargo.toml.template` is unreferenced and lists wasm-dpp. + +**Keep** the `3.0.0` config migration, the "Remove legacy DAPI stack" start task (`startNodeTaskFactory.js:115-133`) and the `platform-dapi-deprecated` profile. Upgrades from installs older than 3.0 still need them. + +### 3.4 System-contract JS tests + +There are 228 mocha cases across the ten packages. Each creates the contract from `schema/v1/*.json` with wasm-dpp and asserts the single `JsonSchemaError`'s `keyword`, `instancePath` and `params`. Underneath, they call the Rust `DataContract::validate_document` (`packages/rs-dpp/src/data_contract/methods/validate_document/mod.rs:17-38`). + +**Problems** +- **They test outdated schemas.** The latest protocol uses dpns v3, dashpay v2, withdrawals v2 and token-history v2 (`packages/rs-platform-version/src/version/system_data_contract_versions/v4.rs:12-21`). About 127 cases test v1 schemas of those four contracts. +- **CI does not run them on rs-dpp changes.** Each filter is `packages//**` only. That is why 14 were red on v5.1-dev until `c9c98b56de` (2026-10-28). + +**Rust already has the pieces** +- App Connect tests (`packages/rs-dpp/src/system_data_contracts.rs:292-379`) cover everything in its JS file. +- Moderation charter tests (`:381-1230`) cover most of its JS file. +- `JsonSchemaError` has `keyword()`, `instance_path()` and `params()` (`packages/rs-dpp/src/errors/consensus/basic/json_schema_error/error.rs:106-122`). + +wasm-dpp2 cannot host these tests: there is no document schema validation binding (`packages/wasm-dpp2/src/data_contract/model.rs:920-927`), and its `ConsensusError` exposes only `message` and `code`. + +**Decision.** Port to Rust in rs-dpp: `system_data_contracts.rs` modules or `packages/rs-dpp/tests/`. Target "every rule in the latest schema version has a Rust test", not a 1:1 copy of the v1 cases. Rust tests do not go in the contract crates themselves: dpp → data-contracts → contract crate means a second crate instance in tests. Also add the `data-contracts` crate, whose id test is at `packages/data-contracts/src/lib.rs:331`, to the nextest list (`.github/workflows/tests-rs-workspace.yml:471-479`). + +### 3.5 CI, Docker and release + +**CI jobs** +- `tests.yml`: the `js-packages` matrix (`:563-582`), `build-js` (`:425-431`, a full `yarn build` that compiles wasm-dpp), `test-suite` (`:636-671`), `test-functional` (`:673-687`), `dashmate-e2e-tests`. +- `tests-build-js.yml:22-31` runs dapi-grpc's `setup-codegen.py --install`. +- Only "PR Hygiene" is a required check on v5.1-dev and v5.0-dev, so removing jobs cannot block merges. + +**Filters.** `js-packages*.yml` alias almost everything to `*wasm-dpp`. Today: +- an rs-dpp change runs the dapi, dapi-client, wallet-lib, `dash`, dashmate and test-suite jobs; +- but **not** the wasm-sdk or evo-sdk jobs. + +Deleting a key whose anchor is aliased elsewhere breaks paths-filter: the `changes` job fails and Tests is skipped. Every removal PR must write the aliases out as literal paths. The manual-dispatch override (`tests.yml:415-416`) runs `test` for every key. + +**Docker** +- `build-js` cooks `wasm-dpp` and `wasm-sdk` (`Dockerfile:665`). +- `packages/wasm-dpp` appears in four `COPY --parents` lists (`:418,547,705,979`). +- `build-js` (`:643-752`) copies the legacy packages (`:726-730`) and builds `+dash` (`:747`) and `+@dashevo/evo-sdk` (`:749`). +- `build-dashmate-helper` (`:821`) and `build-test-suite` (`:877`) are both `FROM build-js`, and they are its only consumers. After PR 2c and PR 8, nothing uses `build-js`, so PR 9 deletes the stage outright. Three `COPY --parents` lists outside it (`:418,547,979`) still name `packages/wasm-dpp`. +- The `test-suite` image (`release.yml:262-272`) is never built in PR CI. Its only external user found is dashpay/dash-network-ci, which pins `1.0.0-dev.5` and has had no commit since 2024-02-21. + +**Release** +- `.github/actions/npm-release-build/action.yaml:126-132` packs every non-private workspace, and `release.yml:173-212` publishes each tarball. A removed or private workspace drops out automatically. +- Only stable releases get the `latest` dist-tag (`release.yml:153-161`). +- `yarn.config.cjs:7-11` offsets (dash-spv +1, `dash` +3, wallet-lib +7) are all legacy. +- Measured: build-npm took 28m56s for v5.0.1, about 10 minutes of it wasm-dpp. The dashmate-helper image takes about 19-22 minutes per architecture because of `build-js`. e2e wall-clock is set by the Drive image build, so dropping wasm-dpp saves runner minutes, not e2e time. + +### 3.6 Code that exists only for the legacy stack + +- **rs-dpp `extended-document`.** Only wasm-dpp enables it (`packages/wasm-dpp/Cargo.toml:38`); `ExtendedDocument` is in 18 rs-dpp files and no other crate. A follow-up. +- **dapi-grpc multi-language codegen:** + - `scripts/build.sh`, `codegen.json`, `setup-codegen.py`; + - `test-client-codegen.yml`; + - `npm-runner-validation.yml:66-69` (not `:62-65`, which is the release build itself); + - `npm-release-build/action.yaml:16`; + - `tests-build-js.yml:22-31`; + - `client_codegen` in `runner-requirements.json:205` / `.arm64.json:118`. These are checked for equality by `test_runner_image.py:356`, so changing them needs a runner-image promotion (`NPM_RUNNER.md:33-37`), done in a later PR. + +### 3.7 Docs and other references + +These are updated in the PR that removes the thing they reference (§7): +- `README.md:176`; +- `packages/README.md:5-7,19-23,29-30`; +- `AGENTS.md:7-8,22-24,28-33,67,90`; +- `CONTRIBUTING.md:72`; +- root `package.json` `test:*` scripts (`:11-28`). Keep `test:suite` while the test-suite job calls it (`tests.yml:655` resolves it from the root); +- the book: `api-reference.md:39-40`, `architecture/overview.md:282`, `introduction.md:163`, `state-transitions/lifecycle.md:221`, `getting-started.md:98`, and the `wasm/binding-patterns.md` and `wasm/error-macros.md` chapters, whose examples are wasm-dpp code; +- `docs/typedoc.json` (its only entry point is js-dash-sdk), `docs/tsconfig.typedoc.json`, `docs/SDK_ARCHITECTURE.md:27,216-229`; +- `book.yml:13-16,89,139-164`; +- `scripts/configure_dotenv.sh:14,16,38-40,53-55` (runs under `set -e` and `touch`es files inside the legacy packages; line 15 is rs-sdk and stays); +- `scripts/configure_test_suite.sh:68-77`; +- `scripts/prepare_docs.sh`; +- `.codecov.yml:27`, `pr.yml:49,51`, `.pre-commit-config.yaml:82,84`, `mkdocs.yml`, `docs/README.md`; +- `.claude/skills/release/SKILL.md`; +- `packages/swift-sdk/run_integration_tests.sh:27`, whose "is it built" sentinel is wasm-dpp's `dist`; +- `.github/scripts/tests/test_dashmate_pack.py:42-54` (uses wallet-lib only as a fixture name). + +Outside this repo: `dashpay/stale_prs_are_bad` lists `packages/js-dash-sdk/`. + +## 4. The near-term blocker: dashcore-lib against Core V24 + +### What breaks, in the order you hit it + +1. **CbTx v4 coinbase.** `coinbasepayload.js:63-80` handles versions up to 3. + - Masternode-list discovery and every wallet holding coinbase-funded coins fail to parse. + - This is already live on testnet. + - Fixed in dashcore-lib master by #318 (merged, unreleased). +2. **ExtAddr (ProTx v3) masternode entries.** The SML merkle root no longer matches. + - dashcore-lib #319 fixes it, plus a 0.25.0 regression that caps LLMQ_60_75 at 24 quorums. + - #319 is open: +8,451 lines, mostly a fixture. +3. **Stale testnet whitelist** in dapi-client's `networkConfigs.js`. #5609 regenerates it. +4. **v2 asset unlocks** (Platform PV15 with V24; sibling `ASSET_UNLOCK_V2_SPEC.md` §0 hands the JS side to this plan). + - `assetunlockpayload.js:96-99` accepts only payload version 1. + - Even with that relaxed, the txid is wrong. A v2 txid is `sha256d` with the trailing 132 payload bytes zeroed (rust-dashcore `qualified_asset_unlock.rs:65-74`, `transaction/mod.rs:247-279`), but dashcore-lib hashes the full serialization (`transaction.js:171-173`). + - No upstream change exists yet. +5. **Unknown special transaction types (10-12) and ProRegTx v3** throw (dashcore-lib #317). The suite's masternode-identity case parses a ProRegTx (`PTS/test/functional/platform/Identity.spec.js:690`). + +### Effects + +- **wallet-lib:** a parse error is uncaught (`TransactionsSyncWorker/utils.js:54-56`, before address filtering). +- **The whole JS e2e suite:** fails at bootstrap on any V24 regtest because of item 1, before item 4 matters. +- **Not affected:** dashmate and rs-dapi. +- **Why CI is green today:** dashmate defaults to `dashpay/dashd:23` (`packages/dashmate/configs/defaults/getBaseConfigFactory.js:57`), and v2 emission is not implemented yet. +- **The deadline:** dashmate's switch to dashd:24, or v2 emission landing while CI runs V24, whichever comes first. +- **Beyond Platform:** Insight (`insight-api`, `dashcore-node`, both `^0.22.0`) parses every block with dashcore-lib. It will break on v2 unlocks once PV15 emits them on a V24 network. Insight is the UTXO source for dash-bridge and xchainjs. This is a PV15 release risk owned by whoever ships Insight; flag it in the asset-unlock-v2 PR. + +### Plan + +This is aligned with draft #5609. + +- **Upstream (dashcore-lib maintainers; Ivan is one).** On master: + - land #319; + - add the v2 asset-unlock rule (version bound **and** stable txid, with the DIP-0027 vectors at `qualified_asset_unlock.rs:303-308`); + - add a raw-payload fallback, so unknown special types or versions stop throwing. + + Release from master under `latest` (0.25.1 or 0.26.0). The release workflow at `v0.22.0` cannot run today (retired runner image, `actions/cache@v2`, a classic `NPM_TOKEN`), so the release needs either the CI update on master or a manual 2FA publish. + + Also backport the v2 rule and the fallback to a **0.22.x** release for `^0.22.0` consumers. Platform does not need it (it moves to the master release), but Insight (`insight-api`, `dashcore-node`) does: its `^0.22.0` range never picks up 0.25+. It must use a non-`latest` dist-tag (`0.22-lts`), because the library's workflow publishes non-"dev" tags as `latest`. Afterwards, check that `dist-tags.latest` is still the master release. This backport is recommended as part of U1. If the maintainers decline it, Insight must move to the master release itself before PV15 activates with V24. +- **Platform (#5609).** Bump every workspace package from `~0.22.0` to that release. 0.22 → 0.25 only adds an optional `llmqParamsOverride` argument. Keep the whitelist refresh. + - Backport to v5.0-dev; that is what gives the final legacy version (§8.2) working discovery on V24. + - A 0.22.1 alone would not have fixed external users anyway: `dash@8.0.1` pins `@dashevo/dapi-client` exactly, so the whitelist fix needs a republish regardless. +- **CI stopgap (PR 0s), only if dashd:24 reaches dashmate's defaults before U1 ships.** It is its own PR with no dependency; PR 0 reverts it. + - Add a temporary `yarn patch` of 0.22.0 with #318 and the v2 rule. + - Set `DAPI_ADDRESSES` in `configure_test_suite.sh`. The platform-test-suite client factories already accept it (`PTS/lib/test/createClientWithoutWallet.js:9-17`), and it skips masternode-list discovery, so #319 is not needed. + - The wallet-lib and `dash` functional tests get their env from `configure_dotenv.sh` (`DAPI_SEED`), so they still run discovery. Give them explicit addresses there too, or skip them for the window. + - Skip the masternode-identity case until it is ported (it moves to `protx info`). +- **Effort.** + - v2 rule and fallback: about 40 lines plus vectors, applied twice (master and the 0.22 backport). + - #319: review and merge. + - The releases: a CI fix or manual publishes. + - #5609: lockfile and cache bump. + +**Porting withdrawals first does not save the suite**, because of item 1. The ported withdrawal test never parses Core transactions in JS (§5.3), so item 4 stops mattering for CI once the port lands. + +## 5. Options for the e2e suite + +### 5.1 The three options + +| | (A) Port to the new JS SDK | (B) Rust e2e (rs-sdk + platform-wallet) | (C) Hybrid | +|---|---|---|---| +| **What** | Port the about 75 portable cases into the **existing** wasm-sdk functional suite (one JS harness, §5.2); Node and Chrome | Port the cases to Rust against the local network. Base: the platform-wallet e2e framework on `feat/rs-platform-wallet-e2e` (draft PR #3549: +66,130 lines, 298 files, unmerged, testnet "bank wallet" oriented, not in CI) | A for SDK, browser and protocol flows; B for wallet, SPV and Core flows | +| **Effort (rough, engineer-weeks)** | 4-6. Asset-lock builder (PR 4) 1-1.5; harness additions 1; porting 2-3 | 5-8, plus landing #3549. Local-network funding and CI 1-2; porting 3-4; SPV sync time in CI | A plus 2-4 | +| **Protocol over a real network** | Yes | Yes | Yes | +| **JS bindings** | Yes (wasm-sdk API; evo-sdk's thin facade stays covered by its unit tests) | **No** | Yes | +| **Browser (wasm, gRPC-web)** | Yes. A wasm-only transport bug was fixed on 2026-10-27 (#5611), so this matters | **No** | Yes | +| **Core wallet / SPV** | No: Core work goes through Core RPC | Yes | Yes | +| **Main risks** | API gaps found mid-port; suite runtime grows (needs batching); SDK_TEST_DATA dependency | A very large unmerged PR; loses browser coverage; SPV in CI is slow and flaky | Two harnesses | + +### 5.2 Recommendation + +**Option A, ported into the existing wasm-sdk functional suite (one JS e2e harness).** At the end, `platform-test-suite` is deleted together with its CI job and image. + +Why one home: +- The wasm-sdk suite already has what a rewritten platform-test-suite would build: the local-network readiness hook, the seeded-identity signers, the Node and karma runners, and a CI job. +- Rewriting platform-test-suite on evo-sdk would leave two JS e2e suites on the same SDK. + +**What we lose:** platform-test-suite can run against devnets and testnet (`scripts/configure_test_suite_network.sh`); the wasm-sdk suite is local-only. Decision D4 in §11. + +**B keeps going on its own track** as the wallet-focused suite. It is not a prerequisite for anything here; once it runs in CI, the repo is at C. + +### 5.3 How the ported tests get funds + +**1. Platform bank, with no Core (most tests; Node and browser).** +- Each runner gets one SDK_TEST_DATA identity (`0x01`, `0x02`, `0x03`). Its key 3 is a CRITICAL TRANSFER key, which is what transfer-to-addresses requires (`identity_signed.rs:15-21`). +- At start, the runner funds one platform address it controls once (`addresses.transferFromIdentity`, `packages/js-evo-sdk/src/addresses/facade.ts:184`). +- Tests then create identities from that address as needed (`addresses.createIdentity`, `:264`). +- One shared SDK client per runner plus a mutex avoids nonce races. wasm-sdk's own tests already need `refreshIdentityNonce`. +- Minimum amounts: 100k credits per input, 500k per output. +- The genesis-seeded *platform addresses* (`.../create_genesis_state/test/addresses.rs:11-16`) have no known keys. They are not a funding source. +- No JS test exercises this transfer-then-create path yet. PR 5 adds one first. + +**2. Core faucet (Node; local network only).** For asset-lock, withdrawal and Core-endpoint tests. +- **Runners that share a network need separate faucet keys.** With a shared key, concurrent asset locks double-spend: `getaddressutxos` lists confirmed UTXOs only, and the local network mines every 60 s (`scripts/setup_local_network.sh:5`). + - Once PR 5 splits Node and browser into separate jobs, each job has its own network and its own `FAUCET_1`/`FAUCET_2` from `dashmate wallet mint`. + - Within one runner, keep a spent-outpoint set, or use `getaddressmempool`. + - Caveat until PR 9: `configure_dotenv.sh:35-36` takes the wallet-lib and `dash` functional tests' faucet from `logs/mint.log`, which every mint overwrites. Any new mint must write its own log, and PR 5 names which `FAUCET_n` each consumer uses. +- Core RPC calls: + - `getaddressutxos`; local_seed has `addressindex` (`setupLocalPresetTaskFactory.js:170`); + - `sendrawtransaction`; + - `getislocks` for InstantSend lock bytes; + - `getrawtransaction` verbose for ChainLock status. +- Credentials: `configure_test_suite.sh` exports local_seed's `core.rpc.port` and the `dashmate` user's password. That is the only user whose whitelist includes `getislocks`; the values are already readable as in `configure_dotenv.sh:65-68`. These go into the suite's env, which the harness PR adds. +- Chain asset-lock proofs also need Platform's `core_chain_locked_height` to reach the transaction's block, as rs-sdk waits for (`packages/rs-sdk/src/core/transaction.rs:402-420`). +- This also covers **instant** asset-lock proofs, which today's suite never does. + +**3. Browser specs that need Core** call the same faucet through a karma middleware: a small Node handler in the karma process, reached over HTTP. That keeps today's browser coverage of identity create and top-up: the asset lock is built in wasm and the identity is created over gRPC-web. + +**4. Withdrawals.** `identities.creditWithdrawal` to a faucet-generated Core address. Assert the document reaches COMPLETED, then check Core with `getaddressutxos` on that address. Verify in PR 6 that asset-unlock outputs are address-indexed. If not, fall back to `getrawtransaction` on the unlock txid. + +**5. Masternode identities.** Read ProRegTx keys with Core RPC `protx info`, instead of parsing the ProRegTx in JS. + +### 5.4 PR 4: an asset-lock builder (the only new SDK feature the port needs) + +**What and where** +- `buildAssetLockTransaction(inputs: {outpoint, txOut, privateKey}[], creditOutputs, changeAddress, feeRate)` returns the signed transaction bytes. +- The logic lives in a Rust crate that compiles to wasm (rs-sdk), on key-wallet's `TransactionBuilder`: + - `add_inputs(Utxo)`, `transaction_builder.rs:239`; + - `set_special_payload`, `:303`; + - the burn output is handled at `:342-394`; + - signing through a small adapter that maps raw keys, since `TransactionSigner` is keyed by derivation path (`:904-960`). Precedent: `packages/rs-platform-wallet/src/masternode/update_service.rs:610-622`. +- wasm-sdk and evo-sdk only marshal. + +**Why not reuse what exists** +- key-wallet's `asset_lock_builder` needs a `&Wallet` with synced accounts (`asset_lock_builder.rs:367-560`), so it does not fit caller UTXOs. +- An upstream key-wallet helper would add a rust-dashcore pin bump in front of PR 4. Prefer rs-sdk. + +**Value beyond tests** +- It un-skips wasm-sdk's `identityCreate`/`identityTopUp` tests. +- Apps that hand-roll this today get a supported path. dash-bridge serializes asset locks itself in `src/transaction/builder.ts`. + +**Not in PR 4: `waitForProof`.** rs-sdk's IS-lock wait uses `tokio::time::{sleep, timeout}` (`packages/rs-sdk/src/core/transaction.rs:23`), which is not wasm-safe. It hard-codes output index 0, and it has no callers. Making it a JS feature means: +- a wasm-safe rework: the transport `sleep` (`rs-dapi-client/src/transport.rs:34-44`) and a select-based timeout; +- an output-index parameter; +- a subscribe-before-broadcast API (DAPI streams don't replay locks); +- recovery by txid into a chain proof; +- an AbortSignal. + +That is **PR 4b, optional**, decided as a product question (D2). The tests don't need it: the faucet gets locks from Core RPC. + +**Not added to evo-sdk:** `core.broadcastTransaction`, `getTransaction`, `getBlockchainStatus`. The tests use Core RPC and a Node gRPC client (§5.5). + +### 5.5 DAPI Core endpoint tests + +- These test rs-dapi, not the SDK. Port them with `@grpc/grpc-js` and `@grpc/proto-loader` on `packages/dapi-grpc/protos/core/v0/core.proto`, Node-only: + - broadcastTransaction, getTransaction, getBlockchainStatus, getBestBlockHeight; + - **`subscribeToBlockHeadersWithChainLocks`**: grpc-js handles server streaming. Compare the streamed header bytes with Core RPC `getblockheader false`, so no X11 hashing is needed in JS. +- `getBlockHash` is JSON-RPC, so a plain `fetch`. +- This keeps the only live tests of rs-dapi's Core endpoints without adding evo-sdk API. rs-dapi-client has no `BlockHeadersWithChainLocks` transport (`packages/rs-dapi-client/src/transport/grpc.rs:918-941` covers only TransactionsWithProofs and MasternodeList). + +### 5.6 Deliberate drops + +Pure Drive rules already have in-process drive-abci tests: +- `InvalidDataContractVersionError` in `data_contract_update`; +- unique-index and unknown-type errors in `batch/tests`; +- reference rules in `owner_reference.rs` and `list_element_reference.rs`. + +**Rule:** keep at least one negative case per transition family, to prove error codes reach JS (`error.code`, `packages/wasm-sdk/src/error.rs:413-416`). Drop other duplicates only when a drive-abci test covers the same rule, and record that test in the PR's case table. + +Keep the asset-lock negative cases. For example, `InvalidInstantAssetLockProofSignatureError` (10513) has no drive-abci test, because in-process tests mock Core. + +## 6. Can we have an SPV wallet in JS? + +### 6.1 What compiles for `wasm32-unknown-unknown` + +Prototyped in an isolated crate in `/tmp`, with the Platform lockfile, the `getrandom` setup from `.cargo/config.toml`, and LLVM clang. The feasibility reviewer re-ran it. + +| Crate (rust-dashcore `aca0202`) | wasm32 | Why | +|---|---|---| +| `key-wallet` (bls, eddsa, serde) | **Compiles** | It is already linked into wasm-sdk through dpp (`cargo tree -p wasm-sdk --target wasm32-unknown-unknown -i key-wallet`) | +| `key-wallet-manager` (bls, eddsa) | **Compiles** | rust-dashcore#590 split it out of key-wallet to keep tokio out of key-wallet for WASM. It uses only tokio `sync`/`rt`/`macros` | +| `dash-spv` | **Fails** | tokio `full` → `net` → mio ("unsupported by mio"); tracing-appender's `symlink`; its P2P layer uses TCP | +| `platform-wallet` | **Fails** | Only because it depends on dash-spv unconditionally. 12 of 180 source files reference `dash_spv` (`spv/`, `dash_spv::EventHandler` as the event trait, `BroadcastResult`, `CheckpointManager`). It also uses `tokio::spawn` and `Instant::now` | + +### 6.2 What this means + +- **Keys, derivation, transaction building and wallet state already work in wasm.** +- **dash-spv cannot run in a browser.** There is no raw TCP, and its sync relies on BIP157/158 filters and QRInfo, which DAPI does not serve. + +### 6.3 A DAPI-stream sync backend: feasible, but a separate project + +1. **Make dash-spv optional in platform-wallet.** Own event/broadcast/checkpoint types in the non-SPV paths; spawn and clock shims for wasm. +2. **Add a `DapiSyncBackend`:** + - `subscribeToTransactionsWithProofs` with a bloom filter from key-wallet-manager's monitored scripts and outpoints, restarted when `monitor_revision` changes; + - `subscribeToBlockHeadersWithChainLocks` for headers and ChainLocks. rs-dapi-client has **no transport for this stream yet** (§5.5), so it is new work. +3. **Upstream, key-wallet-manager:** + - a filtered-block entry point, since `process_block_for_wallets` takes a full `Block` (`wallet_interface.rs:69`); + - reorg detection from the header stream, feeding the existing `truncate_above`. + + rust-dashcore#585 (open) helps. +4. **Lock verification.** dash-spv's `InstantLockValidator` and `ChainLockValidator` need a masternode-list engine fed by QRInfo, which DAPI does not serve. In wasm the realistic source is the trusted quorum-key provider wasm-sdk already uses. +5. **Persistence and bindings:** in-memory or IndexedDB callbacks, plus wasm-sdk and evo-sdk bindings. + +**Trust model:** the same as wallet-lib today, not real SPV. The bloom filter leaks addresses to the DAPI node, the node can withhold transactions, and locks are as good as the quorum-key source. + +**Effort:** about 8-12 engineer-weeks. + +**The test suite needs none of this.** + +## 7. Sequence of PRs + +"→" means "must merge after". Every PR updates the docs and references it makes stale (§3.7), writes CI filter aliases out as literal paths when it deletes a key (§3.5), and commits the lockfiles: `yarn.lock`, `.pnp.cjs`, `.pnp.loader.mjs`, `.yarn/cache`, `Cargo.lock`. + +**Phase 0: unblock (now, in parallel)** +- **U1 (dashcore-lib, upstream).** #319; the v2 asset-unlock rule with vectors; the unknown-type fallback; a release from master; the 0.22.x backport for Insight (§4). +- **PR 0s (no dependency; only if dashd:24 lands first).** The CI stopgap (§4). +- **PR 0 (→ U1).** #5609 on v5.1-dev; backport to v5.0-dev. Reverts PR 0s if it landed. + +**Phase 1: what nothing needs, and dashmate (independent of each other)** +- **PR 1. Delete `packages/dapi` and `packages/bench-suite`.** + - Workspace entries; root `test:*` scripts. + - Filter keys and aliases. + - `tests-js-package.yml:71`, which copies `packages/dapi/.env.example`. + - The cache-key hashes of that file (`local-network/action.yaml:37`, `tests-dashmate.yml:94`) and the e2e path filter that lists it (`tests.yml:137`). + - The `configure_dotenv.sh` dapi lines and the `configure_test_suite.sh:68-77` bench lines, in this PR: the scripts run under `set -e`, and the cache key usually hides the break until some unrelated PR. + - Badges; `test-suite-triggers.yml` (unused). +- **PR 2a. dashmate: drop wallet-lib, dapi-client and withdrawals-contract** (§3.3). Targets **v5.0-dev** and merges forward: it fixes a real operator bug, and a stable dashmate without these dependencies gates `npm deprecate` (§8.2). + - Delete `generateHDPrivateKeys` (file, registration, parameter). + - New probe: TLS always (preferably verifying against the external IP, §3.3), timeout, last error reported, skipped when Platform is disabled. + - Remove `Dockerfile:851-867`. + - dashmate's filter entry becomes `packages/dashmate/**`. + - Delete `Cargo.toml.template`. + - Unit tests for the probe: TLS options, timeout, the disabled-Platform skip. +- **PR 2b. dashmate: drop dashcore-lib** (§3.3). Off the critical path; it serves the security goal. + - The key/address helper, with vectors generated by dashcore-lib. + - The jayson JSON-RPC funding helper. + - Keep the log strings `configure_test_suite.sh` parses: "ProRegTx transaction ID:", "Owner Private Key:", "Address:", "Private key:". + - Coverage comes from the local-network action (`setup_local_network.sh`, `wallet mint`). `test/e2e/localNetwork.spec.js:69-72` skips setup on a cache hit, so bust the cache once. +- **PR 2c (→ 2a).** The dashmate-helper image builds from a JS-only stage instead of `build-js`, and the stale Node 26/cbor comment (`Dockerfile:833-835`) goes. About 20 minutes saved per architecture per release. +- **PR S (→ 2a released in a stable dashmate, and the final legacy release, D3). Stop publishing.** On **both** v5.0-dev and v5.1-dev: + - mark `dash`, wallet-lib, dapi-client, wasm-dpp, dash-spv, grpc-common, dapi-grpc's npm package and the ten contract packages `private`; + - nothing else changes, so the packages stay in the workspace and the old test suite keeps running until the port is done. + + v5.1 prereleases publish under `X.Y-suffix` tags, so v5.1-dev must stop too. This decouples deprecation (§8.2) from the port. +- **PR 3. System-contract schema tests move to Rust** (§3.4). + - Remove the JS tests, wasm-dpp devDependencies and the mocha/lint scaffolding from the ten packages. + - Remove their filter keys, after writing out the aliases that use them (`@dashevo/wasm-dpp`, dashmate). + - Add `data-contracts` to nextest. + +**Phase 2: the port (option A)** +- **PR 4. Asset-lock builder** in rs-sdk, with wasm-sdk and evo-sdk bindings (§5.4). + - Rust unit tests with fixed vectors. + - A wasm-sdk unit test. +- **PR 4b (optional, D2).** wasm-safe `waitForProof`. +- **PR 5 (→ PR 4). Harness additions in wasm-sdk's functional suite:** + - env plumbing: `CORE_RPC_*` and per-runner faucet keys from `configure_test_suite.sh`; + - the Platform bank, plus a functional test of transfer-then-create; + - the Core RPC faucet and its karma middleware; + - the grpc-js Core client; + - Node-only gating for Core specs; + - un-skip `identityCreate`/`identityTopUp`; + - CI: split `test-functional` into Node and browser jobs (batches if runtime needs them); + - filters: wasm-sdk and evo-sdk also on rs-dpp, rs-drive, rs-sdk, wasm-dpp2, the context providers and dash-platform-queries. +- **PR 6 (→ PR 5).** Port withdrawals and Identity (they carry the Core work), and delete those platform-test-suite specs. +- **PR 7 (→ PR 5).** Port DataContract, Document, IndexOnlyDocument, contacts and dpns, applying §5.6, and delete those platform-test-suite specs. +- **PR 8 (→ PR 6 and PR 7).** Port the read-only and Core-endpoint specs (§5.5). Then **delete `platform-test-suite`**: + - the package; + - the `test-suite` job (`tests.yml:636-671`) and `tests-test-suite.yml`; + - the Dockerfile `test-suite` stage and the `release-test-suite-image` job (`release.yml:262-272`); + - the root `test:suite*` scripts; + - the platform-test-suite part of `configure_test_suite.sh`, keeping faucet minting and env for the new suite; + - `configure_test_suite_network.sh`, unless D4 keeps remote runs. + +**Phase 3: delete the stack (→ PR 2a, 2b, 2c, PR 3, PR 8, PR S)** +- **PR 10 (any time before or with PR 9).** The two book chapters on wasm bindings move their examples to wasm-dpp2 (`packages/wasm-dpp2/CONVENTIONS.md`). +- **PR 9. Remove `js-dash-sdk`, `wallet-lib`, `js-dapi-client`, `dash-spv`, `js-grpc-common`, `wasm-dpp` and dapi-grpc's npm `package.json`.** + - Keep the dapi-grpc crate, `protos/`, and both `clients/{core,platform}/v0/rust` example directories. + - Delete the codegen: `scripts/build.sh`, `codegen.json`, `setup-codegen.py`, `test-client-codegen.yml`, and the non-Rust `clients/` trees. + - **Dockerfile:** + - delete the `build-js` stage (`:643-752`), which has no consumers after PR 2c and PR 8; + - drop `packages/wasm-dpp` from the remaining `COPY --parents` lists (`:418,547,979`). + - **CI:** + - the `tests-packges-functional.yml` wallet-lib and `dash` steps; + - the legacy filter keys, with wasm-dpp2's paths written out; + - `tests-build-js.yml:22-31` and the codegen lines in `npm-release-build/action.yaml:16` and `npm-runner-validation.yml:66-69`; + - the `book.yml` TypeDoc job and `docs/typedoc.json`, in the same PR, or the Pages deploy fails. + - **Config and scripts:** + - `yarn.config.cjs` offsets; + - `.yarnrc.yml` audit excludes still needed by others, checked with `yarn why`; + - `configure_dotenv.sh`, `prepare_docs.sh`, `.codecov.yml`, `pr.yml` scopes, `.pre-commit-config.yaml`; + - the Swift sentinel; + - `AGENTS.md` and the READMEs. + +**Follow-ups, not on the critical path** +- Remove rs-dpp `extended-document`. +- Change `client_codegen` in the runner manifests (runner-image promotion). +- Delete the unused `js-packages.yml` and `rs-packages.yml`, and the dead prune step at `tests.yml:380-407`. +- A dashd-rpc wrapper over jayson, extending PR 2b's helper. +- bls via `@noble/curves`. +- The bn.js root resolution (§8.3). + +**Phase 4: npm deprecation (§8.2).** Ivan's explicit go is required; it is outward-facing. + +## 8. External users and deprecation + +### 8.1 Who uses these packages outside this repo + +Sources: +- `npm view`; +- `api.npmjs.org` (point, range and per-version); +- ecosyste.ms and deps.dev; +- read-only `gh api search/code`, on 2026-10-28. + +npm's dependents tab sits behind a Cloudflare challenge. + +**Download numbers.** Packages created a month ago with zero dependents get 360-410 downloads a week; that is this family's noise floor. evo-sdk already has about twice the monthly downloads of `dash` (9,831 against 4,654). + +| Package | External users found | +|---|---| +| `dash` | A few active third parties, e.g. hilawe/dash-tegara-mn-pooling (`^7`; HD top-ups, SML parsing, deep import of `dash/build/.../broadcastStateTransition`), hilawe/dash-mno-verify (`^4`; documents and derived keys; evo-sdk covers it today), EmoBanana/Scatterly, densmirnov/dash-singles. Many ecosyste.ms "dependents" are installs of the unrelated 2011 `dash` package | +| wallet-lib, dapi-client | Mostly DCG repos (below), plus PastaPastaPasta/dapi-explorer | +| wasm-dpp, dash-spv, grpc-common, dapi-grpc, contract packages | None outside lockfiles that come through `dash` | +| dashcore-lib | **Broad and non-Platform:** xchainjs (133★, `^0.25.0`), fireblocks/recovery (`^0.21.0`), anypay, crowdnode.js, Insight (`^0.22.0`), altcoin forks; about 7.9k downloads a week | + +**DCG and dashpay repos on legacy packages** (to migrate before `npm deprecate`; owned outside this repo): + +| Repo | Uses | Notes | +|---|---|---| +| dash-network-deploy | Smoke tests (`test/smoke/dapi.js`) | Per-node probes work with evo-sdk's `addresses: [one]`. `getBlockByHeight`/`getBlockByHash` have no evo-sdk equivalent; Core RPC or the grpc-js client works instead | +| dash-bridge | Builds asset locks **itself** (`src/transaction/builder.ts`) and gets UTXOs from Insight. Uses dapi-client for IS-lock subscription, SML and status; dashcore-lib for parsing and message signing | PR 4 and 4b replace its builder and IS-lock wait. The SML needs stay | +| dash-platform-website (private) | `dash 8.0.0-rc.5`, with deep imports into `dash/build/...` | Its comments document more dash 8.x wallet breakage: dark-gravity-wave rejecting valid blocks, header sync from block 1 | +| dash-3d-nft-marketplace | Presented as the js-sdk example app | — | +| yappr | `register-contract.js` | — | + +**Docs.** docs-platform moved its tutorials to evo-sdk in March 2026 (#133, #134, #136, #152). Still legacy: +- the JavaScript tabs of its two DAPI endpoint reference pages; +- `tutorials/building-platform.md` "Use JS SDK"; +- the versioned 2.0.0 js-dash-sdk pages. + +**dashmate blocks a clean deprecation.** dashmate 5.0.1 depends at runtime on wallet-lib, dapi-client, withdrawals-contract and dashcore-lib. Deprecating them now would warn every dashmate installer. + +### 8.2 Path and minimal timeline + +| When | Action | +|---|---| +| **Weeks 0-2** | U1 and PR 0 (§4). README banners on every legacy package, with **absolute** URLs: the js-dash-sdk banner's relative link resolves against the stale `dashevo/DashJS` repository field on npmjs.com. Add the missing `license` and `repository` fields to the **kept** packages (evo-sdk, wasm-sdk and wasm-dpp2 show no license on npm today). A "Deprecations" section in release notes, a pinned GitHub Discussion, and a banner on the docs-platform legacy pages and on the Read the Docs 2.0.0 version | +| **Final legacy release (D3)** | The last stable release that contains the packages, with PR 0. Its packages are the "final version". Known issues go in the READMEs, not fixes. Acceptance smoke test on testnet: default discovery, `getBestBlockHeight`, identity fetch, document query | +| **Weeks 2-6** | A migration guide in docs-platform, linked from the evo-sdk README, the legacy READMEs and the deprecation message. It maps `Dash.Client` calls to evo-sdk, gives a recipe for funding a one-time address, and has explicit "no equivalent" rows: HD Core wallet, balance and history, sending DASH, the masternode list, raw DAPI gRPC. Funding from Core gets a row of its own: "evo-sdk 5.1 (PR 4); until then, build the asset lock yourself as dash-bridge does". Then migrate the DCG repos, starting with dash-bridge | +| **Once a stable (`latest`) dashmate without the legacy dependencies is out (PR 2a), and PR S has merged on every release branch** | Announce "npm deprecation in 30 days" | +| **30 days later** | `npm deprecate @"*" ""` for `dash`, wallet-lib, dapi-client, wasm-dpp, dash-spv, grpc-common, dapi-grpc, the ten contract packages, `@dashevo/dark-gravity-wave` and `@dashevo/dash-util`. Run it with `--dry-run` first. It needs a human with 2FA; OIDC covers only `publish` | + +**npm mechanics** +- `"*"` includes prereleases. Only versions that already exist are marked; that is why the packages must stop publishing first. +- Undo with an empty message. +- Leave the old dist-tags alone. +- **Never unpublish.** + +**No major version bump for the final version:** a major would not reach `^8` or `~` ranges. **No runtime warnings in wallet-lib or dapi-client:** dashmate 5.0.x loads them on every command. One `process.emitWarning(..., 'DeprecationWarning')` in `dash`'s Client constructor is optional. + +**Total: about 2-3 months, bounded by the dashmate release, the final legacy release and the 30-day notice. Not bounded by the port:** PR S stops publishing while the old suite still runs. + +### 8.3 Security side findings + +- **elliptic:** GHSA-848j-6mx2-7j84 affects every version up to 6.6.1 and has no patched release. dashcore-lib depends on `^6.5.4`; dropping dashcore-lib is the fix. +- **bn.js:** the `.yarnrc.yml` exclude blames dashcore-lib, but dashcore-lib's `^4.12.0` already allows the fixed 4.12.3. Our root resolution `"bn.js": "4.12.0"` pins the vulnerable version. A small independent fix. + +## 9. What we keep + +- **Packages:** js-evo-sdk, wasm-sdk (its functional suite becomes the JS e2e suite), wasm-dpp2, wasm-drive-verify, dashmate, the contract crates, dapi-grpc as a Rust crate with protos. +- **CI:** the local-network CI with SDK_TEST_DATA, running the e2e suite in Node and Chrome. +- **dashmate's** `@dashevo/bls`, `@dashevo/dashd-rpc` (PR 2b adds a jayson helper next to it) and `@dashevo/docker-compose`. + +## 10. Risks + +| Risk | Effect | Mitigation | +|---|---|---| +| dashd:24 reaches dashmate before U1 ships | The legacy suite fails at bootstrap | The §4 stopgap (patch, `DAPI_ADDRESSES`, skip one case); or finish Phase 2 first | +| Insight breaks on v2 unlocks once PV15 emits them on V24 | dash-bridge, xchainjs and explorers lose their UTXO source | Flag in the asset-unlock-v2 PR; an upstream dashcore-lib release before PV15 activates with V24 | +| API gaps found mid-port | The port stalls | Each gap becomes a small wasm-sdk PR; legacy specs keep running until they are replaced | +| The suite's runtime grows (asset-lock and withdrawal cases wait for blocks) | Slow CI | Node and browser split, plus batches (PR 5) | +| SDK_TEST_DATA is a build-time flag (`Dockerfile:21-24`, `scripts/setup_local_network.sh:20-34`) | No bank on stock images | CI and `setup_local_network.sh` already build with it | +| Nonce or UTXO races between runners | Flaky tests | One seeded identity and one faucet key per runner; a client and mutex per runner; a spent-outpoint set | +| Asset-unlock outputs are not address-indexed | The ported withdrawal Core check fails | Verify in PR 6; fall back to `getrawtransaction` | +| A filter-alias break when a key is deleted | The `changes` job fails and Tests is skipped | Write the aliases out in every removal PR | +| A hidden `configure_*.sh` break behind the local-network cache | A later unrelated PR breaks | Change the scripts in the same PR as the deletion; bust the cache once | +| Legacy users stranded at PV15 activation if D3 = v5.0.x | `dash` 8.0.x likely fails on PV15 networks (not verified) | State it in the README and the deprecation message. D3 alternative: final version = v5.1.x stable | + +## 11. Decisions + +### 11.1 Decided by this spec (reasoning in the sections cited) + +- Delete JS DAPI and bench-suite now (§3.1). +- Contract tests go to Rust, targeting the latest schema versions (§3.4). +- One JS e2e harness: wasm-sdk's functional suite (§5.2). +- The header-stream test is ported with grpc-js; it is not exposed in evo-sdk (§5.5). +- Drop dapi-grpc's npm package and multi-language codegen, together with grpc-common (§3.1, PR 9). +- The contract packages stop publishing to npm in the same PR as `dash` (PR S). +- Stop releasing the test-suite image (§3.5; its only external user has been dormant since 2024). +- Do not deprecate dashcore-lib (§8.1). +- No 0.22.1 from Platform's side; align with #5609 (§4). + +### 11.2 For Ivan + +- **D1. dashcore-lib's last fix.** Do U1 and PR 0 (one upstream release from master, and Platform bumps to it), or declare the legacy packages broken on V24 and keep CI on dashd:23 until the port lands? Recommended: do it. It is small next to the port, the final legacy version needs working discovery, and Insight needs the 0.22 backport regardless, once PV15 emits v2 unlocks on V24 networks. +- **D2. `waitForProof` in evo-sdk (PR 4b)?** It is not needed for the tests. It is the piece dash-bridge-like apps need to drop dapi-client. Recommended: later, as a product feature, after PR 4. +- **D3. Which release is the final legacy version?** + - (a) The last v5.0.x stable, then delete on v5.1-dev. Fastest. Legacy users likely break when networks activate PV15. + - (b) The first v5.1.x stable. PR S on v5.1-dev and PR 9 wait until v5.1.0 is cut, and legacy must keep building through the v5.1 cycle. + + Recommended: (a), stated clearly in the deprecation message. +- **D4. Remote-network e2e runs** (testnet or devnets with an env-provided bank). platform-test-suite supports them; the wasm-sdk suite is local-only. Recommended: drop them now and add an env-provided bank later only if someone needs it. + +## 12. Verification plan + +- **PR 1:** CI green; a cache-busted local-network run. +- **PR 2a:** unit tests for the probe (TLS options, timeout, disabled-Platform skip). `tests-dashmate.yml` green. A manual `dashmate start -w` on a self-signed local node and on a TLS-provider node. +- **PR 2b:** helper vectors per network, generated with dashcore-lib before removal. A cache-busted local-network run (setup plus `wallet mint`). +- **PR 2c:** the dashmate-helper image builds and its container starts; `tests-dashmate.yml` green. +- **PR S:** a release dry run (`release.yml` manual dispatch) packs no legacy package on either branch; `yarn constraints` passes. +- **PR 3:** the PR description maps each latest-schema rule to its Rust test. +- **PR 4:** fixed-vector Rust tests (a known transaction from known inputs). A functional identity create from a built asset lock. +- **PRs 5-8:** + - Each ported spec is green in Node and Chrome. + - Show red → green once for consensus-code assertions, by breaking the input deliberately. + - Each PR description has a case table: old case → new case, or "dropped: covered by ``", or "dropped: legacy-only". +- **PR 9:** + - `yarn install` and `yarn build` pass with no legacy workspace. + - `yarn why @dashevo/dashcore-lib` is empty. + - A release dry run (`release.yml` manual dispatch) packs only the kept packages. + - The Drive, rs-dapi and dashmate-helper images build. + - `book.yml` passes, including the rewritten book chapters (PR 10). +- **Deprecation:** `npm view '@*' deprecated` shows the message for stable versions and prereleases. + +## 13. Alternatives rejected + +- **Port the suite to Rust (B) as the replacement:** it loses browser and JS-binding coverage, and it depends on a 66k-line unmerged PR. +- **Rewrite platform-test-suite on evo-sdk next to wasm-sdk's functional suite:** two harnesses on one SDK. +- **Contract tests on wasm-dpp2:** it would need new validation bindings just for tests. +- **A broad evo-sdk Core facade (broadcast, getTransaction, status):** the tests have Core RPC and grpc-js; no product need was shown beyond asset locks. +- **Platform staying on a dashcore-lib 0.22.1:** it wouldn't fix discovery without a republish anyway (`dash` pins dapi-client exactly), and #5609 already targets the master line. The 0.22 backport in U1 exists for Insight, not for Platform. +- **Hand-written asset-lock serialization in the test harness:** that duplicates Rust logic; PR 4 also serves users. +- **Readiness via rs-dapi JSON-RPC `getStatus`:** it reports success when Drive is down. + +## 14. Review log (revision 1 → 2) + +- **Feasibility.** + - key-wallet's `asset_lock_builder` can't take caller UTXOs: PR 4 now uses `TransactionBuilder`. + - rs-sdk's IS-lock wait isn't wasm-safe: split out into optional PR 4b. + - The proof-verifier adapter was deleted too early: moot, since platform-test-suite is now deleted whole in PR 8. + - Dockerfile stages are edited, not deleted. + - No header-stream transport exists: §5.5, §6.3. + - Added: faucet UTXO reuse, the chain-proof height wait, Core RPC env plumbing, a test of the bank path, karma-middleware browser coverage, and the lock-verification and reorg gaps in §6.3. The §6.3 estimate is raised. +- **Scope.** + - dash-bridge already builds asset locks itself, so the "no JS path" claim is removed. + - One JS e2e harness instead of two. + - PR 4 shrunk to one builder; no evo-sdk Core facade. + - Deliberate-drop rule (§5.6). + - PR 2 split. + - Contract tests target the latest schemas. + - Open questions cut from 8 to 4. +- **CI and release.** + - Exact Dockerfile lines. + - Codegen touchpoints, including `tests-build-js.yml` and the runner manifests. + - grpc-common is tied to dapi-grpc. + - Filter-alias rewrites in every PR. + - Script edits move into the deleting PR. + - TypeDoc in PR 9. + - Lockfiles committed. + - `test:suite` resolution. + - Final version gated on a stable release (D3). + - Only "PR Hygiene" is a required check. +- **dashmate.** + - dashd-rpc can't make the funding calls: jayson helper. + - Probe target and TLS pinned; the infinite loop confirmed by reading the code on default providers. + - Probe skipped for Core-only nodes; timeout policy. + - Exact COPY lines; `pack_dashmate.sh` needs no change. + - Helper image off `build-js` (PR 2c). + - Helper pitfalls (version bytes, compressed keys, P2SH). + - Log strings kept; cache-hit coverage gap. +- **External users.** + - 0.22.1 mechanics: tag and workflow, and why it is dropped. + - Final version from a stable line, and the v5.0-dev stop-publishing PR. + - Missed: Insight, dark-gravity-wave, dash-util. + - npm mechanics; migration guide; absolute banner links; metadata for kept packages as a gate; timeline. + +## 15. Consistency check (revision 2 → 2.1) + +- Keep both dapi-grpc Rust example directories. +- PR 9 deletes the codegen files and `test-client-codegen.yml`, and the whole `build-js` stage. +- Corrected Dockerfile lines: `+dash` is at `:747`, the comment at `:833-835`. +- Phase 3 now also waits for 2b and 2c. +- The stopgap is its own PR (0s), and it covers the wallet-lib and `dash` functional tests' env. +- Faucet keys: per network, and the `mint.log` overwrite caveat. +- U1 includes the 0.22 backport for Insight; the D1 reasoning was fixed to match. +- New PR S stops publishing on both release branches, so deprecation no longer waits for the port; the timeline was corrected. +- PR 10 moved before or into PR 9. +- PR 2c needs only 2a. +- Verification added for 2c, S and 10. +- Minor fixes: grpc-common users, `configure_dotenv.sh` lines, `tests.yml:137`. diff --git a/docs/LEGACY_JS_RETIREMENT_WALKTHROUGH.md b/docs/LEGACY_JS_RETIREMENT_WALKTHROUGH.md new file mode 100644 index 00000000000..09e929c6196 --- /dev/null +++ b/docs/LEGACY_JS_RETIREMENT_WALKTHROUGH.md @@ -0,0 +1,158 @@ +# Retiring legacy JS: the plan in plain words + +This is the short version of `docs/LEGACY_JS_RETIREMENT_SPEC.md` (revision 2.1). The spec has the evidence and details. Five independent reviewers checked it: feasibility, scope, CI and release, dashmate, and external users. Their corrections are folded in; the spec's last section lists them. Neither file is meant to be committed. + +> **Superseded in part (2026-10-29)** by `docs/E2E_TEST_ARCHITECTURE_WALKTHROUGH.md`. These parts no longer hold: +> - **"The plan for the e2e suite",** including "How the ported tests get money without wallet-lib": the old suite's protocol cases move to one Rust e2e suite, not into the new JS SDK's suite. The JS SDK no longer needs an asset-lock builder for tests. +> - **The last bullet of finding 4.** The JS SDK's suite stays as the JS smoke and gets no new protocol cases. +> - **Step 5 of "Order of work".** The port now goes to the Rust suite. +> - **In "Where this differs":** the bullets on not rewriting the old suite, on the asset-lock builder, and on the Rust work continuing separately. The Rust work is now the main track. +> - **Decision 4** (runs against testnet or a devnet). It is replaced by the new walkthrough's question about remote runs. +> +> Everything else here still holds. + +## What we are removing + +- the old SDK (`dash`); +- wallet-lib; +- dapi-client; +- wasm-dpp; +- the JS header-chain library (`dash-spv`); +- grpc-common; +- the npm package of dapi-grpc, with its generated clients for JavaScript, Java, Objective-C and Python. The Rust crate and the proto files stay; +- the old JS DAPI server; +- the bench suite. + +All of it sits on dashcore-lib 0.22. The replacement, evo-sdk on top of wasm-sdk and wasm-dpp2, already exists. + +## What we found + +1. **The old JS DAPI server is already dead.** Dashmate stopped running it at version 3.0, and its Docker image hasn't been built for a year. CI still tests it on most rs-dpp pull requests. It can go today, with the bench suite, which no CI job runs. + +2. **Dashmate barely uses the legacy stack, and the one thing it uses has a bug.** + - Its wallet-lib code is never called. + - dapi-client is used for one "is the node ready" check. That check always talks plain, unencrypted gRPC to the node's gateway. On nodes with real certificates, which is the default for testnet and mainnet evonodes, the gateway only speaks TLS, so `dashmate start -w` would wait forever. Two reviewers found this by reading the code; nobody has run it. + - dashcore-lib makes a few random keys and builds two funding transactions during local setup. Dashmate's Core RPC client can't send the needed calls correctly, so we add a tiny JSON-RPC helper instead. + +3. **The contract packages' JS tests check old schemas.** There are 228 of them, and they are the only reason those packages need wasm-dpp. They validate documents against version 1 schemas, while the network uses newer ones for DPNS, DashPay, withdrawals and token history. CI doesn't run them when the Rust code they test changes. That is how 14 of them sat broken until a fix on Oct 28. They call a Rust function that Rust tests can call directly. So we move them to Rust and test the current schemas. + +4. **The e2e test suite is the real work.** + - Every write it makes goes through the old SDK, and every identity it funds goes through wallet-lib and dashcore-lib. + - About 75 cases test real protocol behaviour and are worth keeping. About a dozen test the old libraries themselves and can be dropped. A few more repeat rules that Drive's own Rust tests already check, and may be dropped case by case. + - The new SDK already has its own end-to-end suite running in CI. It covers a good part of the same ground in Node and in the browser, using test identities our CI network creates at startup. + +5. **There is a deadline: Dash Core 24.** dashcore-lib 0.22 can't read Core 24's new coinbase format. + - That already broke the old JS client's node discovery on testnet on Oct 23. + - The whole JS e2e suite breaks the day our local test network moves to Core 24. That comes before Platform's new withdrawal format (v2 asset unlocks) even matters. + - Fixing the withdrawal format properly needs two changes, not one: accept the new version number, and compute the new transaction ID. + - **Outside our repo:** the block explorer Insight uses the same library. It will choke on the new withdrawal format too, once that goes live with Core 24. Whoever releases Insight has to know. + +## The plan for the e2e suite + +I compared three options: + +- **A. Port the tests to the new JS SDK.** Keeps browser coverage and tests the JavaScript API people install. A browser-only transport bug was fixed on Oct 27 (#5611), which shows why that matters. Roughly 4 to 6 engineer-weeks. +- **B. Rewrite them in Rust.** Loses browser and JS coverage. It would build on an unmerged draft of about 66,000 lines (the platform-wallet e2e framework). Roughly 5 to 8 weeks plus landing that draft. +- **C. Both.** Best coverage, two harnesses. + +**I recommend A, with one twist.** Instead of rewriting the old suite package, we move the cases into the new SDK's existing end-to-end suite and then delete the old package. One JS test harness, not two. The Rust work (B) can continue on its own as the wallet-focused suite; nothing here waits for it. + +### How the ported tests get money without wallet-lib + +- **Most tests don't touch Core.** + - Our CI network starts with three test identities that hold plenty of credits and whose keys we know. + - Each test runner moves some credits from one of them to a Platform address it controls, then creates new identities from that address as needed. + - All of this goes through the new SDK, in Node and in the browser. +- **Tests that must touch Core use a Node helper** that talks to the local Core node's RPC. These are asset-lock errors, top-ups from Core, withdrawals and Core endpoints. The helper finds coins, broadcasts, and fetches InstantSend locks. + - Browser tests reach the same helper through the test runner. + - Each runner gets its own funding key, so they never fight over the same coins. + - As a bonus, we'll finally test InstantSend asset-lock proofs. Today's suite never does, because wallet-lib ignores InstantSend locks. +- **The SDK gets exactly one new feature: "build an asset-lock transaction from these coins and keys".** + - It reuses existing Rust code (key-wallet's transaction builder); the bindings only pass data through. + - It also un-skips the new SDK's own "create identity" tests. + - "Wait for the lock" is not part of it. The Rust version of that only works natively today, and the tests get locks from Core directly. That stays an optional later feature. +- **Tests of DAPI's Core endpoints**, including the block-header stream, use a plain Node gRPC client on our proto files. No SDK changes are needed for those. + +## Can we have an SPV wallet in JS? + +I compiled the Rust crates for wasm, and a reviewer re-ran the check: + +- **The wallet parts work.** key-wallet (keys, derivation, transaction building) and key-wallet-manager (wallet state) compile. key-wallet already ships inside wasm-sdk. +- **The sync part does not.** dash-spv needs raw TCP sockets, which browsers don't have. platform-wallet fails only because it always pulls dash-spv in. +- **A DAPI-based sync is possible, but a project of its own** (about 8 to 12 weeks). platform-wallet would get a second sync engine that follows DAPI's streams, the way wallet-lib syncs today. That needs: + - making dash-spv optional; + - a new client for the block-header stream; + - one upstream change so the wallet can take filtered blocks; + - a way to check InstantSend and ChainLock signatures without dash-spv's masternode-list engine; + - browser storage and bindings. +- **What it can't fix.** Its trust is the same as wallet-lib's: the DAPI node learns your addresses and could hide transactions. +- **The test suite needs none of this.** + +## Order of work + +1. **Now:** + - one last dashcore-lib release that understands Core 24 and the new withdrawal format; + - Platform switches to it. This draft pull request already exists, and it also refreshes the stale list of testnet nodes; + - a temporary patch only if Core 24 reaches our CI first. +2. **Delete what nothing needs:** the JS DAPI server and the bench suite. +3. **Dashmate, in three small steps:** + 1. drop wallet-lib and dapi-client and fix the readiness check. This one goes to the current release line too, so operators get the fix; + 2. drop dashcore-lib; + 3. stop building dashmate's Docker image on top of the heavy wasm build. That saves about 20 minutes per release per architecture. +4. **Contract schema tests move to Rust.** +5. **The new SDK gets the asset-lock builder.** The test cases then move into its end-to-end suite in a few batches, starting with withdrawals and identities. When the last batch lands, the old test-suite package, its CI job and its Docker image are deleted. +6. **Delete the legacy packages,** with their CI jobs, Docker steps and docs. +7. **Rewrite two book chapters** whose examples use wasm-dpp code. +8. **Stop publishing, then deprecate on npm.** Once a stable dashmate no longer needs them and the final legacy release is out, we stop publishing the legacy packages from every branch, while the old test suite keeps running until the port is done. Then deprecate on npm, only with your go-ahead. + +## Who uses these packages outside this repo + +- **`dash` (the old SDK):** a handful of active third-party projects, plus many accidental installs of an unrelated old package with the same name. evo-sdk already has twice its monthly downloads. +- **wallet-lib and dapi-client:** mostly our own organisation's repos. + - dash-network-deploy (smoke tests); + - dash-bridge. It already builds its asset-lock transactions itself and uses dapi-client to wait for InstantSend locks and read the masternode list; + - the private website repo, which reaches into the old SDK's internal files. +- **wasm-dpp, dash-spv, grpc-common, dapi-grpc and the contract packages:** no real outside users found. +- **dashcore-lib:** widely used outside Platform, for example by xchainjs, Fireblocks' recovery tool and Insight. So we stop depending on it, but we don't deprecate it. +- **Docs:** the developer-docs tutorials already moved to evo-sdk in March. A few reference pages still show dapi-client examples. + +**One catch: dashmate itself still lists wallet-lib, dapi-client and dashcore-lib.** If we deprecated them on npm today, every dashmate install would print warnings. So a stable dashmate without them has to ship first. + +## What external users get, and when + +- **In the next two weeks:** + - notices in the legacy READMEs, with full links (the current one is broken on npmjs.com); + - a "Deprecations" section in the release notes; + - a pinned GitHub Discussion; + - banners on the old docs pages. + - Also, add the missing license and repository fields to evo-sdk, wasm-sdk and wasm-dpp2. Today npm shows no license for the package we're sending people to. +- **A final version.** The last stable release that still contains these packages, with the Core 24 fixes. We don't fix anything else in it; known problems are listed in the READMEs. +- **A migration guide** in the developer docs, from the old SDK to evo-sdk. It includes honest "no equivalent yet" rows: sending DASH from an HD wallet, balances, the masternode list. +- **`npm deprecate`, with 30 days' notice,** once two things are true: + - a stable dashmate without the legacy packages is out; + - the legacy packages no longer publish from any branch. + + It marks only existing versions, which is why publishing must stop first. Nothing is ever unpublished. +- **Total: roughly two to three months.** This doesn't wait for the test port, because we stop publishing before the old packages leave the repo. + +**Security note.** One of dashcore-lib's crypto dependencies (elliptic) has a known issue with no fixed version, so the only fix is to stop using it, which this plan does. Another warning (bn.js) is caused by a version pin in our own root `package.json`, not by dashcore-lib. That one is a small separate fix. + +## Where this differs from your original idea + +- **We don't rewrite the old test-suite package.** The cases move into the new SDK's existing end-to-end suite, and the old package is deleted. That avoids two parallel JS harnesses. What we lose: the old package could also run against testnet or a devnet; the new home is local-only for now (decision 4 below). +- **The new SDK gets one small feature (the asset-lock builder),** because the tests need to create identities from Core funds. It is not the broader Core feature set I first proposed; reviewers showed the tests don't need that. +- **dashcore-lib gets one more upstream release, despite the retirement.** The final legacy versions need it to work on Core 24, and our CI needs it until the port is done. Platform takes it from the library's main line, matching the draft PR that already exists. A separate backport to the old 0.22 line is for Insight, not for us. +- **More is removed than just the JS packages.** Also the dapi-grpc npm package with its generated clients for other languages, the test-suite Docker image (its only outside user has been inactive since 2024), and the contract packages' npm releases. Their Rust crates stay. +- **The contract tests are rewritten in Rust against today's schemas,** not ported case by case, and not moved to wasm-dpp2. wasm-dpp2 can't validate a document against its schema. +- **The Rust e2e work (option B) is not dropped.** It continues on its own track. + +## Decisions I need from you + +1. **Fix dashcore-lib one last time?** That means one upstream release, which Platform then switches to. The alternative is to declare the old packages broken on Core 24 and keep our CI on Core 23 until the port is done. I recommend the fix: it's small next to the port. Insight needs a similar patch on the library's older 0.22 line anyway, so I'd ask the library's maintainers for that backport in the same go. +2. **Add "wait for the InstantSend lock" to evo-sdk?** The tests don't need it. It is what apps like dash-bridge would need to stop using dapi-client. I recommend doing it later as a product feature. +3. **Which release is the last one with the legacy packages?** + - The last 5.0.x stable. That is fastest. The old packages will then probably stop working when networks move to protocol 15 (not verified). + - The first 5.1.x stable. Then the old packages must keep working for the whole 5.1 cycle, and both "stop publishing" and the final deletion wait until 5.1.0 is released. + + I recommend the 5.0.x option, saying it clearly in the deprecation message. +4. **Do we need e2e runs against testnet or a devnet?** The old suite supports that; its new home doesn't yet. I recommend dropping it for now and adding it later only if someone needs it.