New directory layout for fixture releases: Split by fork and gas limit

[danceratopz]

Background

Test source code is organized by the fork that introduced the functionality being tested:

tests/
├── shanghai/
│   └── eip4895_withdrawals/...   # tests for EIP-4895 (introduced in Shanghai)
├── cancun/
│   └── eip4844_blobs/...         # tests for EIP-4844 (introduced in Cancun)
└── prague/
    └── eip7702_set_code_tx/...   # tests for EIP-7702 (introduced in Prague)

When vectors ("fixtures") are generated ("filled"), each test produces fixtures not just for its source fork, but for every fork from the source fork onwards. For example, a Shanghai withdrawal test generates fixtures for Shanghai, Cancun, Prague, and Osaka.

Problem

Currently all of these fixtures land in a single JSON file per test, so files grow with each new axis value:

1. Consensus fixtures grow with every new hard fork - each JSON contains fixtures for all forks >= the source fork, and will keep growing.
2. Benchmark fixtures grow with each gas-limit value passed to --gas-benchmark-values. The set doesn't grow indefinitely (it's chosen per run), but still causes significant bloat.

Solution

Each JSON file should contain fixtures for exactly one fork and (if applicable) one gas-limit value. This requires a directory layout change in the fixture output (also used in releases).

Current layout

All fork fixtures in the same file:

fixtures/
├── blockchain_tests/
│   └── shanghai/                              # all forks mixed in each file
│       ├── eip3651_warm_coinbase/
│       │   └── warm_coinbase/
│       │       ├── warm_coinbase_call_out_of_gas.json
│       │       └── warm_coinbase_gas_usage.json
│       └── eip4895_withdrawals/
│           └── withdrawals/
│               ├── use_value_in_contract.json  # has fork_Shanghai + fork_Cancun + fork_Prague + fork_Osaka
│               └── ...
├── blockchain_tests_engine_x/
│   ├── pre_alloc/*.json
│   └── shanghai/...
└── state_tests/
    └── shanghai/...

All gas-limit values in the same file:

fixtures/
├── blockchain_tests/
│   └── benchmark/compute/instruction/arithmetic/
│       └── arithmetic.json            # all gas-limits
│                                      # all forks, potentially
└── blockchain_tests_engine_x/
    ├── pre_alloc/*.json
    └── benchmark/...

Proposed layout options

Option A: Group by target fork (for_{fork}/{source})

Top level = target fork. Source and subtree nested inside.

Consensus tests:

fixtures/
├── blockchain_tests/
│   ├── for_shanghai/
│   │   └── shanghai/
│   │       └── eip3651_warm_coinbase/warm_coinbase/
│   │           └── warm_coinbase_gas_usage.json   # only fork_Shanghai fixtures
│   ├── for_cancun/
│   │   ├── shanghai/
│   │   │   └── eip3651_warm_coinbase/warm_coinbase/
│   │   │       └── warm_coinbase_gas_usage.json   # only fork_Cancun fixtures
│   │   └── cancun/
│   │       └── eip4844_shard_blob_transactions/...
│   ├── for_prague/
│   │   ├── shanghai/...
│   │   ├── cancun/...
│   │   └── prague/...
│   └── for_osaka/
│       ├── shanghai/...
│       ├── cancun/...
│       ├── prague/...
│       └── osaka/...
├── blockchain_tests_engine_x/
│   ├── pre_alloc/*.json                           # shared, not duplicated
│   ├── for_shanghai/...
│   └── ...
└── state_tests/
    ├── for_shanghai/...
    └── ...

Benchmark (single fork, multiple gas limits):

blockchain_tests/
├── for_osaka_0002M/
│   └── benchmark/compute/instruction/arithmetic/
│       └── arithmetic.json
└── for_osaka_0003M/
    └── benchmark/compute/instruction/arithmetic/
        └── arithmetic.json

Benchmark (multi-fork, multiple gas limits):

blockchain_tests/
├── for_osaka_0002M/benchmark/compute/...
├── for_osaka_0036M/benchmark/compute/...
├── for_prague_0002M/benchmark/compute/...
└── for_prague_0036M/benchmark/compute/...

Pros: Consumer-friendly - grab for_cancun/ and get everything. Preserves the existing {source}/{eip}/{module} subtree exactly.

Cons: One extra directory level. Tests sourced from Shanghai are scattered across for_shanghai/, for_cancun/, etc. Mitigated for consume via -k fork_Shanghai; additional -m markers can be added if needed.

Option B: Group by source ({source}_for_{fork})

Top level = source + target fork combined. Keeps the test source as the primary grouping.

Consensus tests:

fixtures/
├── blockchain_tests/
│   ├── shanghai_for_shanghai/
│   │   └── eip3651_warm_coinbase/warm_coinbase/
│   │       └── warm_coinbase_gas_usage.json       # only fork_Shanghai fixtures
│   ├── shanghai_for_cancun/
│   │   └── eip3651_warm_coinbase/warm_coinbase/
│   │       └── warm_coinbase_gas_usage.json       # only fork_Cancun fixtures
│   ├── shanghai_for_prague/...
│   ├── shanghai_for_osaka/...
│   ├── cancun_for_cancun/
│   │   └── eip4844_shard_blob_transactions/...
│   ├── cancun_for_prague/...
│   └── cancun_for_osaka/...
├── blockchain_tests_engine_x/
│   ├── pre_alloc/*.json                           # shared, not duplicated
│   ├── shanghai_for_shanghai/...
│   └── ...
└── state_tests/
    ├── shanghai_for_shanghai/...
    └── ...

Benchmark (single fork, multiple gas limits):

blockchain_tests/
├── benchmark_for_osaka_0002M/
│   └── compute/instruction/arithmetic/
│       └── arithmetic.json                        # only gas=2M, fork=Osaka
└── benchmark_for_osaka_0003M/
    └── compute/instruction/arithmetic/
        └── arithmetic.json                        # only gas=3M, fork=Osaka

Benchmark (multi-fork, multiple gas limits):

blockchain_tests/
├── benchmark_for_osaka_0002M/compute/...
├── benchmark_for_osaka_0036M/compute/...
├── benchmark_for_prague_0002M/compute/...
└── benchmark_for_prague_0036M/compute/...

Pros: Familiar - test source immediately visible. shanghai_for_* groups nicely in sorted listings.

Cons: Quadratic top-level directory growth (see below). Consumers must glob *_for_cancun to collect all tests for a fork.

Option C: Split at the leaf ({source}/.../fork_{fork})

Keep the existing {source}/{eip}/{module} hierarchy unchanged. Push the target fork (and gas limit) to a subdirectory at the leaf level.

Consensus tests:

fixtures/
├── blockchain_tests/
│   └── shanghai/
│       ├── eip3651_warm_coinbase/warm_coinbase/
│       │   ├── fork_shanghai/
│       │   │   └── warm_coinbase_gas_usage.json       # only fork_Shanghai fixtures
│       │   ├── fork_cancun/
│       │   │   └── warm_coinbase_gas_usage.json       # only fork_Cancun fixtures
│       │   └── fork_prague/
│       │       └── warm_coinbase_gas_usage.json       # only fork_Prague fixtures
│       └── eip4895_withdrawals/withdrawals/
│           ├── fork_shanghai/
│           │   └── use_value_in_contract.json
│           ├── fork_cancun/
│           │   └── use_value_in_contract.json
│           └── fork_prague/
│               └── use_value_in_contract.json
├── blockchain_tests_engine_x/
│   ├── pre_alloc/*.json                               # shared, not duplicated
│   └── shanghai/...
└── state_tests/
    └── shanghai/...

Benchmark (single fork, multiple gas limits):

blockchain_tests/
└── benchmark/compute/instruction/arithmetic/
    ├── fork_osaka_0002M/
    │   └── arithmetic.json
    └── fork_osaka_0003M/
        └── arithmetic.json

Benchmark (multi-fork, multiple gas limits):

blockchain_tests/
└── benchmark/compute/instruction/arithmetic/
    ├── fork_osaka_0002M/arithmetic.json
    ├── fork_osaka_0036M/arithmetic.json
    ├── fork_prague_0002M/arithmetic.json
    └── fork_prague_0036M/arithmetic.json

Pros: Smallest migration footprint - the existing top-level and mid-level directory hierarchy is untouched. No extra top-level directory layer. Linear growth in leaf subdirectories.

Cons: "Give me everything for fork X" requires a recursive glob (**/fork_cancun/**) rather than a single top-level directory - worse consumer ergonomics. Leaf directories get noisier as forks accumulate.

Quick Comparison

Option A caps the top level at N directories (one per fork) and matches the most common consumer workflow: "give me all tests for fork X" = for_cancun/.

Option B has quadratic top-level directory growth: each source fork produces a directory for every subsequent fork (frontier_for_frontier/, frontier_for_homestead/, ..., frontier_for_osaka/, homestead_for_homestead/, ...). This would lead to ~N(N+1)/2 directories; ~105 by Osaka (~14 forks), growing with every new fork.

Option C avoids top-level restructuring entirely, but forces consumers to use recursive globs (**/fork_cancun/**) to collect all tests for a fork. This trades migration simplicity for worse day-to-day ergonomics.

Consumer impact

Breaking change to both paths and JSON content:

• Paths: Breaking changes to JSON file paths.
• JSON keys: Files now contain one fork's fixtures only - filtering keys for fork_Cancun becomes unnecessary.

[marioevz]

I like option A, but using fork_{fork}/{source} instead of for_{fork}/{source}, because the word for comes out of nowhere and it's going to be hard-coded somewhere in the implementation.

[danceratopz]

Initially, I had filled_for_{fork}/{source} to try and disambiguate the name of the fork under test and test source folder name, as I don't think fork_{fork}/{source} was clear enough. fork_{fork} does match the parametrization in the fixture name, but as a folder I don't think it's obvious on first sight.

[chfast]

One suggestion if you already on the dir structure: it is sightly inconvenient that the dir with fork names are not in the fork's order. Maybe add an numeric prefix as 00_frontier, 01_homestead,...

[marioevz]

For implementation I think we should be able to configure FixtureCollector to inspect the request it's currently filling and append a subdirectory based on it.

E.g. if we are filling normal tests, "instruct" it to look into the fork marker (we don't have one but we could use one for this and other use cases too), get the value, and append a prefix directory to the output path. If we are filling benchmark tests, "instruct" it to look into the fork and gas_benchmark_value markers, and so on.

[danceratopz]

Yeah, it's quite a small change. It's (partially) implemented here:

• feat(test-fill,test-benchmark): organize fixtures into fork and gas-limit subdirs #2134

[spencer-tb]

Option D: Group by target fork, flat

Like Option A but drop the for_ prefix and the nested source fork directory. Structure is {target_fork}/{eip}/{module}/... directly.

Consensus tests:

fixtures/
├── blockchain_tests/
│   ├── shanghai/
│   │   └── eip3651_warm_coinbase/warm_coinbase/
│   │       └── warm_coinbase_gas_usage.json       # only fork_Shanghai
│   ├── cancun/
│   │   ├── eip3651_warm_coinbase/warm_coinbase/
│   │   │   └── warm_coinbase_gas_usage.json       # only fork_Cancun
│   │   └── eip4844_shard_blob_transactions/...
│   └── osaka/
│       ├── eip3651_warm_coinbase/...
│       ├── eip4844_shard_blob_transactions/...
│       └── eip7702_set_code_tx/...

Benchmark (multi-fork, multiple gas limits):

blockchain_tests/
├── osaka_0002M/benchmark/compute/...
├── osaka_0036M/benchmark/compute/...
├── prague_0002M/benchmark/compute/...
└── prague_0036M/benchmark/compute/...

[danceratopz]

Option A was implemented in #2134.