feat: add TaskStorage derive macro and schema infrastructure

[lukesandberg]

Add a new derive macro for TaskStorage a replacement for InnerStorage and CachedDataItem, currently it is unused outside of tests.

What

• add a new derive macro that generates code from the 'schema' in storage_schema.rs
• storage_schema.rs defines the layout of our new struct, for each field we specify
  • type and name
  • category ('data' 'meta' or 'transient')
  • location (inline or lazy or flags for bitfields)
  • whether we should filter_transient when encoding (not everything can contain transient data)
  • Logically this is a replacement for CachedDataItem
• From that we generate a few things

  • TaskFlags a bitfield to hold some data items as well as 'meta' information like the 'snapshot' and 'restore' bits replacing InnerStorageState

  • LazyField enum, defines tall the fields that we dynamically allocate

  • TaskStorage the main storage struct, along with basic accessors for all the datatypes (which helps encapsualte storage strategy)

  • TaskStorageAccessors a trait with default implementations of every accessor which also handle

    • access checking, via check_access
    • mutation tracking via track_mutation
    • putting these behaviors in a trait allows some of the business logic around snapshotting to remain in normal rust code that implements the trait, while all the field updating logic can be autogenerated

  • Serialization and encoding infrastructure

    • routines to encode and decode the 'meta' or 'data' subsets
    • routines to create partial 'snapshots' of meta or data subsets
    • routines to restore data from deserialized snapshots

See https://gist.github.com/lukesandberg/ea35f0e92c1cdeddf80d0498f2978144 for a fully expanded version of the macro

Why

There are a few goals from this redesign of InnerStorage

• (speculative) a single large derive macro will be easier to maintain than the current mix of macro_rules and derive macros we use for CacheDataItem
  • There is one macro to expand when inspecting and the macro generation code is pretty straightforward if voluminous
• We eliminate many match operations on CachedDataItem
  • Today to lookup a piece of data we need to execute several matches to get the the data category and find the storage location. With this new approach those are handled by just calling the correct accessor which statically addresses the storage location
• serialization performance
  • we don't need to 'explode' the storage into Vec<CachedDataItem> anymore and instead can either directly snapshot the storage or encode directly to a buffer. This should eliminate some temporary allocations and should enable a smaller serialization format
• more flexibility
  • mostly this is the same approach as before with a few small improvements, notably invalidator and immutable are now stored as single inline bits reducing lookupcosts and memory for tasks that have these.

How

Most of the code was written by Claude with obviously a ton of feedback, so please review carefully. For now all this code is unused but future PRs will integrate it into the backend

[nextjs-bot]

Allow CI Workflow Run

• approve CI run for commit: caab350

Note: this should only be enabled once the PR is ready to go and can only be enabled by a maintainer

[nextjs-bot]

Allow CI Workflow Run

• approve CI run for commit: caab350

Note: this should only be enabled once the PR is ready to go and can only be enabled by a maintainer

[lukesandberg]

• perf: optimize cell_data serialization format #88401
• refactor: migrate to typed accessors and remove CachedDataItem adapter #88397
• feat: Replace InnerStorage with the generated TaskStorage struct #88355
• feat: add TaskStorage derive macro and schema infrastructure #88338 👈 (View in Graphite)
• docs: add rebuild guidance and ICE troubleshooting to AGENTS.md #88337
• perf(turbo-tasks-backend): use DefaultStorage for AggregationNumber to save memory #88336
• fix(turbo-tasks-backend): use correct TaskDataCategory for is_immutable check #88331
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

[nextjs-bot]

Failing test suites

Commit: 8402228 | About building and testing Next.js

pnpm test-start test/e2e/app-dir/segment-cache/prefetch-runtime/prefetch-runtime.test.ts (job)

• runtime prefetching > should not cache runtime prefetch responses in the browser cache or server-side > private caches should return new results on each request (DD)

Expand output

● runtime prefetching › should not cache runtime prefetch responses in the browser cache or server-side › private caches should return new results on each request

thrown: "Exceeded timeout of 120000 ms for a test.
Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."

  599 |     )
  600 |
> 601 |     it('private caches should return new results on each request', async () => {
      |     ^
  602 |       let page: Playwright.Page
  603 |       const browser = await next.browser('/', {
  604 |         beforePageLoad(p: Playwright.Page) {

  at it (e2e/app-dir/segment-cache/prefetch-runtime/prefetch-runtime.test.ts:601:5)
  at describe (e2e/app-dir/segment-cache/prefetch-runtime/prefetch-runtime.test.ts:546:3)
  at Object.describe (e2e/app-dir/segment-cache/prefetch-runtime/prefetch-runtime.test.ts:6:1)

[nextjs-bot]

Stats from current PR

✅ No significant changes detected

📊 All Metrics

📖 Metrics Glossary

Dev Server Metrics:

• Listen = TCP port starts accepting connections
• First Request = HTTP server returns successful response
• Cold = Fresh build (no cache)
• Warm = With cached build artifacts

Build Metrics:

• Fresh = Clean build (no .next directory)
• Cached = With existing .next directory

Change Thresholds:

• Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
• Size: Changes < 1KB AND < 1% are insignificant
• All other changes are flagged to catch regressions

⚡ Dev Server

Metric	Canary	PR	Change	Trend
Cold (Listen)	507ms	508ms	✓	█▁▁▁█
Cold (Ready in log)	494ms	497ms	✓	▃▆▆▃▅
Cold (First Request)	997ms	997ms	✓	▁██▂▃
Warm (Listen)	509ms	508ms	✓	█▁▁██
Warm (Ready in log)	494ms	491ms	✓	▆▁▁▆█
Warm (First Request)	392ms	389ms	✓	▆▁█▆▆

📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric	Canary	PR	Change	Trend
Cold (Listen)	456ms	456ms	✓	▁▁▅▁▁
Cold (Ready in log)	436ms	436ms	✓	▆▆▆▇▅
Cold (First Request)	1.732s	1.726s	✓	█▆▇█▆
Warm (Listen)	456ms	456ms	✓	█▁███
Warm (Ready in log)	435ms	435ms	✓	█▇▇█▆
Warm (First Request)	1.737s	1.753s	✓	▇▆▆▇▅

⚡ Production Builds

Metric	Canary	PR	Change	Trend
Fresh Build	4.855s	4.849s	✓	▄▁▁▄█
Cached Build	4.900s	4.843s	✓	▃▁▂▄█

📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric	Canary	PR	Change	Trend
Fresh Build	13.955s	13.917s	✓	▄▁▃▃▁
Cached Build	13.985s	13.988s	✓	▄▁▃▃▁
node_modules Size	458 MB	458 MB	✓	▁▁▁▁▁

📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **431 kB** → **431 kB** ✅ -16 B

82 files with content-based hashes (individual files not comparable between builds)

Server

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	787 B	791 B	✓
Total	787 B	791 B	⚠️ +4 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	451 B	450 B	✓
Total	451 B	450 B	✅ -1 B

📦 Webpack

Client

Main Bundles

	Canary	PR	Change
2086.HASH.js gzip	169 B	N/A	-
2161-HASH.js gzip	5.41 kB	N/A	-
2747-HASH.js gzip	4.48 kB	N/A	-
4322-HASH.js gzip	52.8 kB	N/A	-
ec793fe8-HASH.js gzip	62.3 kB	N/A	-
framework-HASH.js gzip	59.8 kB	59.8 kB	✓
main-app-HASH.js gzip	251 B	254 B	🔴 +3 B (+1%)
main-HASH.js gzip	38.6 kB	39 kB	✓
webpack-HASH.js gzip	1.68 kB	1.68 kB	✓
1596.HASH.js gzip	N/A	169 B	-
2658-HASH.js gzip	N/A	52.7 kB	-
6349-HASH.js gzip	N/A	4.46 kB	-
7019-HASH.js gzip	N/A	5.43 kB	-
b17a3386-HASH.js gzip	N/A	62.3 kB	-
Total	225 kB	226 kB	⚠️ +245 B

Polyfills

	Canary	PR	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Total	39.4 kB	39.4 kB	✓

Pages

	Canary	PR	Change
_app-HASH.js gzip	194 B	193 B	✓
_error-HASH.js gzip	182 B	182 B	✓
css-HASH.js gzip	336 B	335 B	✓
dynamic-HASH.js gzip	1.8 kB	1.8 kB	✓
edge-ssr-HASH.js gzip	256 B	256 B	✓
head-HASH.js gzip	352 B	349 B	✓
hooks-HASH.js gzip	385 B	384 B	✓
image-HASH.js gzip	580 B	580 B	✓
index-HASH.js gzip	259 B	258 B	✓
link-HASH.js gzip	2.5 kB	2.51 kB	✓
routerDirect..HASH.js gzip	319 B	317 B	✓
script-HASH.js gzip	385 B	387 B	✓
withRouter-HASH.js gzip	316 B	315 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Total	7.97 kB	7.96 kB	✅ -8 B

Server

Edge SSR

	Canary	PR	Change
edge-ssr.js gzip	125 kB	125 kB	✓
page.js gzip	242 kB	242 kB	✓
Total	367 kB	367 kB	⚠️ +423 B

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	655 B	655 B	✓
middleware-r..fest.js gzip	155 B	156 B	✓
middleware.js gzip	32.9 kB	33.3 kB	🔴 +436 B (+1%)
edge-runtime..pack.js gzip	842 B	842 B	✓
Total	34.6 kB	35 kB	⚠️ +437 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	738 B	738 B	✓
Total	738 B	738 B	✓

Build Cache

	Canary	PR	Change
0.pack gzip	3.67 MB	3.67 MB	✓
index.pack gzip	99.5 kB	99.7 kB	✓
index.pack.old gzip	100 kB	99.1 kB	🟢 1.32 kB (-1%)
Total	3.87 MB	3.87 MB	⚠️ +2.37 kB

🔄 Shared (bundler-independent)

Runtimes

	Canary	PR	Change
app-page-exp...dev.js gzip	304 kB	304 kB	✓
app-page-exp..prod.js gzip	158 kB	158 kB	✓
app-page-tur...dev.js gzip	304 kB	304 kB	✓
app-page-tur..prod.js gzip	158 kB	158 kB	✓
app-page-tur...dev.js gzip	300 kB	300 kB	✓
app-page-tur..prod.js gzip	156 kB	156 kB	✓
app-page.run...dev.js gzip	301 kB	301 kB	✓
app-page.run..prod.js gzip	156 kB	156 kB	✓
app-route-ex...dev.js gzip	68.8 kB	68.8 kB	✓
app-route-ex..prod.js gzip	47.6 kB	47.6 kB	✓
app-route-tu...dev.js gzip	68.8 kB	68.8 kB	✓
app-route-tu..prod.js gzip	47.6 kB	47.6 kB	✓
app-route-tu...dev.js gzip	68.4 kB	68.4 kB	✓
app-route-tu..prod.js gzip	47.4 kB	47.4 kB	✓
app-route.ru...dev.js gzip	68.4 kB	68.4 kB	✓
app-route.ru..prod.js gzip	47.3 kB	47.3 kB	✓
dist_client_...dev.js gzip	324 B	324 B	✓
dist_client_...dev.js gzip	326 B	326 B	✓
dist_client_...dev.js gzip	318 B	318 B	✓
dist_client_...dev.js gzip	317 B	317 B	✓
pages-api-tu...dev.js gzip	41.2 kB	41.2 kB	✓
pages-api-tu..prod.js gzip	31.3 kB	31.3 kB	✓
pages-api.ru...dev.js gzip	41.1 kB	41.1 kB	✓
pages-api.ru..prod.js gzip	31.2 kB	31.2 kB	✓
pages-turbo....dev.js gzip	50.9 kB	50.9 kB	✓
pages-turbo...prod.js gzip	38.3 kB	38.3 kB	✓
pages.runtim...dev.js gzip	50.8 kB	50.8 kB	✓
pages.runtim..prod.js gzip	38.3 kB	38.3 kB	✓
server.runti..prod.js gzip	62.3 kB	62.3 kB	✓
Total	2.69 MB	2.69 MB	⚠️ +3 B

[codspeed-hq]

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing 01-09-task-storage-macro-infrastructure (8402228) with canary (de3e9d4)}

Summary

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks¹

Footnotes

1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩