Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure

[Markzipan]

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:

1. A persistently running frontend server is initialized once when a build starts.
2. build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
3. When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
4. The frontend server processes compilation requests and serves compiled JS files back to build_runner (hot-reload ready).

Major changes:

• Adds a DdcFrontendServerBuilder to our set of DDC builders (enabled via the web-hot-reload config). This builder keeps a PersistentFrontendServer instance alive across rebuilds. Compile/recompile requests are queued via a FrontendServerProxyDriver resource.
• Uses scratch_space to record both 1) the main app entrypoint and 2) updated local files from the entrypoint_marker builder and the module_builder builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The entrypoint_marker builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request.
• Requires that strongly connected components in both the frontend server and build_runner be disabled.

Test changes:

• Extends build_test to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds.
• build_test doesn't support runs_before and other ordering rules in build.yaml, so the above changes allows a kind of imperative ordering, which is important for testing entrypoint_marker.

Minor changes:

• Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
• Added disposal logic to scratch_space so that rebuilds only retain modified files.
• Updated scratch_space package_config.json specs (packageUri and rootUri). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in scratch_space.
• Added file and uuid deps to build_modules.
• Moved around some helper functions.
• Ported some naming functions from the DDC runtime.

Currently doesn't support live-reloading (functionality appears to have been broken a while ago). This'll be added in an upcoming change and permit webdev-like auto-hot-reload on save (on top of manual).

Enable this by adding the following to a project's build.yaml:

global_options:
  build_web_compilers|sdk_js:
    options:
      web-hot-reload: true
  build_web_compilers|entrypoint:
    options:
      web-hot-reload: true
  build_web_compilers|ddc:
    options:
      web-hot-reload: true
  build_web_compilers|ddc_modules:
    options:
      web-hot-reload: true

[github-actions]

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:build	4.0.2	already published at pub.dev
package:build_config	1.2.0	already published at pub.dev
package:build_daemon	4.1.0	already published at pub.dev
package:build_modules	5.1.0	ready to publish	build_modules-v5.1.0
package:build_runner	2.10.2-wip	WIP (no publish necessary)
package:build_test	3.5.2-wip	WIP (no publish necessary)
package:build_web_compilers	4.4.0	ready to publish	build_web_compilers-v4.4.0
package:scratch_space	1.2.0	ready to publish	scratch_space-v1.2.0

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[Markzipan]

@davidmorgan Wow, the tests in build/build_runner/test/integration_tests are perfect for this. I've replaced the build_web_compilers FES driver test with additional web_compilers_test.dart tests, added a separate resource for FES-specific state, and bumped scratch_space's ver.

Thanks for the review - back from vacation!

[davidmorgan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

[davidmorgan]

Welcome back!

Looking good so far.

There are some CI failures, could you take a look at those please?

[davidmorgan]

Hmmmm multiple expectations is a bit awkward, I wonder if there's a nicer way.

Would it work to accumulate lines and return them, then the second expectation can be checked on the result?

Looking at how this method is used, I think maybe something like this:

• most callers are not using the result, so how about the base method is Future<void> expect.
• a few callers are using the single matching line, so how about Future<String> expectAndGetLine for that
• and then I think you can do what you want with the full accumulated output until the match, how about Future<List<String>> expectAndGetBlock

?

[Markzipan]

I like that change, but I the newly added tests' reqs are:

1. we should be able to expect that a test throws
2. we should be able to match on more complex patterns (more than one line of match)

The problem with returning a block is that expect waits on a stream until it times out for 30s or encounters pattern, so we don't have a good way to determine when a block is finished accumulating. Really it should just accept a List<Matcher> and return if all matchers are successful. Thoughts?

[davidmorgan]

When you say "test throws" do you mean "logs an error"? It's supposed to be that no exceptions escape :)

build_runner should always run the build to completion then either print "build succeeded" or "build failed", so it should always work to use one of those or a RegExp that matches either to capture a block.

If this doesn't work could you please point to a specific test so I can understand? Thanks.

[Markzipan]

Yep, logs an error (i.e., "build failed"). Hmm, watcher tests appear to stream the results of a build until some string or Regexp is matched - line by line. I'd like to both match specific text of a line and match the "build failed" text (which is output on a later line), which isn't supported by the current testing scheme (unless I've missed something). The 'can recompile incrementally after invalid edits' test is an example of this. await watch.expect(BuildLog.successPattern);, which seems to be the canonical failure detecting check, doesn't let me check the error text.

[davidmorgan]

So currently you have this

    await watch.expect(
      'Hot reload rejected due to unsupported changes',
      expectFailure: true,
    );

I think you could add a new method expectAndGetBlock, that was the name given above, but maybe linesUntil would be better:

  final block = await watch.linesUntil(BuildLog.failurePattern);
  expect(block, contains(contains('Hot reload rejected due to unsupported changes')));

?

Per my original comment I think "expect" could be three versions: one that returns nothing, one that returns the matched line, and one that returns all the lines. Or maybe drop the single line version, you can always just do ".last" on the block?

[davidmorgan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

Whoops, looks like I had unpublished comments that you addressed by coincidence. Sorry about that! Now published.

[Markzipan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

@davidmorgan Oh no - you didn't see my comments? I can ping you when things are fully ready, but I've been in and out since I'm sick with something mysterious lol

@nshahan Thanks for reviewing! I updated some of the formatting based on your feedback.

It looks like I've encountered a windows pathing issue in some SDK lib. I'll investigate tomorrow :(

[davidmorgan]

@davidmorgan Oh no - you didn't see my comments? I can ping you when things are fully ready, but I've been in and out since I'm sick with something mysterious lol

I think we're back in sync now, I'll keep commenting on changes/comments, and take a final look when we think it's done :) thanks.