Skip to content

[Caching] Reduce the cost of computing compile job cache keys - #2229

Merged
cachemeifyoucan merged 2 commits into
swiftlang:mainfrom
cachemeifyoucan:eng/PR-189417668
Oct 15, 2026
Merged

cachemeifyoucan merged 2 commits into
swiftlang:mainfrom
cachemeifyoucan:eng/PR-189417668

Conversation

@cachemeifyoucan

@cachemeifyoucan cachemeifyoucan commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Computing the compilation cache keys was expensive for large modules in
batch mode. The base cache key stored the entire command-line, including
the content of the file list, as a single blob that was different for
every job and every build, so each key stored a large new blob in the
CAS. The driver also computed the base key again for every output
producing input of a batch job.

This changes the following in swift and swift-driver:

  • Split the base cache key into the arguments that are stable across
    the jobs in a module, the clang arguments, and the input list, each
    stored as a blob that can be shared between keys. The primary inputs
    are stored in the base key node, and the arguments of the options
    labeled with the new ArgumentIsCASID flag are added as references
    to the CAS objects instead of strings, so the new objects for each
    job are small. A labeled CAS ID that is not in the CAS is an error.
  • Store the clang include tree at a fixed position in the base cache key
    and add createCASFileSystemFromCacheKey to replace
    iterateCommandLine, which is used by LLDB.
  • Add swiftscan_cache_compute_keys_from_input_indices to compute the
    cache keys for multiple inputs in one call, and use it in the driver
    when available.
  • Support the ArgumentIsCASID flag in swift-driver's makeOptions.

In a synthetic module with 1000 files, the CAS growth for a one line
edit goes from 86 MB to 0.7 MB when the driver computes the keys for
all the files. In a module with 2653 files, computing the cache keys
while planning a clean build goes from about 2.5s to 0.2s.

The driver change to skip the cache keys for the jobs that are batched
later is in #2227.

rdar://189417668

swift: swiftlang/swift#93076
llvm-project: swiftlang/llvm-project#14603

Support the new `ArgumentIsCASID` flag from swift options, which labels
the options whose argument is a CAS ID when compilation caching is
enabled, and update the options that have the flag.

rdar://189417668
Use `swiftscan_cache_compute_keys_from_input_indices` when available to
compute the cache keys for all the output producing inputs of a compile
job at once, so the command-line is only processed once for a batch job.
Fall back to computing the key for each input otherwise.

rdar://189417668
@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test

@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test windows platform

@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

Windows test doesn't work until swift change is landed due to the new makeOption added.

@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test windows platform

@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test windows platform

1 similar comment
@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test windows platform

@justice-adams-apple

Copy link
Copy Markdown

@swift-ci please test windows

1 similar comment
@cachemeifyoucan

Copy link
Copy Markdown
Contributor Author

@swift-ci please test windows

@cachemeifyoucan
cachemeifyoucan merged commit d970b70 into swiftlang:main Oct 15, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants