Decide upon and document policy for what LLVM versions are acceptable for CI LLVM tooling

[Manishearth]

Our c-tiny and js-tiny tests use lld, which requires a pinned rustc (LLVM_COMPATIBLE_NIGHTLY) and LLVM that are compatible with each other.

We have existing documented policy on when MSRV and pinned Rust toolchains can be upgraded. We have no documented policy on what upgrades are allowed for the LLVM version. Updating MSRV can cascade to requiring an updated pinned Rust toolchain and LLVM version, as in #5934.

The rule we have roughly followed in the past is that an updated LLVM must be "widely available". A sufficient condition for that has been if the LLVM is available on LTS APT, GitHub Actions runners, and XCode (using this table ) This is not necessarily a necessary condition, however, and we should have one documented.

XCode does a major update only once a year and their latest version (from September) uses LLVM 17, whereas LLVM 18 has been out and in use since early 2024. Using LLVM 17 locks us to MSRVs from January: the last time I did a Rust upgrade there was a relatively narrow region of Rust versions where the Rust-LLVM coupling worked with external LLVM and we were not needing to use LLVM 18.

Personally, I do not think that the LLVM version used by software that updates once a year and does not necessarily pick up the latest LLVM from 6 months before its latest update is going to always be a workable upper bound for us. It has worked in the past, but such a litmus test can lead to the pinned nightly lagging behind for almost two years¹. I think our MSRV policy is quite well thought out and should not be overridden by the release schedule of software that may end up putting us back on MSRVs that are more than a year and a half old.

(It's worth highlighting, these binaries are used for two FFI tests: test-c-tiny and test-js-tiny)

---

Here's a proposed litmus test for things that are necessary and sufficient for an LLVM version to be considered "widely available". The relevant software binaries we care about are clang, llvm, and lld: all three must be obtainable at the requested version.

• The binaries must be available on APT on the latest Ubuntu LTS
• The binaries must be available on the GitHub actions runners without adding a PPA
• The binaries must be available via XCode or brew.
• For core ICU4X developers on corporate hardware, the binaries must be available in some normal supported way (For Googlers this amounts to "available on gLinux APT", which may lag slightly behind public APT)
• It must be possible to download and install the binaries on Windows (basically always true)

A binary not being available via XCode is dispreferred but not a blocker: if choosing between two LLVM versions that work equally well, pick the one available on latest XCode, however if the newer one enables a desired MSRV (or something else), it is fine for it to not work on XCode.

Footnotes

1. Bear in mind, Rust uses LLVM trunk, not LLVM releases, so an LLVM release's changes will be found in rustc for a fair number of months before the release. ↩

[Manishearth]

Approval requested:

• @Manishearth
• @sffc
• @zbraniecki
• @hsivonen
• @robertbastian

[hsivonen]

LGTM. (Side note: It would be good to get clang and other things that are part of the LLVM distribution to be included as optionally-installable rustup components built from the same LLVM repo revision as the LLVM instance used by rustc.)

[sffc]

For the MSRV policy, we have three tiers: "always OK", "OK with approval", and "always not OK".

If Xcode is shipping a compatible LLVM, then that's the last box for "always OK". However, I would rather not require homebrew if possible. I can totally see an iOS project that wants to use ICU4X and builds with "latest Xcode" as its toolchain, and that's exactly the environment where the LLVM versions need to match up.

Xcode is slow to update, but not that slow. Rust is just far faster.

That said, if we feel that the project has sufficient benefit from updating LLVM to a version that exceeds what is available in Xcode, then that's something the TC could agree to.

[sffc]

For reference, here is the PR where LLVM 19 was added to nightly: rust-lang/rust#127513

It seems they added it basically as soon as they could once the stable version corresponding to the nightly was to be released after the LLVM 19 release. This is extremely early.

As far as Xcode, I agree that it is regrettably a bit behind as we speak. The latest Xcode was released on 11 Dec 2024, but it is using the LLVM released on 28 Nov 2023 (12 month lag). This does appear to be a bit unusual; for example, Xcode 15.0 was released on 18 Sep 2023 and it used LLVM 16.0.0 which was released on 17 Mar 2023 (6 month lag).

Xcode appears to have a major release approximately once per year, but they have minor releases more frequently, and according to the Wikipedia page, they sometimes update LLVM versions in minor releases (e.g. 14.2, last release of 2022, to 14.3, first release of 2023).

[sffc]

One might say, "we can expect most macOS developers to have homebrew installed, so it is OK if ICU4X requires an LLVM available only via homebrew."

Reasons I don't buy this argument:

1. It is very possible to do development without homebrew. On my personal M2 macbook, I installed rustup with the shell script, and I have been able to do everything I want in ICU4X without homebrew.
2. I haven't been able to find definitive data on what proportion of iOS developers or Rust developers have homebrew or macports installed. The best I found is Stack Overflow Survey 2023 which has 33% of developers on macOS and 22% of developers using homebrew. That's a lot, but far from everyone.
3. We benefit from reducing adoption barriers as much as possible, and installing a custom LLVM version, whether via homebrew or otherwise, is a barrier. Developers are busy and if something doesn't work out of the box, and they get distracted by an email or a meeting, they might never come back.
4. As a reference point, the Flutter iOS tutorial makes no reference to homebrew. The only reference to any package manager is CocoaPods which is installable from gem which comes built-in with macOS.
5. Toolchain versions are more disruptive than other things that can be installed from homebrew. You can't simply "brew install clang-18" and be done; you need to change your Xcode build to actually use the new clang version.
6. Mac developers may be less inclined to install third-party software than developers on other platforms.

Okay, this is a bit of a soap box for me. I would rather not get into an extended point-by-point argument.

[Manishearth]

Xcode is slow to update, but not that slow. Rust is just far faster.

It's possible that XCode 16.3 or 16.4 will release with LLVM 18, but it's not guaranteed, and it seems like XCode has in the past been inconsistent about doung two LLVM upgrades per XCode version.

Looking at the dates as far as I can tell we've gotten lucky in the past few years, where we've found versions that work. This matches my experience, almost every time I've done a Rust upgrade for ICU4X I've ended up spending a bunch of time picking a nightly toolchain and realizing that, for the Rust upgrades we've chosen, the range of nightlies that works before we hit LLVM issues is actually quite small! In other words, if we had picked even a slightly newer Rust, we wouldn't have been able to make it work. And I don't think we've been particularly aggressive about updating Rust versions in these scenarios, I tend to be rather conservative when making these PRs.

I've been rather frustrated by this in the past and have even considered proposing we remove the LLVM-linked tests (I think it's just optim5); I'm not fully convinced they provide proportional value.

I would rather not get into an extended point-by-point argument.

FWIW I don't think it's particularly productive to put forth a list of points and then say you don't want to get into an extended point by point argument. But yes, I don't want to either. I'd like to understand priorities here, and not go deep into what the usage of brew is.

---

TLDR: I think my issue is less about brew and more about whether this LLVM version actually meaningfully constrains our clients in the first place

In general, I would care much more about the development environment required for this stuff if it were something we expect most of our users to need. But we don't, not for most users. It's not even something we expect most of our developers to need. This is the development environment for a particularly niche FFI CI tutorial/benchmark, one that I find myself running to need super rarely (and I touch FFI all the time), and I do not think most users will need. In fact I think I've needed to¹ run this job more often when trying to pick versions for a rust upgrade than when doing FFI/size work.

There really are two facets to this and we should be clear about which one we are discussing. Are we looking at these as benchmarks, or tutorials?

As benchmarks, I really do not care if a niche benchmark needs a slightly weirder setup. These benchmarks are nice for understanding the impact of a library change on optimized codesize, but we don't need to do that super often.
Our CI also requires dart, and the recommended way of installing dart on Mac is brew. I'd say test-c-tiny and test-js-tiny are at least as niche as test-dart. It does not seem to be that you feel differently in the context of dependencies of benchmarks/tests, but please let me know if you do.

As tutorials, it gets more interesting. As a tutorial, this is something we are asking some of our users to try. Then the question becomes if that group is niche and if requiring strangeness in the toolchain is a problem.

My overall feeling there is that the folks using the full LLVM-paired optimization will be relatively niche, and, more importantly, they will almost 100% be people who are exerting far more control over their toolchain anyway. This is basically embedded developers, and C toolchains for embedded developers are a huge mess (especially on Mac/XCode, which is basically not commonly used in embedded development anyway). I don't think these developers will be copying what we do in our makefiles, but rather using that as signposts to figure out how to tweak their already tuned toolchain.

When it comes to ICU4X clients who care about size I do not think any of them have expressed interest in tying rustc to LLVM versions. If they were to do so, I suspect they would choose the versions based on what they already use (in the case of monorepo projects, this will almost always be "close to trunk"). Basically all larger Rust-plus-C/C++ embedded projects handle their own toolchains to deal with this; this is a solved problem for them, they do not require our guidance on which Rust/LLVM versions to pick. Pure C/++ embedded projects may not have yet paired toolchains but it's typical for them to have a complex bespoke toolchain setup already, and our guidance may help them figure out what they need to do, but probbaly not be something they would be able to just follow directly².

So, I do not think the LLVM version used in these makefiles is an actual constraint on clients. I could be convinced otherwise, but I think this is my core point of contention: for the users for whom this tutorial is relevant, I do not think the actual version used here constrains them in any meaningful way.

Footnotes

1. Distinguishing "needing to" and "end up running by dint of calling test-c or something" here. I do understand that this CI task test-c-tiny is currently bundled inside test-c so people touching C/JS ffi will typically end up running this task. I'd be in support of splitting them out, I've previously been annoyed by test-c taking a while because it needs to recompile everything under test-c-tiny. ↩

2. We probably should expand the readme's of test-c-tiny and test-js-tiny to talk much more about each step and also what it means to pair LLVM/Rust versions to help these users. ↩

[sffc]

My overall feeling there is that the folks using the full LLVM-paired optimization will be relatively niche, and, more importantly, they will almost 100% be people who are exerting far more control over their toolchain anyway.

Ok, let's drill down on this point.

I don't believe these users are niche or should be considered niche. I want LTO-linked ICU4X to be the standard way app developers include us in their C++/Swift projects.

These can be regular app developers, not just professionals with custom toolchains.

The first value proposition of ICU4X is that we give small binary size. LTO is how we achieve that. Sure, gc_sections and binaryen and other alternatives that don't use LTO give some wins, but LTO gives us more.

Especially since Apple invested a lot in LLVM specifically for use in Xcode, it seems totally reasonable that we consider Xcode to be the standard environment we want to support when it comes to LTO.

[Manishearth]

So, firstly, to make sure we are on the same page: we are not talking about LTO-linked ICU4X. We are talking about cross-language LTO-linked ICU4X, specifically the use of -Clinker-plugin-lto paired with ThinLTO. LTO works just fine on stable Rust paired with any Clang. ThinLTO as well, I believe, though some flags may require nightly. The test also uses a panic-immediate-abort build which is unfortunately nightly only, but doesn't care about clang. -Clinker-plugin-lto is the only part there that cares about clang.

We don't even measure the benefit of -Clinker-plugin-lto in our size benchmarks. If this is to be the recommended way for normal FFI users to use ICU4X, I think we should at least know what the benefits are.

I did a quick benchmark on optim5 with and without linker-plugin-lto on the fixeddecimal test, it's a difference from 33K to 34K. That does not at all line up with "cross-language LTO gives us more". We could remove the cross-language LTO from that test and still get the bulk of the wins of (thin)LTO, panic-immediate-abort, two-phaselinking, gc_sections, and debug symbol stripping, which is what I was referring to in my previous comment about "proposing we remove the LLVM-linked tests". I apologize if that wasn't clear, "LLVM-linked" was specifically referring to the tests which are "linked" to the LLVM version, which is not particularly clear use of terminology in this context 😄 .

"ICU4X has small binary size as a value proposition" is something I quite agree with. I do not think this automatically implies any particular set of flags that we expect users to achieve this with, and if we wish to go down that route I think that needs careful thought and consensus. I also think it needs far more measurements than what we have in tree right now.

I also think that the statement "-Clinker-plugin-lto is the standard way app developers include ICU4X in their C++/Swift projects" is not one that I have considered true for the project so far and would not be in favor of considering true for the project in the future. Not saying that you are proposing that, but in case this discussion goes that way, I'd be against it.

[sffc]

LTO works just fine on stable Rust paired with any Clang. ThinLTO as well, I believe, though some flags may require nightly.

To me "LTO" is synonymous with the Clang procedure that involves the compilation step emitting LLVM bitcode with metadata, and then the linker merging the bitcode from all compilation units.

According to the Clang docs:

In ThinLTO mode, as with regular LTO, clang emits LLVM bitcode after the compile phase. The ThinLTO bitcode is augmented with a compact summary of the module.

gc-sections does something similar but without the LLVM bitcode metadata, so it doesn't do as good of a job. As with anything in this area, mileage will vary, but LTO should "always" be better since it operates with more information.

[Manishearth]

To me "LTO" is synonymous with the Clang procedure that involves the compilation step emitting LLVM bitcode with metadata, and then the linker merging the bitcode from all compilation units.

That's still ambiguous. The linker is invoked in multiple phases in our compilation here, it can be invoked by rustc using LLVM bitcode for Rust, it can be invoked by Clang using LLVM bitcode for Clang, and if you use -Clinker-plugin-lto, the Clang invocation is able to consume Rust's LLVM bitcode as well. It is only this final case where LLVM versions need to be in concordance. Everything else works fine and is a relatively normal set of flags to use.

My statement stands: Regardless of what people use "LTO" to mean, the specific optimization that is causing problems here is -Clinker-plugin-lto. This optimization is a relatively niche one, which does bring benefits, but the benefits it brings are slight compared to just using regular LTO, ideally with two compilation steps and ThinLTO.

(benchmarks from #5948)

# note: LTO builds are bloated unless stripped
# stripped here refers to stripping + gc_sections
# panic-abort here refers to panic-immediate-abort std

-rwxr-x--- 1 manishearth primarygroup   47416 Dec 25 12:03 panic-abort-clang.elf
-rwxr-x--- 1 manishearth primarygroup   47416 Dec 25 12:03 panic-abort-clang-stripped.elf

-rwxr-x--- 1 manishearth primarygroup  133520 Dec 25 12:03 panic-abort-lto-clang-twostep.elf
-rwxr-x--- 1 manishearth primarygroup   34040 Dec 25 12:03 panic-abort-lto-clang-twostep-stripped.elf

-rwxr-x--- 1 manishearth primarygroup   33296 Dec 25 12:09 panic-abort-linker-plugin-lto-clang-twostep.elf
-rwxr-x--- 1 manishearth primarygroup   33296 Dec 25 12:08 panic-abort-linker-plugin-lto-clang-twostep-stripped.elf

gc-sections/stripping + two-step ThinLTO provides almost all the benefits available to LTO. Cross language LTO is providing a 2% improvement here.

If this discussion were about choices made by ICU4X that prevent this 2% codesize win I would definitely consider it worth thinking about. But this discussion is about choices made by ICU4X that make this codesize win a little bit harder to achieve on certain platforms. Other codesize win options are already hard to achieve, build-std requires nightly (many, many clients have a distaste of Rust nightly) and we have experienced build-std breakage in the past. I haven't properly benchmarked it but I suspect the build-std-panic-immediate-abort option gets a roughly similar order of magnitude of codesize win in some cases.

(n.b. It's unclear to me if we should always advocate for ThinLTO: The point of using ThinLTO is link-time performance, not binary size, and my benchmarks here don't measure thin vs regular but instead unconditionally use thin for two-step. We may wish to check that at some point, but that's not the core issue in this discussion so I haven't drilled deeper)

[Manishearth]

Quick notes of discussion with @sffc:

• We should come up with better documentation for size-relevant benchmarks (Improved c-tiny benchmark #5948 (comment))
• A policy of the form "pinned LLVM must be available on Debian LTS/brew/GitHub Actions, and ideally is available on XCode" is acceptable: similar to the MSRV policy, this would mean that when LLVM is not available on XCode we ask for TC approval. @sffc and I believe TC approval would not be challenging for this case, with the argument that 2.0 is a good opportunity to make a big MSRV update, after which we can be conservative with MSRV, providing a smoother user experience.

We should run this by WG (probably TC) at some ICU4X meeting. This is basically a tweak of my proposal above (#5935 (comment)) with XCode being a soft requirement that can be broken with TC consensus.

[sffc]

Another nitpick: please include a place to look to see whether the requirements "must be available on Debian LTS", "GitHub Actions", and "brew" are met. For Xcode, we use the Wikipedia article. I don't want to be looking around and fishing through places like pkgs.org in order to determine whether the requirement is satisfied.

[Manishearth]

There's no such place for Github Actions, but that's easily discovered by trying it on CI which you have to do anyway to make such a PR. GitHub Actions almost always works if the Ubuntu LTS works, it's worth including in the policy as a signpost for "please do not add the LLVM PPA to our GitHub CI if you feel the need to".

XCode

Policy: Should be available on the latest XCode, but policy can be overridden by TC approval in cases where there is a strong case for a new feature.

Command: none

Verification: Look at the bottom of XCode's Wikipedia page and check that the latest released XCode has an LLVM version equal to or greater than the one being used.

Brew

Policy: Must be available via brew.

Command: brew install llvm@18 on OSX.

Verification: Visit https://formulae.brew.sh/formula/llvm#default and ensure the llvm@N package is listed, for version N.

Ubuntu

Policy: Must be available on the oldest still-supported Ubuntu LTS.

Command apt install llvm-18 on Ubuntu LTS.

Verification: Package should be available on oldest still-supported Ubuntu LTS. Visit the page https://launchpad.net/ubuntu/+source/llvm-toolchain-18 (with the number replaced with the desired LLVM version), and scroll down to see if the oldest active Ubuntu LTS is there. Currently, that is 20.04 Focal Fossa, starting April 2025 it will be Jammy Jellyfish.

Debian

Policy: Must be available on Debian testing.

Command apt install llvm-18 on Debian testing.

Verification: Visit the page https://packages.debian.org/search?keywords=llvm-18&searchon=names&suite=testing&section=all (with the number 18 replaced by the desired LLVM version) and ensure the package is listed under "Exact matches".

Note: "Debian stable" is typically very old and is not what most Debian-based systems use. E.g. Debian stable is currently on Rust 1.63, which is more than two years old. Stable is intended to be extremely stable without even performing backwards-compatible upgrades.

GitHub Actions

Policy: Must be available on GitHub actions ubuntu-latest (i.e. ICU4X CI should be able to just apt install the LLVM)

Command apt install llvm-18 on ICU4X CI.

Verification: ubuntu-latest is always older than the oldest still-supported Ubuntu LTS, so this does not need to be verified separately. There are potential edge cases where GitHub's package repository mirrors are lagging behind where this policy basically means that we should wait a few days for things to smooth out.

ICU4X developer machines

ICU4X developers may request additional criteria for their machines in case their machines use nonstandard package management. These criteria will be listed below.

Policy: Should be available on ICU4X developer machines based on criteria below, with overrides requiring approval from affected developers.

Verification: Affected ICU4X developers are in charge of testing this: if developer A wishes to perform an LLVM upgrade to a version satisfying all of the above policies, they may do so without checking if it works on developer B's machine, but developer B is allowed to block or revert the PR if they have issues.

The current criteria are:

• Available on GLinux: GLinux packages are based on Debian testing so this should almost always be true based on previous criteria, but there may be occasional cases where things lag behind for a week or two.

[Manishearth]

PR with policy so far in #5949. Can mark as draft if desired.

[sffc]

We should include rpm releases, too. Proposed policy:

Required: LLVM version should be available in the latest Fedora:

1. Determine the latest Fedora here: https://fedoraproject.org/
2. Check for the package by substituting the version into the following URL. For example, currently it is version 41: https://dl.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/l/

Preferred but can be overridden with TC approval: LLVM version should be available in the "epel" repository for the latest Rocky Linux release:

1. Determine the latest Rocky Linux release by visiting this URL: https://rockylinux.org/download
2. Check for the package in EPEL by substituting the version into the following URL. For example, currently it is version 9: https://dl.fedoraproject.org/pub/epel/9/Everything/source/tree/Packages/l/

Currently Fedora has LLVM 19, and Rocky Linux (RHEL) has LLVM 16.

[Manishearth]

I'm fine with requiring RPM/Fedora.

For EPEL/Rocky/RHEL I would like a very clear justification and I do not think it should go in the initial policy. We do not know the needs of clients on RHEL, and we do not have RHEL machines to play around with.

At the moment, it is unclear if "available on RPM" is sufficient for RHEL users. I suspect it may be sufficient.

It certainly appears to be that the official Red Hat docs for RHEL suggest installing LLVM 17 using yum without any fancy steps. They also have steps for LLVM 18.

EPEL to me does not appear to be the primary source of packages for RHEL. It's an additonal community thing.

---

My general rule of thumb here is that for constraints that are likely to become blockers for us, I wish to put thought into establishing them before we do so. If we are establishing a rule for, say, RHEL based on EPEL, we must understand:

• Is EPEL the default/recommended way of getting LLVM on RHEL?
  • Current answer: does not appear to be
• Are alternate ways of getting LLVM considered onerous on RHEL?
  • This is subjective, but we should hear from RHEL users.
  • We had a similar discussion on brew vs XCode, and the point of XCode being the core driver of iOS development was compelling.
• What is the likelihood of people wishing to do size-optimized builds on RHEL and caring about the above two points?
  • For example, the above issues are bolstered by the size-optimization-needing users (phone app devs) being the ones tied to XCode. But I'm not aware of RHEL systems being used as a host or target for size optimizations.

(I don't have as strong a rule of thumb for saying "yes" to RPM because RPM is fast and unlikely to be the blocker here. If RPM is lagging behind to that extent, it's likely other things are as well)

[sffc]

the official Red Hat docs for RHEL

yum module install llvm-toolset indeed seems like a good place to look.

EPEL to me does not appear to be the primary source of packages for RHEL. It's an additonal community thing.

I use Rocky Linux for my services in Octave Online and rely heavily on EPEL. It's annoying when things aren't in EPEL and I'm forced to source it from somewhere else.

[Manishearth]

I'm rather against including EPEL in the criteria in that case. Rocky is not a major Linux distribution, and it is not a major platform for size-optimized builds. If enough clients come to us with a desire for size-optimized builds on Rocky. There are lots of Linux distributions and package managers and not all of them have the latest packages for everything.

The primary targets for size-optimized builds are going to be iOS mobile development (done on OSX, XCode) and Android mobile development (primarily done on Windows in Android Studio, with some Linux/OSX, but Android Studio does not work well there). We've spent a bunch of time optimizing devex for OSX users (which I think is definitely worth doing), but the devex for size-optimized builds on Windows is still "download LLVM from the website" and that is unlikely to change. In the face of that, I do not see why we should optimize devex for size optimized builds on platforms like Rocky.

One of my goals here is for the LLVM policy to not end up with an effective rule of "ask TC approval for each LLVM upgrade". LLVM upgrades have been a pain point that I consider disproportionate to their benefit for a long time, and while I think the XCode policy we have come up with here strikes a good balance (we expect XCode to be manageable some of the time, and a little friction in the other times is fine), EPEL's old packages will give us a de facto policy of "ask the TC for every LLVM upgrade".

[sffc]

I use Rocky to build binary artifacts in CI which I then deploy on Rocky VMs. I use Rocky because it doesn't require a paid license but is more stable than Fedora. Google Cloud Platform and numerous other large players have gotten behind Rocky ever since Red Hat Inc discontinued CentOS.

But, to be clear, I'm okay with yum module install llvm-toolset being the yardstick instead of EPEL.

[Manishearth]

Ah, that works for me. I don't wish to apply my rule of thumb to platforms that tend to keep LLVM up to date in a timely manner (as mentioned before, the rule of thumb is for "constraints that are likely to become blockers"), since the cases where that constraint doesn't hold are very likely to be cases where either we are cutting it too close (we should wait a bit), or where that platform is having LLVM troubles (in which case it seems fine for us to wait a bit).

I added it as a "SHOULD, override with TC approval" criteria, using pkgs.org as the verification source. Currently pkgs.org has LLVM 18 on Rocky and 19 on RHEL (but you can explicitly request older versions with yum module install, so that's fine).