test-fast on ubuntu-24.04-arm intermittent SIGSEGV or SIGBUS in rustc
#1790
Comments
Byron
added
help wanted
|
Thanks for reporting! I didn't notice this yet and hope that this will be a rare occurrence. If not, like proposed, it could be made 'non-blocking'. |
|
Actually, it just failed on main: https://github.com/GitoxideLabs/gitoxide/actions/runs/12903033259/job/35977612391 Maybe it's best to just make it non-blocking right away. |
|
The error messages suggest to force a minimum stack size for I am about to look into whether setting that helps. Then I'll open a PR to improve the situation one way or another. (Making it non-blocking for PRs by splitting it out of Rerunning the check may make it pass, since the failure is intermittent. But I am not recommending that as a substitute for a change that would make the check fail less often (or not at all) or that would change how we treat failures of that check. |
This is to investigate the problem on the `test-fast` job with the new ARM64 runner described in GitoxideLabs#1790. This experiment does not produce useful results yet, because it has no way to distinguish happenstance from correlation. To do that, I need either to rerun each job repeatedly, or further parameterize the matrix to do that. I'll be doing the latter, but right now this dimension has size 1 (i.e., the only value of `number` is `0`) so I don't start a large number of jobs when something is broken due to a mistake in the workflows.
The previous experiment[1][2] didn't have enough of memory-related
errors to clearly show which values of the variables have an
effect, though it *looked* like the memory-related errors in
`rustc` only happened in Ubuntu 24.04 (not 22.04) and only happened
on the stable channel (not beta). That's one reason to increase the
total number of jobs in the experiment.
Another reason is that the memory-related errors are more varied.
Not all were true memory errors involving SIGSEGV and SIGBUS
anymore. Some were, same as reported in [3]. But some others were
panics, looking like this (the index and slice vary but, in each,
the start index is much larger than the length):
thread 'rustc' panicked at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/compiler/rustc_serialize/src/opaque.rs:269:45:
range start index 159846347648097871 out of range for slice of length 39963722
Since the distribution of errors across jobs might also have
related to the order and times in which jobs started, for example
if there are inadvertent differences between different hosts (the
ARM64 Linux runners are in preview, so this seems plausible, though
fairly unlikely), this now expresses the repetition with two
variables: a high-order one, listed first in the matrix, and a
low-order one, listed last in the matrix.
Besides to allow more reps with the same values of the meaningful
variables, the reason to stop testing with `RUST_MIN_STACK` is that
it didn't seem to make a difference other than to change the
message shown, which suggests setting it to an even higher value.
[1]: e71b0cf
[2]: https://github.com/EliahKagan/gitoxide/actions/runs/12903958398
[3]: GitoxideLabs#1790
As suggested in: GitoxideLabs#1790 (comment) It likely won't have to be kept this way. But making it nonrequired for now makes it so that investigating what triggers the SIGSEGV (and SIGBUS) errors -- as well as other errors that were found while investigating that (d9e7fdb, e71b0cf, 5a71963) -- doesn't have to be rushed.
I've gone ahead and done this in #1792. I suspect it can be adjusted and made blocking again, but I'm not done with the research to figure out how, so I think it does make sense to make it non-blocking temporarily. |
|
In both e71b0cf (results) and 5a71963 (results), a pattern emerges: the memory errors on ARM64 Linux CI runners seem only ever to happen on the This remains the case even if we regard panics that suggest but do not prove memory errors, such as large range start indices that are much bigger than the range, to be memory errors. I observed these in 5a71963. A third kind of error, which is a memory error even if narrowly defined, also happened only in 5a71963, and only once: in Another kind of error that I had originally assumed was unrelated was that For jobs where the So it looks like it would be sufficient to change Why do (22.04, stable), (22.04, beta), and (24.04, beta) all work, and it is only (24.04, stable) that does not? I think #1792 may still be the best first move, until I've investigated that question more. So far I've look at issues related to the runner images, and the only issue that seems maybe related, indirectly, is actions/partner-runner-images#36. I have not yet looked at what has changed between the stable and beta channels of the toolchain the ARM64 jobs are using. |
|
The SIGSEGV and "range start index" errors are rust-lang/rust#135867. |
This is to investigate the problem on the `test-fast` job with the new ARM64 runner described in GitoxideLabs#1790. This experiment does not produce useful results yet, because it has no way to distinguish happenstance from correlation. To do that, I need either to rerun each job repeatedly, or further parameterize the matrix to do that. I'll be doing the latter, but right now this dimension has size 1 (i.e., the only value of `number` is `0`) so I don't start a large number of jobs when something is broken due to a mistake in the workflows.
This makes two changes, with the intent of producing a usable test: - Removes `nightly`, since a test is currently failing on it. It can be tested later in case it fixes the SIGSEGV bug, if other changes don't help. - Have `number` take on 16 values instead of just one. This is to make it possible to figure something out about how often the failure happens with the other variables and whether the other variables make a difference. This is needed because the failures are nondeterministic, may not even usually happen, or may happen less often but still happen for some combination of the other variables. (See GitoxideLabs#1790 for context.)
The previous experiment[1][2] didn't have enough of memory-related
errors to clearly show which values of the variables have an
effect, though it *looked* like the memory-related errors in
`rustc` only happened in Ubuntu 24.04 (not 22.04) and only happened
on the stable channel (not beta). That's one reason to increase the
total number of jobs in the experiment.
Another reason is that the memory-related errors are more varied.
Not all were true memory errors involving SIGSEGV and SIGBUS
anymore. Some were, same as reported in [3]. But some others were
panics, looking like this (the index and slice vary but, in each,
the start index is much larger than the length):
thread 'rustc' panicked at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/compiler/rustc_serialize/src/opaque.rs:269:45:
range start index 159846347648097871 out of range for slice of length 39963722
Since the distribution of errors across jobs might also have
related to the order and times in which jobs started, for example
if there are inadvertent differences between different hosts (the
ARM64 Linux runners are in preview, so this seems plausible, though
fairly unlikely), this now expresses the repetition with two
variables: a high-order one, listed first in the matrix, and a
low-order one, listed last in the matrix.
Besides to allow more reps with the same values of the meaningful
variables, the reason to stop testing with `RUST_MIN_STACK` is that
it didn't seem to make a difference other than to change the
message shown, which suggests setting it to an even higher value.
[1]: e71b0cf
[2]: https://github.com/EliahKagan/gitoxide/actions/runs/12903958398
[3]: GitoxideLabs#1790
When using `dtolnay/rust-toolchain` with the `toolchain` key to specify a channel, the action version should be given as `@master`. But I accidentally kept it at `@stable`! This caused `beta` and `nightly` to refer to the most recent beta and nightly builds *prior* to the current stable version. That made the conclucions about beta and nightly builds inaccurate. This rectifies that error and repeats the experiment. See e71b0cf (1f3f6b5), GitoxideLabs#1790, and rust-lang/rust#135867 for context. (I made this mistake in both experiment 1 and experiment 2, having wrongly thought I'd changed `@stable` to `@master` for experiment 1. This commit just repeats experiment 1, but experiment 2 should also be repeated for the same reason.)
As noted in the preceding commit, when I ran experiments 1 and 2 the first time, I accidentally used `dtolnay/rust-toolchain@stable` instead of `dtolnay/rust-toolchain@master`, even though the latter is needed to use current values of the `toolchain` key rather than the builds they referred to at the time the most recent stable build was updated. The preceding commit redid experiment 1 with that fixed. This commit redoes experiment 2 with te same fix. See 5a71963 (1b3e2cd), GitoxideLabs#1790, and rust-lang/rust#135867 for context.
This is to investigate the problem on the `test-fast` job with the new ARM64 runner described in GitoxideLabs#1790. This experiment does not produce useful results yet, because it has no way to distinguish happenstance from correlation. To do that, I need either to rerun each job repeatedly, or further parameterize the matrix to do that. I'll be doing the latter, but right now this dimension has size 1 (i.e., the only value of `number` is `0`) so I don't start a large number of jobs when something is broken due to a mistake in the workflows.
This makes two changes, with the intent of producing a usable test: - Removes `nightly`, since a test is currently failing on it. It can be tested later in case it fixes the SIGSEGV bug, if other changes don't help. - Have `number` take on 16 values instead of just one. This is to make it possible to figure something out about how often the failure happens with the other variables and whether the other variables make a difference. This is needed because the failures are nondeterministic, may not even usually happen, or may happen less often but still happen for some combination of the other variables. (See GitoxideLabs#1790 for context.)
The previous experiment[1][2] didn't have enough of memory-related
errors to clearly show which values of the variables have an
effect, though it *looked* like the memory-related errors in
`rustc` only happened in Ubuntu 24.04 (not 22.04) and only happened
on the stable channel (not beta). That's one reason to increase the
total number of jobs in the experiment.
Another reason is that the memory-related errors are more varied.
Not all were true memory errors involving SIGSEGV and SIGBUS
anymore. Some were, same as reported in [3]. But some others were
panics, looking like this (the index and slice vary but, in each,
the start index is much larger than the length):
thread 'rustc' panicked at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/compiler/rustc_serialize/src/opaque.rs:269:45:
range start index 159846347648097871 out of range for slice of length 39963722
Since the distribution of errors across jobs might also have
related to the order and times in which jobs started, for example
if there are inadvertent differences between different hosts (the
ARM64 Linux runners are in preview, so this seems plausible, though
fairly unlikely), this now expresses the repetition with two
variables: a high-order one, listed first in the matrix, and a
low-order one, listed last in the matrix.
Besides to allow more reps with the same values of the meaningful
variables, the reason to stop testing with `RUST_MIN_STACK` is that
it didn't seem to make a difference other than to change the
message shown, which suggests setting it to an even higher value.
[1]: e71b0cf
[2]: https://github.com/EliahKagan/gitoxide/actions/runs/12903958398
[3]: GitoxideLabs#1790
When using `dtolnay/rust-toolchain` with the `toolchain` key to specify a channel, the action version should be given as `@master`. But I accidentally kept it at `@stable`! This caused `beta` and `nightly` to refer to the most recent beta and nightly builds *prior* to the current stable version. That made the conclucions about beta and nightly builds inaccurate. This rectifies that error and repeats the experiment. See e71b0cf (1f3f6b5), GitoxideLabs#1790, and rust-lang/rust#135867 for context. (I made this mistake in both experiment 1 and experiment 2, having wrongly thought I'd changed `@stable` to `@master` for experiment 1. This commit just repeats experiment 1, but experiment 2 should also be repeated for the same reason.)
As noted in the preceding commit, when I ran experiments 1 and 2 the first time, I accidentally used `dtolnay/rust-toolchain@stable` instead of `dtolnay/rust-toolchain@master`, even though the latter is needed to use current values of the `toolchain` key rather than the builds they referred to at the time the most recent stable build was updated. The preceding commit redid experiment 1 with that fixed. This commit redoes experiment 2 with te same fix. See 5a71963 (1b3e2cd), GitoxideLabs#1790, and rust-lang/rust#135867 for context.
In case the installation method makes a difference. Also, this brings back testing of the unstable toolchain. This has just one job for each meaningful combination, so mistakes in the experiment workflow can be found before doing nine times as much work. The experiment this prepares should hopefully shed more light on GitoxideLabs#1790 (or increase confidence in the observations so far), but this is just preparation: variation across runs will likely be due to the bug being nondeterministic.
In case the installation method makes a difference. Also, this brings back testing of the unstable toolchain. This has just one job for each meaningful combination, so mistakes in the experiment workflow can be found before doing nine times as much work. The experiment this prepares should hopefully shed more light on GitoxideLabs#1790 (or increase confidence in the observations so far), but this is just preparation: variation across runs will likely be due to the bug being nondeterministic.
This varies: - `ubuntu-22.04-arm` vs. `ubuntu-24.04.arm` GHA runner. - Installing Rust via the `rust-toolchain` action vs. with curl.sh. - Installing the stable vs. beta Rust toolchain. - Installing nextest via `install-action` quickinstall/binstall. *If* this also confirms that the only fully consistent factor in whether errors happen is `ubuntu-22.04-arm` vs. `ubuntu-24.04.arm`, then that will make it clearer that the problem is likely specific to the `ubuntu-24.04.arm` runner. See GitoxideLabs#1790 and rust-lang/rust#135867 for context.
|
I made a mistake in the testing described above (experiments 1 and 2) that slightly affected the results. It is surprising that the effect was only slight. The mistake was that I forgot to change To rectify this, I redid those two experiments with the version of that action fixed in the workflow. Oddly, the beta version continued to have very few problems. It did fail occasionally, as detailed below, which may be new, but the difference between the stable and the beta remained stark, though not as much so as the difference between 24.04 and 22.04 (as detailed below, the 22.04 runner seems to have none of these problems, with no exceptions). I found that to be very strange, so I decided to run an "experiment 3" to test if the problem might have something to do with the way the Rust toolchain was being installed: whether it was installed using an action, versus by running the bootstrap script shown the installation command on the website. While I was at it, I also tested if it might have something to do with how nextest was installed, though in hindsight I probably didn't have to test that because not all failures have to do with nextest. In short, I found that these do not seem to matter. Experiment 3 also helps confirm and clarify some of the previous results, aside from that. The three experiments (the first two redone, and the third) can be examined at:
As before, I reran jobs only when Experiment 3 seems to show:
The new testing also reinforces previous findings:
I'll open a PR soon to change the runner from Edit: I've opened #1802 for this. |
In the AArch64/ARM64 (64-bit, non-containerized) test-fast job, this uses the `ubuntu-22.04-arm` runner instead of the `ubuntu-24.04-arm` runner. This is to avoid the errors described in GitoxideLabs#1790, i.e., to work around rust-lang/rust#135867. Such problems have not been observed on the 22.04 runner, including in tests intended to find them, and switching to it seems to be a complete workaround for the problem. In contrast, continuing to use the 24.04 runner, but attempting to work around the problem by switching from the stable to the beta channel, looks like it would greatly decrease the frequency of the errors but not eliminate them. A problem with `actions/checkout` failing is likewise observed on the 24.04 runner only, so using 22.04 avoids that too. Because that seems like a complete workaround, this also reverts 50da7cb (GitoxideLabs#1792). That is to say that the ARM64 test-fast job is again in the `test-fast` matrix. It is capable of cancelling or being cancelled by the other `test-fast` checks. Code duplication in the workflow is somewhat decreased. The job will again block PR auto-merge. Similar errors do not seem to have occurred in the `test-32bit` job that runs an arm32v7 Docker image in `ubuntu-24.04-arm`, and it is not clear that changing the runner image would help with GitoxideLabs#1780, nor even if that issue is still happening. Therefore, it is not changed there at this time. This affects only ARM Linux runners. The x86-64 runners continue to use `ubuntu-latest`, which is currently resolved to `ubuntu-24.04`, and that does not need to be changed. Likewise, the `macos-latest` runners use ARM processors (Apple Silicon) and they are fine. Various experiments were done in a separate workflow. This commit also removes that workflow, because it is not actively needed anymore, and because, if kept, it would have to be modified to avoid running hundreds of extra checks on each and every push.
Current behavior 😯
Since #1777, a
test-fastCI job runs onubuntu-24.04-arm, which is one of the newly more available 64-bit ARM (AArch64/ARM64) Linux runners. This job initially worked with no problems in this job: the ARM failures mentioned in #1777 and #1778 and tracked in #1780 apply to atest-32bitjob, occur only with Docker, and do not affecttest-fast.However, the ARM64
test-fastnow intermittently fails withSIGBUSorSIGSEGVinrustc. This is probably a bug inrustscor another component of the Rust toolchain for ARM64, but I have not reproduced it locally or otherwise ruled out a problem on the runner image. The job, like the othertest-fastjobs, uses a stable toolchain.Expected behavior 🤔
The compilation should complete, or give an error, but not crash.
SIGSEGVandSIGBUSshould not occur.The underlying bug is not in gitoxide, but I'm opening this issue in gitoxide to track the problem with the affected CI job here, which may need to be removed, skipped, or made
continue-on-error.Git behavior
Not applicable.
Steps to reproduce 🕹
Run or rerun the test-fast (ubuntu-24.04-arm) job on any commit. It seems less likely to happen if
rust-cacheis able to retrieve cache dependencies, since there is less to build, but it happens even when caching retrieves everything except what is built from this repository's workspace. It may be necessary to rerun a job multiple times to observe the problem.A few runs that show this are:
The first link is to a run on the
mainbranch in my fork. The following are relevant pieces of the output of that one run (not separate runs):The text was updated successfully, but these errors were encountered: