roachtest: schemachange/secondary-index-multi-version failed [timed out: expected n1 to be at new cluster version, but is still at 25.2]

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/secondary-index-multi-version failed with artifacts on master @ b8495513102a1857855b48ece7fd29fe5414c064:

(mixedversion.go:903).Run: mixed-version test failure while running step 70 (wait for all nodes (:1-3) to acknowledge cluster version <current> on system tenant): timed out after 10m0s: expected n1 to be at cluster version 25.3-upgrading-to-25.4-step-014, but is still at 25.2: not upgraded yet
test artifacts and logs in: /artifacts/schemachange/secondary-index-multi-version/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• mvtDeploymentMode=separate-process
• mvtVersions=v24.3.20 → v25.1.10 → v25.2.6 → v25.4.0-alpha.2-dev-b8495513102a1857855b48ece7fd29fe5414c064 (master)
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-55406

[spilchen]

It seems like it could be a duplicate of #155322

[spilchen]

It looked like the test framework didn't perform the upgrade. test-eng can you take a look?

[blathers-crl]

cc @cockroachdb/test-eng

[DarrylWong]

Looks like the same issue as #154923 where the upgrade to 25.4 hangs, but we also don't see any retry logs. Seems somewhat concerning that these are popping up so frequently and only on 25.4, but I'm unsure how to debug further.

I want to see a repro with --vmodule=auto_upgrade=2, but unable to get one locally. Maybe I just put out a PR and hope we hit it in the nightlies?