roachtest: adding defensive code in ceph/reef test

[sravotto]

We have seen sporadic failures in the ceph tests, due to failures in creating users in the ceph object gateway.

To address this we are adding code to check that the gateway is up by submitting a read only request, before attempting to add the user.

Epic: none

Fixes: #148731

Release note: None

[cockroach-teamcity]

This change is 

[BramGruneir]

With some minor tweaks.

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jeffswenson)

---

pkg/cmd/roachtest/tests/s3_microceph.go line 178 at r1 (raw file):

	cmd := `sudo radosgw-admin user list`
	var err error
	for i := 0; i < 10; i++ {

Do we have a standard retry loop that we use with other tests? There must be some prior art here. If not, this is good enough.

---

pkg/cmd/roachtest/tests/s3_microceph.go line 180 at r1 (raw file):

	for i := 0; i < 10; i++ {
		// Sleep for few seconds, then try the command.
		time.Sleep(2 * time.Second)

Why not put the sleep after the command fails?
It might be worth writing the err to a log as an info with a bit of context.

[sravotto]

Updated based on comments.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @BramGruneir and @jeffswenson)

---

pkg/cmd/roachtest/tests/s3_microceph.go line 178 at r1 (raw file):

Previously, BramGruneir (Bram Gruneir) wrote…

Do we have a standard retry loop that we use with other tests? There must be some prior art here. If not, this is good enough.

Done

---

pkg/cmd/roachtest/tests/s3_microceph.go line 180 at r1 (raw file):

Previously, BramGruneir (Bram Gruneir) wrote…

Why not put the sleep after the command fails?
It might be worth writing the err to a log as an info with a bit of context.

Reusing RunE retry loop.

[BramGruneir]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @jeffswenson)

[sravotto]

TFTR

bors r+

[craig]

Build succeeded:

• linux_amd64_fips_build
• examples_orms
• acceptance
• unit_tests
• local_roachtest
• local_roachtest_fips
• linux_amd64_build
• lint
• docker_image_amd64
• check_generated_code

[blathers-crl]

Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches.

---

Issue #148731: branch-release-24.3, branch-release-25.1, branch-release-25.2.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}