remote_execution: Retry BatchReadBlobs requests

[9999years]

We've seen a bunch of (transient) failures with our build cache when downloading artifacts. This adds retry logic for BatchReadBlobs requests similar to the RemoteExecutionClient::new_retry logic:

buck2/app/buck2_execute/src/re/client.rs

Lines 194 to 217 in 83a21c0

	pub async fn new_retry(re_config: &RemoteExecutionConfig) -> buck2_error::Result<Self> {
	// Loop happens times-1 times at most
	for i in 1..re_config.connection_retries {
	match Self::new(re_config).await {
	Ok(v) => return Ok(v),
	Err(e) => {
	let e: buck2_error::Error = e.into();
	if e.find_typed_context::<RemoteExecutionError>().is_none() {
	// If we cannot connect to RE due to some non-RE error, we should not retry
	// And should just return the error immediately as it's unlikely to be flakey
	return Err(e.into());
	}
	tracing::warn!(
	"Failed to connect to RE, retrying after sleeping {} seconds: {:#?}",
	i,
	e
	);
	tokio::time::sleep(Duration::from_secs(i as u64)).await;
	}
	}
	}
	Self::new(re_config).await
	}

In the future, we'll likely want more sophisticated tuning of backoff parameters (minimum & maximum delays, jitter, exponential backoff, etc.). I'm not able to test this at the moment as the failures are flaky.

[facebook-github-bot]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this in D80221251. (Because this pull request was imported automatically, there will not be any future comments.)