object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found

[daviewales]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, object_store automatically detects Azure credentials from environment variables and IMDS. object_store also supports retrieving credentials from Azure CLI, however, it does not automatically try this.
If you want to retrieve credentials from Azure CLI, you need to manually specify use_azure_cli=True.

Describe the solution you'd like

Ideally, object_store would try use_azure_cli=True if no environment or IMDS credentials can be found. This makes it possible to easily test code on your local machine (based on your Azure CLI credentials), then deploy the code unchanged to an Azure VM, and it will automatically use the environment or IMDS credentials, without requiring any code changes.

Describe alternatives you've considered

A workaround is to add AZURE_USE_AZURE_CLI=True to your local environment variables while testing. However, this is an extra step.

Additional context

Most Azure libraries fall back to the Azure CLI credential if environment or IMDS credentials are not found. For example, fsspec/adlfs, azure.identity.DefaultAzureCredential, polars, pandas.

I'm aware of apache/arrow-rs#6470, but I'm wondering if simply adding a fallback to an already supported authentication method is more in scope than completely replicating DefaultAzureCredential.

Related to this downstream obstore issue: developmentseed/obstore#267 (comment)

[kylebarron]

Thanks for creating this issue. I don't know enough about Azure to argue for or against this.

I assume this would be considered a breaking change, and so it would need a resolution within the next few days (unless we want to wait a while).

[tustvold]

My 2 cents is I would not be expecting an application to be shelling out unless I told it to do so, especially given IIRC the azure binary doesn't have a very collision resistant name. I'm not familiar enough with Azure tooling to know if this is an unusual expectation.

That being said, applications wanting this behaviour by default could override the default options

[kylebarron]

I think that's a fair rationale for keeping the default here as false.

While I want the Python bindings to minimize the differences from object_store for best maintainability, I think this is a place where it might be reasonable to override the default on the application side for easy usability for Python end users. developmentseed/obstore#292

[daviewales]

Thanks. I understand the hesitation to shell out to Azure CLI. On the other hand, this is a common behaviour of many Azure tools.

Looking at the code, I think that the only Azure CLI command used is az account get-access-token.
If shelling out is the only concern, it should be possible to inspect the tokens stored in $AZURE_CONFIG_DIR/accessTokens.json, where $AZURE_CONFIG_DIR is ~/.azure by default.

This article notes that this file contains both the latest "accessToken", (60 minute expiry) and also a long lived (90 day) "refreshToken".

If the access token is expired, it should be possible to POST the refresh token to the "_authority" to get a fresh access token. For example, if the "_authority" is https://login.microsoftonline.com/common, then you should be able to get a new 60 minute access token by POSTing the refresh token to https://login.microsoftonline.com/common/oauth2/v2.0/token. Other parameters from the accessTokens.json may also be required. See this SO post for an example.

[kylebarron]

That being said, applications wanting this behaviour by default could override the default options

I implemented this in obstore in developmentseed/obstore#292, however I had initially expected that it would try the Azure CLI, and if it didn't exist or couldn't access credentials, then it would still try instance metadata credentials.

But rereading
https://github.com/apache/arrow-rs/blob/95c42a7e235dbc689dbf50636aea514331c714c9/object_store/src/azure/builder.rs#L1015-L1028
shows that I was wrong; if use_azure_cli is set to True by default, then instance credentials will never be checked unless the end user explicitly sets use_azure_cli to False

[sugibuchi]

If shelling out is the only concern, it should be possible to inspect the tokens stored in $AZURE_CONFIG_DIR/accessTokens.json, where $AZURE_CONFIG_DIR is ~/.azure by default.

This is not a feasible solution.

• .azure/accessTokens.json is an old location. The current version of Azure CLI uses .azure/msal_token_cache.json.
• .azure/msal_token_cache.json is encrypted in some environments (for example, Windows)

[daviewales]

Two thoughts:

1. Would you consider adding an option (use_azure_cli_fallback?) for the user to choose whether they want to fall back to Azure CLI? This allows a user to express the intention 'Use the first available credential, and if none are available, try Azure CLI.' The current use_azure_cli option does not support this, as enabling it skips searching other credentials.
2. The encrypted ~/.Azure/msal_token_cache.bin used by more recent Azure CLI versions can be decrypted by any process running as the current user. For example, the following Python proof of concept successfully decrypts it:

import sys
from pathlib import Path

import win32crypt # <-- requires pywin32: https://pypi.org/project/pywin32/

token_cache = Path("~/.Azure/msal_token_cache.bin").expanduser()

with open(token_cache, 'rb') as file:
    decrypted_bytes = win32crypt.CryptUnprotectData(file.read())[1]

print(decrypted_bytes.decode())

The CryptUnprotectData function is available in the various win32 Rust crates:

• windows_sys
• windows
• winapi

[tustvold]

No objection to 1, 2, however, feels like a bit of a hack and I would prefer to avoid the additional dependencies.

[kylebarron]

No objection to 1

Does that mean that when use_azure_cli_fallback is true, instance credentials will never be tried (which are the current fallback)?

[daviewales]

My expectation is that Azure CLI would be tried after the ImdsManagedIdentityProvider.
Only if creation of ImdsManagedIdentityProvider fails should Azure CLI be tried.

arrow-rs-object-store/src/azure/builder.rs

Line 1018 in 3e12e1a

let msi_credential = ImdsManagedIdentityProvider::new(

[kylebarron]

I think the creation of ImdsManagedIdentityProvider never fails. It's only at runtime, after creation, when the user makes the first request, that the instance credentials are fetched, and by that time you wouldn't be able to go back and use the azure CLI.

[daviewales]

Hmm... I guess the subsequent creation of the TokenCredentialProvider succeeds too?

arrow-rs-object-store/src/azure/builder.rs

Line 1024 in 3e12e1a

Arc::new(TokenCredentialProvider::new(

I suspect that the Azure Identity library does this is by using a credential chain, and deferring the choice of which credential to use until the token is fetched at runtime:

The identity it uses depends on the environment. When an access token is needed, it requests one using these identities in turn, stopping when one provides a token:

[kylebarron]

Hmm... I guess the subsequent creation of the TokenCredentialProvider succeeds too?

Yes, that will never fail.

ImdsManagedIdentityProvider can't be evaluated in build() because it requires a network call and therefore must be async. It appears that no part of fetching the Azure CLI token is actually async, so in principle you could synchronously try to fetch credentials from the CLI as part of build(), and if that fails, then you fall back to IMDS.

arrow-rs-object-store/src/azure/credential.rs

Lines 880 to 946 in 7272f70

	/// Fetch a token
	async fn fetch_token(&self) -> Result<TemporaryToken<Arc<AzureCredential>>> {
	// on window az is a cmd and it should be called like this
	// see https://doc.rust-lang.org/nightly/std/process/struct.Command.html
	let program = if cfg!(target_os = "windows") {
	"cmd"
	} else {
	"az"
	};
	let mut args = Vec::new();
	if cfg!(target_os = "windows") {
	args.push("/C");
	args.push("az");
	}
	args.push("account");
	args.push("get-access-token");
	args.push("--output");
	args.push("json");
	args.push("--scope");
	args.push(AZURE_STORAGE_SCOPE);
	match Command::new(program).args(args).output() {
	Ok(az_output) if az_output.status.success() => {
	let output = str::from_utf8(&az_output.stdout).map_err(|_| Error::AzureCli {
	message: "az response is not a valid utf-8 string".to_string(),
	})?;
	let token_response = serde_json::from_str::<AzureCliTokenResponse>(output)
	.map_err(|source| Error::AzureCliResponse { source })?;
	if !token_response.token_type.eq_ignore_ascii_case("bearer") {
	return Err(Error::AzureCli {
	message: format!(
	"got unexpected token type from azure cli: {0}",
	token_response.token_type
	),
	});
	}
	let duration =
	token_response.expires_on.naive_local() - chrono::Local::now().naive_local();
	Ok(TemporaryToken {
	token: Arc::new(AzureCredential::BearerToken(token_response.access_token)),
	expiry: Some(
	Instant::now()
	+ duration.to_std().map_err(|_| Error::AzureCli {
	message: "az returned invalid lifetime".to_string(),
	})?,
	),
	})
	}
	Ok(az_output) => {
	let message = String::from_utf8_lossy(&az_output.stderr);
	Err(Error::AzureCli {
	message: message.into(),
	})
	}
	Err(e) => match e.kind() {
	std::io::ErrorKind::NotFound => Err(Error::AzureCli {
	message: "Azure Cli not installed".into(),
	}),
	error_kind => Err(Error::AzureCli {
	message: format!("io error: {error_kind:?}"),
	}),
	},
	}
	}
	}