Merge #621

621: Update credential precedence to match AWS CLI r=omus a=omus

I noticed there were some credential precedence ordering differences between AWS.jl and AWS CLI. I ended up doing some experimentation with pairing different AWS CLI settings to determine the precedence ordering used by AWS CLI. Here are the results of those tests:

- aws `--profile` used over env `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`
- aws `--profile` used over env `AWS_PROFILE`
- env `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` used over env `AWS_PROFILE`
- env `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` used over config file `sso_*`
- config file `sso_*` used over `~/.aws/credentials` (if exists)
- `~/.aws/credentials` (if exists) used over config file `credential_process`
- config file `credential_process` used over config file `aws_access_key_id`/`aws_secret_access_key`
- config file `aws_access_key_id`/`aws_secret_access_key` used over EC2 instance metadata
- config file `aws_access_key_id`/`aws_secret_access_key` used over `AWS_CONTAINER_CREDENTIALS_FULL_URI`

Using `aws-cli/2.11.13 Python/3.11.3 Darwin/22.4.0 source/arm64 prompt/off`

Notes:
- Defining `sso_account_id` or `sso_role_name` in a profile without other `sso_*` keys results in an error about missing required configuration. Defining `sso_start_url` and `sso_region` by themselves doesn't produce this error.
- Specifying the AWS credential file with `AWS_SHARED_CREDENTIALS_FILE` just replaces `~/.aws/credentials`
- Tested this by specifying bad credentials in one source and valid ones in the other. As I didn't have an SSO setup to test against I could only force these to fail.
- Some additional testing was done to verify that the credential preference ordering is linear. I didn't find any examples of non-linear ordering.

Co-authored-by: Curtis Vogt <[email protected]>

Project.toml

@@ -1,7 +1,7 @@
 name = "AWS"
 uuid = "fbe9abb3-538b-5e4e-ba9e-bc94f4f92ebc"
 license = "MIT"
-version = "1.85.0"
+version = "1.86.0"
 
 [deps]
 Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

src/AWSCredentials.jl

@@ -23,7 +23,8 @@ export AWSCredentials,
     external_process_credentials,
     localhost_is_ec2,
     localhost_is_lambda,
-    localhost_maybe_ec2
+    localhost_maybe_ec2,
+    sso_credentials
 
 function localhost_maybe_ec2()
     return localhost_is_ec2() || isfile("/sys/devices/virtual/dmi/id/product_uuid")
@@ -41,20 +42,22 @@ The fields `access_key_id` and `secret_key` hold the access keys used to authent
 [Temporary Security Credentials](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) require the extra session `token` field.
 The `user_arn` and `account_number` fields are used to cache the result of the [`aws_user_arn`](@ref) and [`aws_account_number`](@ref) functions.
 
-AWS.jl searches for credentials in a series of possible locations and stops as soon as it finds credentials.
-The order of precedence for this search is as follows:
+AWS.jl searches for credentials in multiple locations and stops once any credentials are found.
+The credential preference order mostly [mirrors the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html#cli-chap-authentication-precedence)
+and is as follows:
 
-1. Passing credentials directly to the `AWSCredentials` constructor
+1. Credentials or a profile passed directly to the `AWSCredentials`
 2. [Environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
-3. Shared credential file [(~/.aws/credentials)](http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html)
-4. AWS config file [(~/.aws/config)](http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html).
-   This includes [Single Sign-On (SSO)](http://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) credentials.
-   SSO users should follow the configuration instructions at the above link, and use `aws sso login` to log in.
-5. Assume Role provider via the aws config file
-6. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
+3. [Web Identity](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-oidc)
+4. [AWS Single Sign-On (SSO)](http://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) provided via the AWS configuration file
+5. [AWS credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (e.g. "~/.aws/credentials")
+6. [External process](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html) set via `credential_process` in the AWS configuration file
+7. [AWS configuration file](http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) set via `aws_access_key_id` in the AWS configuration file
+8. [Amazon ECS container credentials](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)
+9. [Amazon EC2 instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
 
 Once the credentials are found, the method by which they were accessed is stored in the `renew` field
-and the DateTime at which they will expire is stored in the `expiry` field.
+and the `DateTime` at which they will expire is stored in the `expiry` field.
 This allows the credentials to be refreshed as needed using [`check_credentials`](@ref).
 If `renew` is set to `nothing`, no attempt will be made to refresh the credentials.
 Any renewal function is expected to return `nothing` on failure or a populated `AWSCredentials` object on success.
@@ -110,15 +113,21 @@ Checks credential locations in the order:
 function AWSCredentials(; profile=nothing, throw_cred_error=true)
     creds = nothing
     credential_function = () -> nothing
+    explicit_profile = !isnothing(profile)
     profile = @something profile _aws_get_profile()
 
-    # Define our search options, expected to be callable with no arguments.
-    # Throw NoCredentials if none are found
+    # Define the credential preference order:
+    # https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html#cli-chap-authentication-precedence
+    #
+    # Note that the AWS CLI documentation states that EC2 instance credentials are preferred
+    # over ECS container credentials. However, in practice when `AWS_CONTAINER_*`
+    # environmental variables are set the ECS container credentials are prefered instead.
     functions = [
-        env_var_credentials,
+        () -> env_var_credentials(explicit_profile),
+        credentials_from_webtoken,
+        () -> sso_credentials(profile),
         () -> dot_aws_credentials(profile),
         () -> dot_aws_config(profile),
-        credentials_from_webtoken,
         ecs_instance_credentials,
         () -> ec2_instance_credentials(profile),
     ]
@@ -314,13 +323,14 @@ function ec2_instance_credentials(profile::AbstractString)
 end
 
 """
-    ecs_instance_credentials() -> Union{AWSCredential, Nothing}
+    ecs_instance_credentials() -> Union{AWSCredentials, Nothing}
 
-Retrieve credentials from the local endpoint. Return `nothing` if not running on an ECS
-instance.
+Retrieve credentials from the ECS credential endpoint. If the ECS credential endpoint is
+unavailable then `nothing` will be returned.
 
 More information can be found at:
-https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
+- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
+- https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
 
 # Returns
 - `AWSCredentials`: AWSCredentials from `ECS` credentials URI, `nothing` if the Env Var is
@@ -331,13 +341,23 @@ https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
 - `ParsingError`: Invalid HTTP request target
 """
 function ecs_instance_credentials()
-    if !haskey(ENV, "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
+    # The Amazon ECS agent will automatically populate the environmental variable
+    # `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` when running inside of an ECS task. We're
+    # interpreting this to mean than ECS credential provider should only be used if the
+    # `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` variable is set.
+    # – https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
+    if haskey(ENV, "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
+        endpoint = "http://169.254.170.2" * ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
+    else
         return nothing
     end
 
-    uri = ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
-
-    response = @mock HTTP.request("GET", "http://169.254.170.2$uri")
+    response = try
+        @mock HTTP.request("GET", endpoint; retry=false, connect_timeout=5)
+    catch e
+        e isa HTTP.Exceptions.ConnectError && return nothing
+        rethrow()
+    end
     new_creds = String(response.body)
     new_creds = JSON.parse(new_creds)
 
@@ -355,12 +375,15 @@ function ecs_instance_credentials()
 end
 
 """
-    env_var_credentials() -> Union{AWSCredential, Nothing}
+    env_var_credentials(explicit_profile::Bool=false) -> Union{AWSCredentials, Nothing}
 
 Use AWS environmental variables (e.g. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.)
 to create AWSCredentials.
 """
-function env_var_credentials()
+function env_var_credentials(explicit_profile::Bool=false)
+    # Skip using environmental variables when a profile has been explicitly set
+    explicit_profile && return nothing
+
     if haskey(ENV, "AWS_ACCESS_KEY_ID") && haskey(ENV, "AWS_SECRET_ACCESS_KEY")
         return AWSCredentials(
             ENV["AWS_ACCESS_KEY_ID"],
@@ -375,9 +398,11 @@ function env_var_credentials()
 end
 
 """
-    dot_aws_credentials(profile=nothing) -> Union{AWSCredential, Nothing}
+    dot_aws_credentials(profile=nothing) -> Union{AWSCredentials, Nothing}
 
-Retrieve AWSCredentials from the `~/.aws/credentials` file
+Retrieve `AWSCredentials` from the AWS CLI credentials file. The credential file defaults to
+"~/.aws/credentials" but can be specified using the env variable
+`AWS_SHARED_CREDENTIALS_FILE`.
 
 # Arguments
 - `profile`: Specific profile used to get AWSCredentials, default is `nothing`
@@ -405,11 +430,45 @@ function dot_aws_credentials_file()
 end
 
 """
-    dot_aws_config(profile=nothing) -> Union{AWSCredential, Nothing}
+    sso_credentials(profile=nothing) -> Union{AWSCredentials, Nothing}
+
+Retrieve credentials via AWS single sign-on (SSO) settings defined in the `profile` within
+the AWS configuration file. If no SSO settings are found for the `profile` `nothing` is
+returned.
+
+# Arguments
+- `profile`: Specific profile used to get `AWSCredentials`, default is `nothing`
+"""
+function sso_credentials(profile=nothing)
+    config_file = @mock dot_aws_config_file()
+
+    if isfile(config_file)
+        ini = read(Inifile(), config_file)
+        p = @something profile _aws_get_profile()
+
+        # get all the fields for that profile
+        settings = _aws_profile_config(ini, p)
+        isempty(settings) && return nothing
+
+        sso_start_url = get(settings, "sso_start_url", nothing)
+
+        if !isnothing(sso_start_url)
+            access_key, secret_key, token, expiry = _aws_get_sso_credential_details(p, ini)
+            return AWSCredentials(access_key, secret_key, token; expiry=expiry)
+        end
+    end
+
+    return nothing
+end
+
+"""
+    dot_aws_config(profile=nothing) -> Union{AWSCredentials, Nothing}
 
-Retrieve AWSCredentials for the default or specified profile from the `~/.aws/config` file.
-Single sign-on profiles are also valid. If this fails, try to retrieve credentials from
-`_aws_get_role()`, otherwise return `nothing`
+Retrieve `AWSCredentials` from the AWS CLI configuration file. The configuration file
+defaults to "~/.aws/config" but can be specified using the env variable  `AWS_CONFIG_FILE`.
+When no credentials are found for the given `profile` then the associated `source_profile`
+will be used to recursively look up credentials of source profiles. If still no credentials
+can be found then `nothing` will be returned.
 
 # Arguments
 - `profile`: Specific profile used to get AWSCredentials, default is `nothing`
@@ -436,6 +495,11 @@ function dot_aws_config(profile=nothing)
             access_key, secret_key, token = _aws_get_credential_details(p, ini)
             return AWSCredentials(access_key, secret_key, token)
         elseif !isnothing(sso_start_url)
+            # Deprecation should only appear if `dot_aws_config` is called directly
+            Base.depwarn(
+                "SSO support in `dot_aws_config` is deprecated, use `sso_credentials` instead.",
+                :dot_aws_config,
+            )
             access_key, secret_key, token, expiry = _aws_get_sso_credential_details(p, ini)
             return AWSCredentials(access_key, secret_key, token; expiry=expiry)
         else

src/utilities/credentials.jl

@@ -71,7 +71,7 @@ function _aws_get_role(role::AbstractString, ini::Inifile)
     duration_seconds = get(settings, "duration_seconds", nothing)
 
     credentials = nothing
-    for f in (dot_aws_credentials, dot_aws_config)
+    for f in (sso_credentials, dot_aws_credentials, dot_aws_config)
         credentials = f(source_profile)
         credentials === nothing || break
     end