Dispatch error when retrieving secret from Secrets Manager

[snspinn]

My Secrets Manager code is failing to retrieve credentials but the error message is just "dispatch failure" (log attached below). I see log lines (DEBUG) with stuff like config file not found path=~/.aws/config and current throughput: 0 B/s is below minimum: 1 B/s. I would expect that AWS lambda runtime would have all CA certs available so I've been ignoring these errors. If it was an IAM retrieval error, I would expect more detailed logging (as far as I can see, IAM permissions are good).

This work fine, on my local, but fails on AWS. What am I doing wrong?

Runtime settings:

   Runtime:     Amazon Linux 2023
   Handler:      bootstrap
   Architecture:     x86_64

Can't share the complete setup but my secrets manager code is the following:

// Start of function..
let shared_config = aws_config::defaults(BehaviorVersion::v2025_01_17())
      .region("eu-west-1")
      .load()
      .await;

let client = Client::new(&config);
let resp = match client.get_secret_value().secret_id(name).send().await {
       Ok(r) => r,
        Err(e) => {
            error!("Missing secret '{}': {}", name, e);
             return Err(format!("{e}"));
        }
};

let secret_string = match resp.secret_string() {
        Some(s) => s,
        None => {
            return Err(format!("Missing secret '{name}' in SecretsManager"));
        }
};

let rds_secret = match serde_json::from_str::<RDSSecret>(secret_string) {
        Ok(s) => s,
        Err(e) => {
             error!("Error parsing secret '{}': {}", name, e);
              return Err(format!("Parsing secret error"));
        }
 // Some more code & end of function      

Running with DEBUG set gives me the following log output:
log-events-viewer-result.csv

[snspinn]

I've updated the error messaging to use some more context:

let resp = match self.client.get_secret_value().secret_id(name).send().await {
       Ok(r) => r,
       Err(e) => {
           error!("Could not retrieve secret '{}': {:?}", name, e);
...
...

The log line now reads

ERROR Lambda runtime invoke{requestId="1b61cb28-13c0-49c7-a952-a53508cfbc23" xrayTraceId="Root=1-682da77b-6d2a38f96ccd11e9239ef956;Parent=6442f41ea2a4b175;Sampled=0;Lineage=1:0546552b:0"}: Could not retrieve secret 'arn:aws:secretsmanager:eu-west-1:1234512345:secret:RDS/User': DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper_util::client::legacy::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 3.1s }), connection: Unknown } }) 

How can there be a timeout (consistent) when running on AWS?

[snspinn]

Starting to think this is a bug, on AWS, given these log lines..

DEBUG Lambda runtime invoke{requestId="1b61cb28-13c0-49c7-a952-a53508cfbc23" xrayTraceId="Root=1-682da77b-6d2a38f96ccd11e9239ef956;Parent=6442f41ea2a4b175;Sampled=0;Lineage=1:0546552b:0"}:Secrets Manager.GetSecretValue{rpc.service="Secrets Manager" rpc.method="GetSecretValue" sdk_invocation_id=3492809 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: connecting to 34.254.79.238:443
...
DEBUG Lambda runtime invoke{requestId="1b61cb28-13c0-49c7-a952-a53508cfbc23" xrayTraceId="Root=1-682da77b-6d2a38f96ccd11e9239ef956;Parent=6442f41ea2a4b175;Sampled=0;Lineage=1:0546552b:0"}:Secrets Manager.GetSecretValue{rpc.service="Secrets Manager" rpc.method="GetSecretValue" sdk_invocation_id=3492809 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: current throughput: 0 B/s is below minimum: 1 B/s

[landonxjames]

That second log is an error associated with Stalled Stream Protection. There are instructions for disabling it at that link that might help in your case. I wouldn't think Secret Managers get_secret_value would be a streaming API though.

[snspinn]

Thanks. I might try increasing the grace_period but the fact that the throughput is being logged at 0 B/s isn't encouraging (it's the same in all the other logs lines).

Wouldn't have thoughts it's a streaming API either but I haven't done anything non-standard with it so this should be default behaviour.

[snspinn]

Ok... so all I had to do was move the Lambda out of the VPC it was running in..

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.