Bug: sam cli build stuck on mounting....

[moryachok]

Description:

sam cli build --use-container stuck when I use extras install in requirements txt.

Examples of requirements.txt:
pyiceberg[s3fs,glue] - stuck
pyiceberg[s3fs] - works
pyiceberg[glue] - works

Steps to reproduce:

add pyiceberg[s3fs,glue] to your requirements.txt and run build

Observed result:

2024-04-09 00:16:26,330 | Config file location:                                                                
/workdir/samconfig.toml 
2024-04-09 00:16:26,333 | Loading configuration values from [default.['build'].parameters]                     
(env.command_name.section) in config file at                                                                   
'/workdir/samconfig.toml
'...                                                                                                           
2024-04-09 00:16:26,335 | Configuration values successfully loaded.                                            
2024-04-09 00:16:26,335 | Configuration values are: {'stack_name': 'iceberg-metadata-quality-metrics-lambda',  
'cached': True, 'parallel': True}                                                                              
2024-04-09 00:16:26,341 | Using SAM Template at                                                                
/workdir/template.yaml  
2024-04-09 00:16:26,382 | Using config file: samconfig.toml, config environment: default                       
2024-04-09 00:16:26,383 | Expand command line arguments to:                                                    
2024-04-09 00:16:26,384 |                                                                                      
--template_file=/workdir
/template.yaml --use_container --skip_pull_image --parallel --mount_with=READ --build_dir=.aws-sam/build       
--cache_dir=.aws-sam/cache --cached                                                                            
2024-04-09 00:16:26,751 | 'build' command is called                                                            
2024-04-09 00:16:26,752 | Starting Build use cache                                                             
2024-04-09 00:16:26,753 | Starting Build inside a container                                                    
2024-04-09 00:16:26,756 | No Parameters detected in the template                                               
2024-04-09 00:16:26,789 | There is no customer defined id or cdk path defined for resource                     
IcebergMetricsLambda, so we will use the resource logical id as the resource id                                
2024-04-09 00:16:26,790 | 0 stacks found in the template                                                       
2024-04-09 00:16:26,791 | No Parameters detected in the template                                               
2024-04-09 00:16:26,811 | There is no customer defined id or cdk path defined for resource                     
IcebergMetricsLambda, so we will use the resource logical id as the resource id                                
2024-04-09 00:16:26,812 | 1 resources found in the stack                                                       
2024-04-09 00:16:26,812 | Found Serverless function with name='IcebergMetricsLambda' and CodeUri='lambda/'     
2024-04-09 00:16:26,813 | --base-dir is not presented, adjusting uri lambda/ relative to                       
/workdir/template.yaml  
2024-04-09 00:16:26,818 | 1 resources found in the stack                                                       
2024-04-09 00:16:26,818 | Found Serverless function with name='IcebergMetricsLambda' and CodeUri='lambda/'     
2024-04-09 00:16:26,820 | Instantiating build definitions                                                      
2024-04-09 00:16:26,824 | Same function build definition found, adding function (Previous:                     
BuildDefinition(python3.9,                                                                                     
/workdir/lambda, Zip,   
081c274373128da86d62672696f9bd7b875a5b61f6b0c3d96e460bf358971ab0, 926ae397-f412-4f55-aceb-e3cb2b5b9e00, {}, {},
x86_64, []), Current: BuildDefinition(python3.9,                                                               
/workdir/lambda, Zip, , 
f9a545cb-15fc-4272-a385-fef103c336b2, {}, {}, x86_64, []), Function:                                           
Function(function_id='IcebergMetricsLambda', name='IcebergMetricsLambda', functionname='IcebergMetricsLambda', 
runtime='python3.9', memory=128, timeout=60, handler='app.lambda_handler', imageuri=None, packagetype='Zip',   
imageconfig=None,                                                                                              
codeuri='/workdir/lambda
', environment=None, rolearn=None, layers=[], events=None, metadata={'SamResourceId': 'IcebergMetricsLambda'}, 
inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None,                 
function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build 
with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))                          
2024-04-09 00:16:26,830 | Async execution started                                                              
2024-04-09 00:16:26,831 | Invoking function functools.partial(<bound method                                    
ParallelBuildStrategy.build_single_function_definition of                                                      
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x11160b7a0>>,                                
<samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x11156e600>)                                  
2024-04-09 00:16:26,832 | Running incremental build for runtime python3.9 for following resources              
(IcebergMetricsLambda)                                                                                         
2024-04-09 00:16:26,833 | Waiting for async results                                                            
2024-04-09 00:16:26,835 | Cache is invalid, running build and copying resources for following functions        
(IcebergMetricsLambda)                                                                                         
2024-04-09 00:16:26,836 | Building codeuri:                                                                    
/workdir/lambda runtime:
python3.9 metadata: {} architecture: x86_64 functions: IcebergMetricsLambda                                    
2024-04-09 00:16:26,838 | Building to following folder                                                         
/workdir/.aws-sam/build/
IcebergMetricsLambda                                                                                           
2024-04-09 00:16:26,850 | Checking free port on 127.0.0.1:6651                                                 
2024-04-09 00:16:26,856 | Requested to skip pulling images ...                                                 
                                                                                                               
2024-04-09 00:16:26,857 | Mounting                                                                             
/workdir/lambda as      
/tmp/samcli/source:ro,delegated, inside runtime container   

Expected result:

build finish successfully

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MACOS
2. sam --version: reproduced on versions 1.78.0 and 1.114.0
3. AWS region: us-east-1

# Paste the output of `sam --info` here
{
  "version": "1.114.0",
  "system": {
    "python": "3.12.2",
    "os": "macOS-13.6.4-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "25.0.3",
    "aws_cdk": "2.24.1 (build 585f9ca)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[lucashuy]

Thanks for raising this issue. I can reproduce the issue where it is stuck building the project. It is actually doing something, however that isn't being streamed back to terminal while the container runs. It looks like pip is having problems trying to resolve the dependencies between glue and s3fs.

It looks like the glue extra installs botocore==1.34.80:

botocore==1.34.80
    # via
    #   boto3
    #   s3transfer

While s3fs will install botocore==1.34.51:

botocore==1.34.51
    # via aiobotocore

Both of these were installed separately in their own venv, and pip-compile was used to list the installed child dependencies.

pip is trying to download and compare every single dependency between them to find something that works.

It looks like apache/iceberg-python#534 in the iceberg repository fixes most of the issues with dependency resolution when I pulled it down to quickly test. Can you check in with the folks over there just to confirm?

As a workaround for now, you can pin boto3 and botocore just before installing pyiceberg, like this:

boto3==1.34.51
botocore==1.34.51
pyiceberg[s3fs,glue]

[moryachok]

Hey @lucashuy, your workaround worked for me, thanks for deep dive.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.