-
Couldn't load subscription status.
- Fork 800
Description
Describe your environment
Runtime: Python 3.9, amd64
Dependencies: Only those that come with the OpenTelemetry layer for AWS Lambda (v1.14.0)
Cloud Provider: AWS (specifically, AWS Lambda)
This issue does still appear to exist in main, but I have not had a chance to verify. The issue centers around the format of the handler string (it breaks if it is slash-delimited) and the instrumenter currently loads ORIG_HANDLER without modification, ultimately leading to attempting to patch an invalid import path (e.g., package1/package2/module.target).
Lines 372 to 377 in 80d0b89
| lambda_handler = os.environ.get(ORIG_HANDLER, os.environ.get(_HANDLER)) | |
| # pylint: disable=attribute-defined-outside-init | |
| ( | |
| self._wrapped_module_name, | |
| self._wrapped_function_name, | |
| ) = lambda_handler.rsplit(".", 1) |
Steps to reproduce
I created https://github.com/myles2007/OpenTelemIssue-PythonLambdaLayer-v1.14.0 which defines a sample application which reproduces the error, includes CloudFormation for the infrastructure, and commands to deploy the example. From there, you only need to invoke the Lambda function named SlashPathWithLayerBroken.
If you would prefer to recreate manually, the basic reproduction steps are:
- Create a Python application with a structure like:
package1 / package2/ module.py handler (function) - Deploy the application using AWS Lambda and specify the handler as:
package1/package2/module.handlerWithout instrumentation applied, this works.
- Enable the region-appropriate Python Lambda Layer for OpenTelemetry:
- For
us-east-1:arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-14-0:1 - Also set
AWS_LAMBDA_EXEC_WRAPPERto "/opt/otel-instrument"
- For
- Invoke the Lambda function and receive the following error:
{ "errorMessage": "Unable to import module 'otel_wrapper': No module named 'package1/package2/module'", "errorType": "Runtime.ImportModuleError", "requestId": "b8fefc59-1377-4cf3-aec6-ba7ea8cdfa84", "stackTrace": [] }
What is the expected behavior?
As it does without instrumentation, the invocation should succeed without issue even when slash-delimited paths are used.
What is the actual behavior?
Invocation with the AWS Lambda auto instrumentation in enabled results in an import error at runtime.
Additional context
Lambda function handlers can be specified in at least the two following ways for Python Lambda functions:
package1.package2.module.handlerpackage1/package2/module.handler
Either of these forms is valid and executes module.handler as expected when the function is invoked. However, the OpenTelemetry Lambda layer does not properly handle the second form. If it is enabled, invocation results in the following error response when slash-based paths are used:
{
"errorMessage": "Unable to import module 'otel_wrapper': No module named 'package1/package2/module'",
"errorType": "Runtime.ImportModuleError",
"requestId": "b8fefc59-1377-4cf3-aec6-ba7ea8cdfa84",
"stackTrace": []
}The otel_wrapper.py script handles both forms by modifying the ORIG_HANDLER path (see here), but AwsLambdaInstrumentor().instrument() does not and instrumenting the function fails with the error above.
I believe the error originates within the call to wrapt.wrap_function_wrapper here when patching occurs because the slash-based path format is not a valid Python import path.
Thoughts on Resolving the Issue
Below are a few thoughts on how this issue might be resolved, including a simple workaround. I'm happy to contribute a change, but would like guidance from maintainers on the preferred approach, whether one of mine or something different.
Workaround
This issue can easily be worked around by specifying the handler location using dot-based paths. Nothing about project structure needs to change.
Modify ORIG_HANDLER in otel_wrapper.py
Modifying the ORIG_HANDLER environment variable within otel_wrapper.py before instrumentation occurs by replacing it with the modified path that is computed for use within itself resolves the issue.
While this works, it does mutate the ORIG_HANDLER variable in a way which makes it no longer truly the original handler path. Side effects outside of the intended scope are definitely possible, but it might also help prevent similar issues sharing ths root cause.
This is the fix I implemented when debugging this issue. It does work and requires only minimal modifications. This change would be made in https://github.com/open-telemetry/opentelemetry-lambda, not this repository.
Modify AwsLambdaInstrumentor.instrument
The AwsLambdaInstrumentor.instrument method could be modified to locally compute the path to the ORIG_HANDLER as otel_wrapper.py does (i.e., by replacing / with .). This would ensure that the path is valid for import, preventing the error.
This fix duplicates the path modification logic, but it reduces the potential of unexpected side effects by keeping the scope small and targeted.
Please let me know if anything is unclear or if more information is needed somewhere. Thanks!