Description
Describe the bug
An AsyncPredictor
(returned when deploying an async inference endpoint) exposes .serializer
and .deserializer
properties like a typical Predictor
does... But behaviour of these properties is not consistent in async because of the way the class wraps around an internal .predictor
.
Overriding a predictor's .serializer
and .deserializer
properties after creation is useful and expected functionality (IMO) because:
- As far as I understand, the current de/serializer class architecture requires setting content_type/accept independently of individual requests. Therefore the only way to switch sent or received content type is by updating the de/serializer or re-creating the entire predictor.
- In the past, not all predictor classes have supported
serializer
anddeserializer
constructor arguments (see feature: all predictors support serializer/deserializer overrides #1997): So some samples have expected to set these properties after constructing.
To reproduce
# Create some async predictor:
predictor = some_pytorch_model.deploy(
async_inference_config=sagemaker.async_inference.AsyncInferenceConfig(
output_path="s3://doc-example-bucket/folder",
max_concurrent_invocations_per_instance=2,
),
)
# Override de/serializers (PyTorch defaults to Numpy):
predictor.serializer = sagemaker.serializers.JsonSerializer()
predictor.deserializer = sagemaker.deserializers.JsonDeserializer()
# Make a prediction:
resp = predictor.predict({ "hi": "there" })
Expected behavior
The endpoint receives a request with ContentType
and Accept
matching the configured serializers (application/json
in the above example).
Actual behavior
Because AsyncPredictor uses its own serializer, the input request is as expected.
...But because it uses .predictor property's deserializer, the overrides do not affect the response: default NumpyDeserializer and application/x-npy
Accept headers are still used.
Screenshots or logs
N/A
System information
A description of your system. Please provide:
- SageMaker Python SDK version: 2.86.2 (but checked problem seems to still affect
master
- see links) - Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch (but should be general)
- Framework version: 1.10
- Python version: py3.8
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
Today we can work around this by overriding de/serializers on both the outer (async) and inner (sync) predictor objects, as follows:
async_predictor.serializer = JsonSerializer()
async_predictor.deserializer = JsonDeserializer()
async_predictor.predictor.serializer = async_predictor.serializer
async_predictor.predictor.deserializer = async_predictor.deserializer
...But I'd recommend a better solution would be to make AsyncPredictor.serializer
and AsyncPredictor.deserializer
into @property
s that just read from and write to the inner .predictor.(de)serializer
?