Skip to content

BaseRequest.text() leaks UnicodeDecodeError for bodies that fail to decode with the declared charset #13099

Description

@HrachShah

Describe the bug

BaseRequest.text() and the application/x-www-form-urlencoded branch of BaseRequest.post() catch LookupError (an invalid charset name) and re-raise as HTTPUnsupportedMediaType (415), but they do not catch UnicodeDecodeError (the body bytes are not valid for the declared charset). The result is a raw UnicodeDecodeError reaching the user, with a frame pointing at bytes_body.decode(encoding) instead of a clean 415.

text() is the public name users reach for when they want the body as a string, and the existing test test_request_with_wrong_content_type_encoding already asserts that the wrong-charset-name case maps to 415 — the wrong-bytes case should behave the same.

To Reproduce

import asyncio
from aiohttp.test_utils import make_mocked_request
from aiohttp import web

async def main():
    from aiohttp.base_protocol import BaseProtocol
    from aiohttp.streams import StreamReader
    from aiohttp.helpers import DEFAULT_CHUNK_SIZE

    loop = asyncio.get_event_loop()
    protocol = BaseProtocol(loop=loop)
    payload = StreamReader(protocol, DEFAULT_CHUNK_SIZE, loop=loop)
    payload.feed_data(b"\xff\xfe")
    payload.feed_eof()
    req = make_mocked_request("POST", "/", payload=payload, headers={"Content-Type": "text/html; charset=utf-8"})
    print(await req.text())  # UnicodeDecodeError leaks

asyncio.run(main())

Expected behavior

The decode error should map to HTTPUnsupportedMediaType (415), matching the existing behaviour for an invalid charset name. The text() docstring already says "Return BODY as text using encoding from .charset" — the implicit promise is that charset/bytes mismatches surface as 415, not as a leaked decode exception.

Logs/tracebacks

Traceback (most recent call last):
  File "/home/user/repro.py", line 17, in <module>
    asyncio.run(main())
  ...
  File "/path/to/aiohttp/web_request.py", line 644, in text
    return bytes_body.decode(encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Notes

The same problem exists in the urlencoded branch of post() (line 769 of web_request.py): bytes_query.decode(charset) is also wrapped in except LookupError, so the same body that triggers a leaked exception from text() will also leak from post() if the request Content-Type is application/x-www-form-urlencoded.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions