UnicodeEncodeError: 'ascii' codec can't encode character in DeepTeam red teaming attacks

[mahi2694]

Describe the bug
I am facing a recurring UnicodeEncodeError when running DeepTeam red teaming attacks against my model callback. Despite configuring my environment fully for UTF-8 (all locale variables, PYTHONIOENCODING, and Python default encoding are UTF-8), and attempting to encode/decode inputs in UTF-16 and UTF-32 inside the callback, the error persists.

Error :
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 7: ordinal not in range(128)

To Reproduce

1. Run the simplest DeepTeam example with a basic bias vulnerability and model_callback returning simple strings.
2. Observe the following error traceback (full traceback included below).
3. Try encoding inputs inside the callback as UTF-16 or UTF-32; error is unchanged.
4. Environment variables:
   LANG=en_US.UTF-8
   LC_ALL=en_US.UTF-8
   PYTHONIOENCODING=utf-8

Python encoding confirmed as UTF-8 by:
import sys, locale print(sys.stdout.encoding) print(sys.getdefaultencoding()) print(locale.getpreferredencoding())

Expected behavior
DeepTeam should correctly handle Unicode prompts or sanitize content appropriately to avoid passing non-ASCII characters to ASCII-only components (like HTTP headers).

Screenshots

(deepteam-py310) mahithamahesh@Mahithas-MacBook-Pro-2 deepteam % python sample.py
💥 Generating 1 attacks (for 1 vulnerability types across 1 vulnerability(s)):   0%|                                                                                 | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/Users/mahithamahesh/mahitha-work/deepteam/sample.py", line 12, in <module>
    risk_assessment = red_team(model_callback=model_callback, vulnerabilities=[bias], attacks=[prompt_injection])
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/red_team.py", line 31, in red_team
    risk_assessment = red_teamer.red_team(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/red_teamer/red_teamer.py", line 87, in red_team
    return loop.run_until_complete(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/red_teamer/red_teamer.py", line 236, in a_red_team
    await self.attack_simulator.a_simulate(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/attacks/attack_simulator/attack_simulator.py", line 143, in a_simulate
    attack_results = await asyncio.gather(*simulate_tasks)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/attacks/attack_simulator/attack_simulator.py", line 127, in throttled_simulate_baseline_attack
    result = await self.a_simulate_baseline_attacks(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/attacks/attack_simulator/attack_simulator.py", line 228, in a_simulate_baseline_attacks
    return await vulnerability.a_simulate_attacks(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepteam/vulnerabilities/bias/bias.py", line 243, in a_simulate_attacks
    res, _ = await self.simulator_model.a_generate(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepeval/models/retry_policy.py", line 584, in attempt
    return await coro
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/deepeval/models/llms/openai_model.py", line 370, in a_generate
    completion = await client.beta.chat.completions.parse(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/openai/resources/chat/completions/completions.py", line 1621, in parse
    return await self._post(
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/openai/_base_client.py", line 1794, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/openai/_base_client.py", line 1515, in request
    request = self._build_request(options, retries_taken=retries_taken)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/openai/_base_client.py", line 495, in _build_request
    headers = self._build_headers(options, retries_taken=retries_taken)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/openai/_base_client.py", line 439, in _build_headers
    headers = httpx.Headers(headers_dict)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/httpx/_models.py", line 156, in __init__
    bytes_value = _normalize_header_value(v, encoding)
  File "/Users/mahithamahesh/miniconda3/envs/deepteam-py310/lib/python3.10/site-packages/httpx/_models.py", line 82, in _normalize_header_value
    return value.encode(encoding or "ascii")
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 7: ordinal not in range(128)
💥 Generating 1 attacks (for 1 vulnerability types across 1 vulnerability(s)):   0%|                                                                                 | 0/1 [00:03<?, ?it/s]
(deepteam-py310) mahithamahesh@Mahithas-MacBook-Pro-2 deepteam % 

Desktop

• OS: iOS
• Browser NA

Name: deepteam
Version: 0.2.6

Name: deepeval
Version: 3.6.6

Python 3.10.18

[mahi2694]

Hi, any update on this?

[trewwwsec]

Upgrade your httpx. The current httpx (0.28.1) already handles Unicode headers correctly.

The bug was in old httpx versions (pre-2021) that defaulted to ASCII for header encoding.