Passing images from action to LLM

[jscheid]

I have an action that I'd like to return both structured data as well as an image that I want the LLM to "see".

format_tool_result_content collapses the action result into a JSON string, so images embedded in the result map are sent as text, not as image content blocks the LLM can see.

I think I could work around it with a patch similar to this:

def format_tool_result_content({:ok, %{"__content_parts__" => parts} = result, _effects}) do
  clean = Map.delete(result, "__content_parts__")
  [ReqLLM.Message.ContentPart.text(encode_or_inspect(clean)) | parts]
end

... but that seems like a hack. Do you have a suggestion for a more holistic solution?