fix: translate OpenAI image_url content into RubyLLM attachments (#51) (#54)

* fix: translate OpenAI image_url content into RubyLLM attachments (#51)

A multimodal user message added to the transcript in OpenAI content-array
form (a `text` part plus `{ type: "image_url", image_url: { url: ... } }`)
was passed to RubyLLM's `add_message` verbatim. RubyLLM treats a raw array
of OpenAI content hashes as plain text, so the image was silently dropped —
a vision model received text only and confabulated an answer, with no error.

Add `Raix::MultimodalContentAdapter`, which translates `image_url` parts into
RubyLLM attachments before `add_message`:
- base64 `data:` URIs are decoded into a binary StringIO, because RubyLLM's
  `Attachment` does not recognize `data:` URIs and would treat one as a
  filesystem path.
- http(s) URLs are passed through as-is (Attachment fetches them).

Anything that is not an array of hashes containing at least one `image_url`
part is returned untouched, so text completions and the Anthropic
cache_control multipart shape are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: changelog for image_url multimodal fix (#51)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: obie

CHANGELOG.md

@@ -6,6 +6,7 @@
 - `Raix::Configuration` no longer defaults `temperature` to `0.0`. The default was being injected into every request payload, which OpenRouter rejects with `404 No endpoints found that can handle the requested parameters` when routed to providers whose `supported_parameters` list omits `temperature` (notably Anthropic's Claude 4.7 family) and `provider.require_parameters: true` is set — which Raix sets automatically whenever `json: true` is passed to `chat_completion`. Callers that want a specific temperature should set one explicitly (`self.temperature = 0.0` on the including class, or `Raix.configure { |c| c.temperature = 0.0 }` globally); when unset, Raix now omits the parameter and the provider's own server-side default applies. `max_tokens`, `max_completion_tokens`, and `model` defaults are unchanged.
 - `Raix::FunctionToolAdapter` now forwards the full JSON-Schema dict for each function parameter to RubyLLM instead of rebuilding it from `type` + `description` only. Rich schema fields like `additionalProperties`, `items`, `enum`, and nested `properties` were silently dropped, leaving providers (notably Gemini's structured output via OpenRouter) to invent degenerate shapes for `type: object` arguments — e.g. emitting `{"prefix" => false}` instead of `{"prefix:title" => "..."}`. Function declarations with rich object schemas now reach the provider intact.
 - The outer tool-args schema continues to inject `additionalProperties: false` and `strict: true` by default for OpenAI strict-mode compatibility, but consumers can override either by setting them explicitly in the function declaration.
+- Multimodal `transcript` content is no longer silently dropped on the RubyLLM backend. OpenAI-style content arrays (a `text` part plus one or more `{ type: "image_url", image_url: { url: ... } }` parts) were passed to RubyLLM verbatim, which treats the array as plain text — so a vision model received text only and confabulated an answer. Raix now translates `image_url` parts into RubyLLM attachments via `Raix::MultimodalContentAdapter`, decoding base64 `data:` URIs into binary IO (RubyLLM's `Attachment` does not natively recognize `data:` URIs) and passing http(s) URLs through. Text-only completions are unaffected (#51).
 
 ## [2.0.4] - 2026-05-19
 

lib/raix/chat_completion.rb

@@ -355,7 +355,7 @@ def ruby_llm_request(params:, model:, messages:, openai_override: nil)
           chat.with_instructions(content)
         when "user"
           has_user_message = true
-          chat.add_message(role: :user, content:)
+          chat.add_message(role: :user, content: MultimodalContentAdapter.translate(content))
         when "assistant"
           if msg[:tool_calls] || msg["tool_calls"]
             chat.add_message(role: :assistant, content:, tool_calls: msg[:tool_calls] || msg["tool_calls"])

lib/raix/multimodal_content_adapter.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/hash/indifferent_access"
+require "base64"
+require "stringio"
+
+module Raix
+  # Translates OpenAI-style multimodal content arrays (a `text` part plus one or
+  # more `image_url` parts) into a RubyLLM::Content so images survive the trip to
+  # the provider.
+  #
+  # RubyLLM's `add_message`/`ask` treat a raw array of OpenAI content hashes as
+  # plain text, so an `{ type: "image_url", image_url: { url: ... } }` part is
+  # silently dropped and a vision model receives text only. See
+  # https://github.com/OlympiaAI/raix/issues/51
+  #
+  # Anything that is not an array of hashes containing at least one `image_url`
+  # part is returned untouched, so existing text completions are unaffected.
+  class MultimodalContentAdapter
+    def self.translate(content)
+      new(content).translate
+    end
+
+    def initialize(content)
+      @content = content
+    end
+
+    def translate
+      return @content unless translatable?
+
+      parts = @content.map(&:with_indifferent_access)
+      attachments = parts.select { |part| part[:type].to_s == "image_url" }
+                         .filter_map { |part| attachment_source(part.dig(:image_url, :url)) }
+      return @content if attachments.empty?
+
+      text = parts.select { |part| part[:type].to_s == "text" }.filter_map { |part| part[:text] }.join("\n")
+      RubyLLM::Content.new(text.empty? ? nil : text, attachments)
+    end
+
+    private
+
+    def translatable?
+      @content.is_a?(Array) &&
+        @content.all? { |part| part.is_a?(Hash) } &&
+        @content.any? { |part| (part[:type] || part["type"]).to_s == "image_url" }
+    end
+
+    # RubyLLM::Attachment recognizes http(s) URLs, file paths, and IO objects, but
+    # not base64 `data:` URIs (it would treat one as a filesystem path). Decode
+    # those into a binary StringIO, which Attachment handles as an IO source.
+    def attachment_source(url)
+      return if url.nil? || url.empty?
+      return url unless url.start_with?("data:")
+
+      match = url.match(/\Adata:[^;,]*;base64,(.+)\z/m)
+      return url unless match
+
+      io = StringIO.new(Base64.decode64(match[1]))
+      io.set_encoding(Encoding::BINARY) if io.respond_to?(:set_encoding)
+      io
+    end
+  end
+end

spec/raix/multimodal_content_adapter_spec.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+require "spec_helper"
+
+RSpec.describe Raix::MultimodalContentAdapter do
+  # 2x2 solid-red PNG
+  let(:red_png_base64) do
+    "iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAIAAAD91JpzAAAAEUlEQVR4nGP8z8Dwn4EIwDiqEAAQOAQBjEZ1pgAAAABJRU5ErkJggg=="
+  end
+  let(:data_uri) { "data:image/png;base64,#{red_png_base64}" }
+
+  describe ".translate" do
+    it "turns a data-URI image_url part into a RubyLLM::Content with a decoded image attachment" do
+      content = [{ type: "image_url", image_url: { url: data_uri } }]
+
+      result = described_class.translate(content)
+
+      expect(result).to be_a(RubyLLM::Content)
+      expect(result.attachments.size).to eq(1)
+      attachment = result.attachments.first
+      expect(attachment).to be_image
+      expect(attachment.mime_type).to eq("image/png")
+    end
+
+    it "turns an http image_url part into a RubyLLM::Content with a URL attachment" do
+      content = [{ type: "image_url", image_url: { url: "https://example.com/red.png" } }]
+
+      result = described_class.translate(content)
+
+      expect(result).to be_a(RubyLLM::Content)
+      attachment = result.attachments.first
+      expect(attachment).to be_url
+      expect(attachment.source.to_s).to eq("https://example.com/red.png")
+    end
+
+    it "keeps the text part alongside the image attachment" do
+      content = [
+        { type: "text", text: "What color is this?" },
+        { type: "image_url", image_url: { url: data_uri } }
+      ]
+
+      result = described_class.translate(content)
+
+      expect(result.text).to eq("What color is this?")
+      expect(result.attachments.size).to eq(1)
+    end
+
+    it "accepts string-keyed parts (OpenAI JSON shape)" do
+      content = [{ "type" => "image_url", "image_url" => { "url" => data_uri } }]
+
+      result = described_class.translate(content)
+
+      expect(result.attachments.size).to eq(1)
+    end
+
+    it "returns plain string content unchanged" do
+      expect(described_class.translate("just text")).to eq("just text")
+    end
+
+    it "leaves a text-only content array untouched" do
+      content = [{ type: "text", text: "hello" }]
+
+      expect(described_class.translate(content)).to equal(content)
+    end
+  end
+end