feat(provider/google): Url support for pdfs and images in tool results

[songkeys]

Background

This PR solves issue #8014 for google provider, mirroring the changes from #9733 .

Google gemini API supports uploading pdf with url: https://ai.google.dev/gemini-api/docs/document-processing#large-pdfs-urls

Summary

Uses the image-url and file-url types to send urls to the provider.

Manual Verification

Used the example files to verify the functionality.

Checklist

• Tests have been added / updated (for bug fixes / features)
• Documentation has been added / updated (for bug fixes / features)
• A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
• I have reviewed this pull request (self-review)

Future Work

Related Issues

Related: #9733
Fixes #8014

[vercel]

The default MIME type for image-url tool results is 'image/*', but this is a wildcard pattern used in HTTP Accept headers, not a valid concrete MIME type. It should be 'image/jpeg' to be consistent with how the same file handles generic image types elsewhere (line 95) and to match patterns in other providers.

View Details

📝 Patch Details

diff --git a/packages/google/src/convert-to-google-generative-ai-messages.test.ts b/packages/google/src/convert-to-google-generative-ai-messages.test.ts
index 861244b61..771ac99a8 100644
--- a/packages/google/src/convert-to-google-generative-ai-messages.test.ts
+++ b/packages/google/src/convert-to-google-generative-ai-messages.test.ts
@@ -498,4 +498,48 @@ describe('assistant messages', () => {
       ],
     });
   });
+
+  it('should default to image/jpeg for url-based image content with unknown extension', async () => {
+    const result = convertToGoogleGenerativeAIMessages([
+      {
+        role: 'tool',
+        content: [
+          {
+            type: 'tool-result',
+            toolName: 'image-generator',
+            toolCallId: 'image-gen-2',
+            output: {
+              type: 'content',
+              value: [
+                {
+                  type: 'image-url',
+                  url: 'https://example.com/image.xyz',
+                },
+              ],
+            },
+          },
+        ],
+      },
+    ]);
+
+    expect(result).toEqual({
+      systemInstruction: undefined,
+      contents: [
+        {
+          role: 'user',
+          parts: [
+            {
+              fileData: {
+                mimeType: 'image/jpeg',
+                fileUri: 'https://example.com/image.xyz',
+              },
+            },
+            {
+              text: 'Tool executed successfully and returned this image as a response',
+            },
+          ],
+        },
+      ],
+    });
+  });
 });
diff --git a/packages/google/src/convert-to-google-generative-ai-messages.ts b/packages/google/src/convert-to-google-generative-ai-messages.ts
index f41da214d..2341d1893 100644
--- a/packages/google/src/convert-to-google-generative-ai-messages.ts
+++ b/packages/google/src/convert-to-google-generative-ai-messages.ts
@@ -226,7 +226,7 @@ export function convertToGoogleGenerativeAIMessages(
                 case 'image-url': {
                   const mimeType = inferMimeTypeFromUrl(contentPart.url, {
                     extensionMap: imageExtensionMimeTypes,
-                    defaultMimeType: 'image/*',
+                    defaultMimeType: 'image/jpeg',
                   });
 
                   parts.push(

Analysis

Invalid MIME type image/* used as default for tool result image URLs

What fails: convertToGoogleGenerativeAIMessages() in packages/google/src/convert-to-google-generative-ai-messages.ts uses image/* as the default MIME type when a tool result image URL has an unknown file extension, causing Google's Generative AI API to reject requests with an invalid MIME type.

How to reproduce:

const result = convertToGoogleGenerativeAIMessages([
  {
    role: 'tool',
    content: [
      {
        type: 'tool-result',
        toolName: 'image-generator',
        toolCallId: 'gen-1',
        output: {
          type: 'content',
          value: [
            {
              type: 'image-url',
              url: 'https://example.com/image.xyz', // Unknown extension
            },
          ],
        },
      },
    ],
  },
]);
// result.contents[0].parts[0].fileData.mimeType === 'image/*'

Result: Requests with mimeType: 'image/*' are sent to the Google Generative AI API. Since image/* is a wildcard pattern for Accept headers (RFC 2045), not a valid concrete MIME type, the API rejects the request.

Expected: Should default to image/jpeg to match:

1. The established pattern in the same file (lines 93-95): part.mediaType === 'image/*' ? 'image/jpeg' : part.mediaType
2. Other providers' implementations: Groq, OpenAI, and OpenAI-compatible all convert image/* to image/jpeg
3. MIME type specifications: Concrete MIME types (e.g., image/jpeg, image/png) are required in Content-Type headers and API payloads; wildcards are only valid in Accept headers

Fix: Changed line 229 from defaultMimeType: 'image/*' to defaultMimeType: 'image/jpeg' and added test case for unknown file extensions to prevent regression.