[FEATURE REQ] Support files and images as tool call outputs.

[RGBKnights]

Describe the feature or improvement you are requesting

Update FunctionCallOutputResponseItem to include the output property as a string or array.
This would remove a bunch of hacks we need to do now to allow the Model to see files generated by tools.
The array of objects should allow the input text, input image, input file:

Input text - A text input to the model.
text:string - The text input to the model (Required)
type:string - The type of the input item. Always input_text (Required)

Input image - An image input to the model. Learn about image inputs
type:string - The type of the input item. Always input_image (Required)
detail:string - The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto. (Optional)
file_id:string - The ID of the file to be sent to the model (Optional)
image_url:string - The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL (Optional)

Input file - A file input to the model.
type:string - The type of the input item. Always input_file (Required)
file_data:string - The base64-encoded data of the file to be sent to the model (Optional)
file_id:string - The ID of the file to be sent to the model (Optional)
file_url:string - The URL of the file to be sent to the model (Optional)
filename:string - The name of the file to be sent to the model (Optional)

Additional context

Original Post Announcing the feature:
https://x.com/OpenAIDevs/status/1971618905941856495

Documentation for the feature:
https://platform.openai.com/docs/api-reference/responses/create#responses-create-input-input-item-list-item-function-tool-call-output-output

[jsquire]

Hi @RGBKnights. Thanks for reaching out and for your inquiry. This feature area of OpenAI is a recent addition and is not yet represented in the REST API spec used for client generation. Once the REST API spec is updated to include this area and a new version published, we'll include it in the client.

[RGBKnights]

@jsquire please correct me if i am wrong but I just check the SPEC and based on the most recent version from https://github.com/openai/openai-openapi It has been updated to match the online documentation.

FunctionCallOutputItemParam:
      properties:
        id:
          anyOf:
            - type: string
              description: The unique ID of the function tool call output. Populated when this item is returned via API.
              example: fc_123
            - type: 'null'
        call_id:
          type: string
          maxLength: 64
          minLength: 1
          description: The unique ID of the function tool call generated by the model.
        type:
          type: string
          enum:
            - function_call_output
          description: The type of the function tool call output. Always `function_call_output`.
          default: function_call_output
          x-stainless-const: true
        output:
          description: Text, image, or file output of the function tool call.
          anyOf:
            - type: string
              maxLength: 10485760
              description: A JSON string of the output of the function tool call.
            - items:
                discriminator:
                  propertyName: type
                anyOf:
                  - $ref: '#/components/schemas/InputTextContentParam'
                  - $ref: '#/components/schemas/InputImageContentParamAutoParam'
                  - $ref: '#/components/schemas/InputFileContentParam'
              type: array
        status:
          anyOf:
            - $ref: '#/components/schemas/FunctionCallItemStatus'
              description: >-
                The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when
                items are returned via API.
            - type: 'null'
      type: object
      required:
        - call_id
        - type
        - output
      title: Function tool call output
      description: The output of a function tool call.

[jsquire]

@RGBKnights: Client generation is based on a TypeSpec representation, the current published version of which can be found locally in specification/base/typespec. The translation process to TypeSpec is owned by another team and performed upstream, with a drop being published for our use.

OpenAI recently made a transition to OpenAPI v3.1 for their source version, which impacted many of the downstream processes, including translation. As a result, there has been a delay in some of the new features being made available in the format we need for consumption. We're expecting this to be resolved by end of month. Unfortunately, there will be some impact of the change in the TypeSpec drop as well, so it will take a bit of time for us to fully adjust and integrate.