How to get structured output but use tools and retain messages

[ppetrick]

I have a rag chatbot, currently using usechat and streamtext, configured with openai.
Ideally, I'd like to format the output of my responses from openai, as I want to output not just text, but blocks of citations (with links), something like this:
export const rulesResponseSchema = z.object({ answer: z.string().describe('Answer to the user\'s question.'), citations: z.array(z.object({ pageNum: z.number().describe('Page number of the rule.'), citation: z.string().describe('Citation to the rule.'), })), });

I know there's streamObject and useObject, but streamObject doesn't let you use tools, and useObject doesn't manage the conversations the same way that useChat does. How might I go about achieving this? Or am I missing something obvious?

[lgrammel]

Streaming data might be a better fit for your use case: https://sdk.vercel.ai/docs/ai-sdk-ui/streaming-data

[lgrammel]

Would this be an option: https://sdk.vercel.ai/docs/ai-sdk-core/agents#structured-answers

[ppetrick]

Works great for the first question in the chat. But when I ask a second question, I get a
MessageConversionError [AI_MessageConversionError]: ToolInvocation must have a result
when callign convertToCoreMessages, which I gather is because the structured content isn't what the conversion function expects?

[lgrammel]

You can inject some fake tool call result (e.g. a string "processed" or something). It is required to ensure the conversation is consistent and can be processed by the LLM, which expects result parts for tool calls.

[ppetrick]

Appreciate the responses. Ended up using streaming data and nesting a generateobject in the onfinish of the streamtext to generate the citation object. Feels a bit clunky, but gets the job done.

[lgrammel]

We have added a experimental_output feature to generateText that allows you to create structured information while calling (openai) tools.

https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data#structured-output-with-generatetext

[mrasoahaingo]

But how do we combine generateText + structured output + useChat?

[lgrammel]

streamText supports generating structured output with AI SDK 4.0.24 or newer: https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data#structured-outputs-with-generatetext-and-streamtext

[benjaminalgreen]

@lgrammel how do you return and access the generated output?

I have tried to both return result.toDataStreamResponse(); and result.mergeIntoDataStream(dataStream); and both simply send incorrectly formatted JSON to the messages on useChat.

Alternatively, I've tried an approach like the following, which does return the structured object, but sends the full JSON with each new chunk, rather than the delta.

  let latestChunk = null;

         try {
             for await (
                 const partialOutput of experimental_partialOutputStream
             ) {
                 // Update our latest chunk
                 latestChunk = {
                     chain_of_thought: partialOutput.chain_of_thought ?? "",
                     tools_called:
                         partialOutput.tools_called?.filter((tool) =>
                             tool !== undefined
                         ) ?? [],
                     answer: partialOutput.answer ?? "",
                 };

                 // Only write the latest state
                 dataStream.writeData(latestChunk);
             }
         } catch (error) {
             console.error("Error processing stream:", error);
             dataStream.writeData({ error: String(error) });
         }
     },

[mrasoahaingo]

Hello I have fixed it on client side with an util who can parse and repair the partial json so you don't have to do anything on backend side
#4277