chore: Add non-streaming fallback for non-streaming OpenAI APIs, such as as Z-AI GLM coding API. #1292
Conversation
Jesssullivan
requested review from
andreynering and
meowgorithm
and removed request for
a team
|
All contributors have signed the CLA ✍️ ✅ |
… as as Z-AI GLM coding API
|
I have read the Contributor License Agreement (CLA) and hereby sign the CLA. |
|
There is additional work to do to ensure |
|
Added the following struct updates for compatibility, assuming the approach of extending the OpenAI provider type.
Will next evaluate #1294 mmangkad:add-reasoning-content-support with these updates before proceeding. Also, I guess at this point I'll join the discord if that is where it'd be more appropriate to discuss the addition of a seperate zai_glm specific provider. |
…M support Fixes non-streaming response handling for OpenAI-compatible APIs that require extra_body parameters like Z.AI GLM models with thinking mode. Root cause: extra_body parameters (temperature, stream, thinking) were set on the OpenAI client but NOT applied to individual request params. The OpenAI Go SDK doesn't automatically propagate client-level settings. Changes: - Add ReasoningContent field to ProviderResponse for o1/GLM thinking support - Fix critical bug by applying extra_body to request params via SetExtraFields() - Extract reasoning_content from both streaming and non-streaming responses - Add fallback mechanism for APIs returning complete responses instead of streams - Improve error handling for reasoning extraction - Update Anthropic and Gemini providers for struct compatibility Tested with Z.AI GLM-4.6 API - thinking mode parameters now correctly sent.
Integrates PR charmbracelet#1294 from @mmangkad to add support for reasoning_content field in streaming responses from custom OpenAI-compatible servers. This complements our existing non-streaming reasoning_content support, providing full coverage for APIs like Z.AI GLM that use reasoning_content instead of the standard reasoning field used by OpenAI o1 models. Changes: - Add streaming reasoning_content extraction in openai.go - Emit EventThinkingDelta events for reasoning_content chunks - Add clarifying comments distinguishing standard vs custom reasoning fields - Maintain consistent error handling pattern with existing code Credits: Based on PR charmbracelet#1294 by @mmangkad
|
Hey! Thanks for the work on this! Before you go too deep on it, would you mind trying out #1171 and seeing how it works for you? It's a fairly major refactor on how we're working with providers and I'd love to hear if it solves the problems you're working on in this PR. |
Rad, thanks so much @meowgorithm! Awesome 😸 from a quick zing through the refactor, it looks like the extra_body params from the provider config is not yet being passed through Fantasy API calls ( 😶🌫️ I think, gonna need to snoop a bit more), but given the refactor I’ll switch gears to retool this MR from #1171 and swing this MR to point to the feature branch. Ultimately I think GLM needs providerCfg.ExtraBody passed through to the API call. 🤘 |
|
Okay nice—the bulk the provider logic is in Fantasy now. If you think something is missing on that side PRs are definitely welcome. Let us know! |
Add support for per-call custom parameters via provider_options.extra_fields, working alongside the existing extra_body SDK-level defaults. This creates a two-layer architecture for custom API parameters: 1. SDK-level (extra_body): Applied at provider creation via WithJSONSet() - Scope: Provider-wide defaults for all requests - Config: providers[].extra_body 2. Per-call level (provider_options.extra_fields): Applied per-request via SetExtraFields() - Scope: Per-request customization (e.g., Z.AI GLM thinking mode) - Config: providers[].provider_options.extra_fields Changes: - internal/config/config.go: Add ProviderOptions field to ProviderConfig - internal/llm/provider/provider.go: Add providerOptions to providerClientOptions - internal/llm/provider/openai.go: Extract and apply extra_fields from provider_options The two mechanisms are complementary and work together without duplication. Builds on: @kujtimiihoxha's extra_body SDK-level support (#6bb85dc) Co-authored-by: Kujtim Hoxha <[email protected]>
3 participants
I have read the Contributor License Agreement (CLA) and hereby sign the CLA.
Relates to #1237
Add a fallback in the stream() function to handle non-streaming complete responses by calling send() and emitting the result as streaming events.
CONTRIBUTING.md.