Implement log probabilities for chat completions #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Log probabilities support was implemented for Ollama's OpenAI-compatible chat completion endpoints.
Key changes include:
logprobs
(boolean) andtop_logprobs
(integer) were added toopenai/openai.go
'sChatCompletionRequest
.LogProb
andTopLogProb
structs were introduced inapi/types.go
andopenai/openai.go
to define the log probability response schema.llm/server.go
'sCompletionRequest
was updated to includeLogProbs
andTopLogProbs
, and thecompletion
struct now includesCompletionProbabilities
for parsingllama.cpp
responses. TheCompletion
function now passesn_probs
to the underlyingllama.cpp
server.openai/openai.go
'sChatMiddleware
was modified to extract log probability settings from incoming requests and store them in the Gin context.server/routes.go
'sChatHandler
now retrieves these settings from the context and passes them to thellm.CompletionRequest
.openai/openai.go
(toChatCompletion
,toChunk
) andserver/routes.go
to transformllama.cpp
'scompletion_probabilities
into the OpenAI-compatiblelogprobs
format, including token, logprob, bytes, and top logprobs.openai/openai_test.go
, and aLOGPROBS_IMPLEMENTATION.md
file was created for documentation.