feat: update OpenVINO to use llama.cpp backend - #4377
Conversation
0020765 to
213c2d8
Compare
Switch the OpenVINO inference provider from OVMS (OpenVINO Model Server) to a llama.cpp-based backend. This simplifies the provider by removing graph/config file generation and using direct llama-server invocation. - Point to new container image quay.io/bmahabir/openvino:latest - Remove OVMS-specific code: graph.pbtxt, config-all.json, prePerform - Mount single model file instead of directory (like LlamaCppPython) - Add GGML_OPENVINO_DEVICE=CPU by default - Add GGML_OPENVINO_DEVICE=GPU + GGML_OPENVINO_STATEFUL_EXECUTION=1 when experimental GPU is enabled - Use llama-server entrypoint with env var expansion for host/port Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Brian <bmahabir@bu.edu>
213c2d8 to
c7c86ad
Compare
gastoner
left a comment
gastoner
left a comment
There was a problem hiding this comment.
Do you have some testing steps for this PR?
I do have a laptop with intel cpu but no dedicated GPU
But I should be able to check the env vars in the container right?
| }, | ||
| "openvino": { | ||
| "default": "quay.io/ramalama/openvino@sha256:e026ecbdf6ae222a193badad5b0dd2253362e366e22c8b402f5a492803b10fd5" | ||
| "default": "quay.io/bmahabir/openvino:latest" |
There was a problem hiding this comment.
Will change once ramalama adds the new upstream openvino image
I built it and pushed it to quay under my name for testing
|
@gastoner thanks for taking a look! So for testing download the unsloth llama3.2 model and use it for a recipe or a model service. Under the hood that model has the openvino backend so it should use the new container and args! What i do is if the gpu selection is enabled in setting ill pass it so technically you can disable it to use cpu but what ive found is the container is smart and will fall back to cpu anyways if its not available |
Screen.Recording.2026-04-09.at.3.22.08.AM.mov |
| }, | ||
| "openvino": { | ||
| "default": "quay.io/ramalama/openvino@sha256:e026ecbdf6ae222a193badad5b0dd2253362e366e22c8b402f5a492803b10fd5" | ||
| "default": "quay.io/bmahabir/openvino:latest" |
There was a problem hiding this comment.
issue: can't merge with this image we need a release image reference using sha
| if (isApplicationOptionsWithModelInference(options)) { | ||
| // if the recipe has a defined backend, we gives priority to using an inference server | ||
| if (options.recipe.backend && options.recipe.backend === options.model.backend) { | ||
| // OpenVINO uses llama.cpp backend, so treat openvino models as compatible with llama-cpp recipes |
There was a problem hiding this comment.
issue: this is hacky we need multi backend support that is another PR
3 participants
Summary
quay.io/bmahabir/openvino:latestprePerformoverride)GGML_OPENVINO_DEVICE=CPUby default,GGML_OPENVINO_DEVICE=GPU+GGML_OPENVINO_STATEFUL_EXECUTION=1when experimental GPU is enabledTest plan
🤖 Generated with Claude Code