-
|
Hello, I am trying to configure Ollama and encountered an issue after modifying the config.toml file as shown below:
Here is my docker-compose.yml file:
When I send the the Respond: {"modelfile":"# Modelfile generated by "ollama show"\n# To build a new Modelfile based on this, replace FROM with:\n# FROM starcoder:1b\n\nFROM /root/.ollama/models/blobs/sha256-397f02a8d32c293bcb63e2578a03a3d8430d8ec744f5b3180cc677e702fcd2cf\nTEMPLATE {{ .Prompt }}\nPARAMETER stop \u003c|endoftext|\u003e\n","parameters":"stop "\u003c|endoftext|\u003e"","template":"{{ .Prompt }}","details":{"parent_model":"","format":"gguf","family":"starcoder","families":null,"parameter_size":"1B","quantization_level":"Q4_0"},"model_info":{"general.architecture":"starcoder","general.file_type":2,"general.parameter_count":1237870592,"general.quantization_version":2,"starcoder.attention.head_count":16,"starcoder.attention.head_count_kv":1,"starcoder.attention.layer_norm_epsilon":0.00001,"starcoder.block_count":24,"starcoder.context_length":8192,"starcoder.embedding_length":2048,"starcoder.feed_forward_length":8192,"tokenizer.ggml.bos_token_id":0,"tokenizer.ggml.eos_token_id":0,"tokenizer.ggml.merges":null,"tokenizer.ggml.model":"gpt2","tokenizer.ggml.scores":null,"tokenizer.ggml.token_type":null,"tokenizer.ggml.tokens":null,"tokenizer.ggml.unknown_token_id":0},"modified_at":"2024-07-29T13:04:37.756033766Z"} Similarly: Respond: {"models":[{"name":"starcoder:1b","model":"starcoder:1b","modified_at":"2024-07-29T13:04:37.756033766Z","size":726080827,"digest":"77e6c46054d95d9c92f96c93df31948ed64116416a0d1ce2b882ca1641d71625","details":{"parent_model":"","format":"gguf","family":"starcoder","families":null,"parameter_size":"1B","quantization_level":"Q4_0"}}]} ErrorThe error I am getting is:
DetailsWhen I check the Ollama logs (running as a Docker container), there are no logs from Tabby. Requests from the browser are visible in the logs, but when I run Tabby, no logs are inserted. InvestigationThe issue seems to be in the RequestCould anyone help me diagnose and fix this issue? Your assistance would be greatly appreciated! Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
|
Hi - have you checked whether you're able to access |
Beta Was this translation helpful? Give feedback.
-
|
@wsxiaoys Thanks a lot for your quick response. I have still a problem with having code suggestions. If you can help I would greatly appreciate it. |
Beta Was this translation helpful? Give feedback.
-
|
This is useful. I am running Tabby on Docker Desktop. There are some issues with the Ollama interface documentation; you should not use Here is the corrected configuration in TOML format: [model.chat.http]
kind = "openai/chat"
model_name = "qwen2.5:1.5b"
api_endpoint = "http://host.docker.internal:11434/v1"
[model.completion.http]
kind = "openai/completion"
model_name = "qwen2.5-coder:7b-instruct"
api_endpoint = "http://host.docker.internal:11434/v1"
prompt_template = ""
[model.embedding.http]
kind = "openai/embedding"
model_name = "nomic-embed-text"
api_endpoint = "http://host.docker.internal:11434/v1"And here is the corresponding Docker Compose configuration: version: '3.8'
services:
open-webui:
image: ghcr.io/open-webui/open-webui:cuda
container_name: open-webui_test
restart: always
ports:
- "3339:8080"
volumes:
- open-webui:/app/backend/data
environment:
- NVIDIA_VISIBLE_DEVICES=all
extra_hosts:
- "host.docker.internal:host-gateway"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
tabby:
image: registry.tabbyml.com/tabbyml/tabby
container_name: tabby_test
command: serve
restart: always
volumes:
- "$HOME/.tabby:/data"
ports:
- "8080:8080"
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
open-webui:
external: true |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Thank you for your response. With your advice, the Docker container is up and running. I checked However, I am encountering another issue now. Tabby is not suggesting any codes. The web server is up, and it is connected to the VSCode extension. When ı try this command: I am getting eventually:
when ı looked at the ollama's logs : ✔ Container ollama Created 0.0s Also a while later it gives some warnings about cuda: ollama | cuda driver library failed to get device context 800time=2024-07-31T07:56:14.718Z level=WARN source=gpu.go:399 msg="error looking up nvidia GPU memory" Images:
Request: If you have any suggestions on how to resolve this, I would greatly appreciate it. Thank you. |
Beta Was this translation helpful? Give feedback.
Hi - have you checked whether you're able to access
http://localhost:11434from within the container? You might need to use thenetwork: hostoption to achieve that.