Module
Ollama
Testcontainers version
1.20.1
Using the latest Testcontainers version?
Yes
Host OS
Windows 11
Host Arch
x86
Docker version
Docker Desktop 4.33.1
docker version
Client:
Version: 27.1.1
API version: 1.46
Go version: go1.21.12
Git commit: 6312585
Built: Tue Jul 23 19:57:57 2024
OS/Arch: windows/amd64
Context: desktop-linux
Server: Docker Desktop 4.33.1 (161083)
Engine:
Version: 27.1.1
API version: 1.46 (minimum version 1.24)
Go version: go1.21.12
Git commit: cc13f95
Built: Tue Jul 23 19:57:19 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.19
GitCommit: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
runc:
Version: 1.7.19
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker Desktop 4.34.2
docker version
Client:
Version: 27.2.0
API version: 1.47
Go version: go1.21.13
Git commit: 3ab4256
Built: Tue Aug 27 14:17:17 2024
OS/Arch: windows/amd64
Context: desktop-linux
Server: Docker Desktop 4.34.2 (167172)
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:15 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
What happened?
While doing a few test runs with OllamaContainer start, Docker Desktop for Windows offered to update itself. I did it, and immediately after the update I got this new error during OllamaContainer startup.
Relevant log output
23:35:12.207 [main] INFO tc.ollama/ollama:0.3.12 -- Container ollama/ollama:0.3.12 is starting: a93d17ce634c2f2ca1ae284f3e1b83ea12d92fc6b84ef46a63a8ddf3ec8fe28c
23:35:12.495 [main] ERROR tc.ollama/ollama:0.3.12 -- Could not start container
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'\nnvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown"}
Additional Information
I tried downgrading from 4.34.2 to 4.33.1, and the new OllamaContainer startup error went away.
Out of curiosity, I updated to 4.34.2 again and debugged OllamaContainer.java. I think I found a probable cause.
- Run
docker info CLI against Docker Desktop 4.33.1 => Runtimes: io.containerd.runc.v2 runc
- Run
docker info CLI against Docker Desktop 4.34.2 => Runtimes: io.containerd.runc.v2 nvidia runc => New nvidia runtime added
- OllamaContainer.java line 38 checks if
nvidia runtime is listed in docker info output.
- I am running Win 11 Pro on AMD 7600x with iGPU. I don't have a discrete nVidia GPU.
To confirm my theory, I ran these docker CLI commands against Docker Desktop 4.34.2. I was able to reproduce the error logged by OllamaContainer startup.
Start ollama/ollama:0.3.12 with default runtime => Success
docker run --rm -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.3.12
c439ad3e9f053f80e362794955753d5993e794d69fbc870829b1af5264b14af1
Start ollama/ollama:0.3.12 with nvidia runtime => Failure with same error logged by OllamaContainer
docker run --rm --runtime=nvidia -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.3.12
3511b245dafedf705d3ccdacc1cfc07d45efcaa47543eea27d3800f8051290df
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown.
I think the nvidia runtime check in OllamaContainer.java worked OK up to Docker Desktop 4.33.x, but the change in 4.34 broke that check.
If you know of a workaround, I would be happy to try it. However, I did a quick search and I don't think Docker Desktop offers an option to remove, hide, or disable the nvidia runtime. For now, I am unblocked by downgrading to 4.33.1.
BTW, I am guessing other Win platforms are affected, but I only have the AMD 7600 iGPU to try an reproduce:
- AMD 7000/9000 series iGPU
- AMD 8000 series APU (i.e. I think APU means beefier iGPU; trade off of less CPU cores for more iGPU)
- Discrete AMD GPU
- Discrete Intel GPU
- Others?
Module
Ollama
Testcontainers version
1.20.1
Using the latest Testcontainers version?
Yes
Host OS
Windows 11
Host Arch
x86
Docker version
Docker Desktop 4.33.1
Docker Desktop 4.34.2
What happened?
While doing a few test runs with OllamaContainer start, Docker Desktop for Windows offered to update itself. I did it, and immediately after the update I got this new error during OllamaContainer startup.
Relevant log output
Additional Information
I tried downgrading from 4.34.2 to 4.33.1, and the new OllamaContainer startup error went away.
Out of curiosity, I updated to 4.34.2 again and debugged
OllamaContainer.java. I think I found a probable cause.docker infoCLI against Docker Desktop 4.33.1 =>Runtimes: io.containerd.runc.v2 runcdocker infoCLI against Docker Desktop 4.34.2 =>Runtimes: io.containerd.runc.v2 nvidia runc=> Newnvidiaruntime addednvidiaruntime is listed indocker infooutput.To confirm my theory, I ran these docker CLI commands against Docker Desktop 4.34.2. I was able to reproduce the error logged by OllamaContainer startup.
Start
ollama/ollama:0.3.12with default runtime => SuccessStart
ollama/ollama:0.3.12withnvidiaruntime => Failure with same error logged by OllamaContainerI think the
nvidiaruntime check in OllamaContainer.java worked OK up to Docker Desktop 4.33.x, but the change in 4.34 broke that check.If you know of a workaround, I would be happy to try it. However, I did a quick search and I don't think Docker Desktop offers an option to remove, hide, or disable the
nvidiaruntime. For now, I am unblocked by downgrading to 4.33.1.BTW, I am guessing other Win platforms are affected, but I only have the AMD 7600 iGPU to try an reproduce: