As I said in the title, I am trying to have GPU support for my Ollama docker container. While I am a software developer, I am NOT a unix expert. I've been through numerous guides and posts, but the result is always the same.
I just installed a new Nvidia RTX 3060 card, and the server saw it immediately:
NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
Linux hal 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux
I am getting this error when I launch Ollama:
In my research, it seems like there an incompatibility with the version of cuda I am using. I've tried a dozen ways to try to correct this. I don't know if this is a clue or not, but for the life of me I can't get "invidia-smi" to work. I did update /etc/apt/sources.list to the following:
Any help would be GREATLY appreciated!
I just installed a new Nvidia RTX 3060 card, and the server saw it immediately:
NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
Linux hal 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux
I am getting this error when I launch Ollama:
Code:
ollama-webui-1 | time=2024-11-04T21:24:13.944Z level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths=[]ollama-webui-1 | time=2024-11-04T21:24:13.944Z level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=libcudart.so*ollama-webui-1 | time=2024-11-04T21:24:13.944Z level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /app/backend/libcudart.so* /usr/local/lib/ollama/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"ollama-webui-1 | time=2024-11-04T21:24:13.946Z level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.4.99 /usr/local/lib/ollama/libcudart.so.11.3.109]"ollama-webui-1 | cudaSetDevice err: 35ollama-webui-1 | time=2024-11-04T21:24:13.947Z level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.4.99: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama"ollama-webui-1 | [b]cudaSetDevice err: 35[/b]ollama-webui-1 | time=2024-11-04T21:24:13.947Z level=DEBUG source=gpu.go:578 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.11.3.109: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama"ollama-webui-1 | time=2024-11-04T21:24:13.947Z level=DEBUG source=amd_linux.go:416 msg="amdgpu driver not detected /sys/module/amdgpu"ollama-webui-1 | time=2024-11-04T21:24:13.947Z level=INFO source=gpu.go:384 msg="no compatible GPUs were discovered"ollama-webui-1 | time=2024-11-04T21:24:13.947Z level=INFO source=types.go:123 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="62.8 GiB" available="58.3 GiB"Code:
deb https://deb.debian.org/debian bookworm main contrib non-freedeb https://deb.debian.org/debian bookworm-updates main contrib non-freedeb https://deb.debian.org/debian-security bookworm-security main contrib non-freeStatistics: Posted by adamcodes716 — 2024-11-04 22:59 — Replies 7 — Views 160