This video shows how you can make use of your Intel Arc GPU to run local llms like deepseek with ollama in your Windows PC.
If you have gotten ollama from ollama.com and you have Intel Arc gpu, you might have noticed that it doesnt use the gpu but the model is loaded on the RAM and uses the CPU. which renders the llms model inefficient or at least you dont get the speed that you could If it used the GPU.
I demonstrate 3 ways to run the deepseek-r1:7b
1. With ollama downloaded from ollama.com
2. ipex-llm based ollama standalone zip
3. ipex-llm based ollama with dockerized container
Out of these 3, #2 and #3 uses GPU. Among them , for me #3 gave the best result.
Here is the docker command for #3
# pull intel ipex llm image
docker pull intelanalytics/ipex-llm-inference-cpp-xpu:latest
# create docker container of ipex-llm based ollama
docker run -itd –device=/dev/dri –privileged -v C:/Users/YOUR_USER/ollama/models:/models -v /usr/lib/wsl:/usr/lib/wsl -e no_proxy=localhost,127.0.0.1 -p 11434:11434 -e OLLAMA_HOST=0.0.0.0:11434 -e OLLAMA_NUM_GPU=999 -e ZES_ENABLE_SYSMAN=1 -e SYCL_CACHE_PERSISTENT=1 –memory=”32G” –name=ipex-llm-inference-cpp-xpu-container3 -e bench_model=”mistral-7b-v0.1.Q4_0.gguf” -e DEVICE=0 –shm-size=”16g” intelanalytics/ipex-llm-inference-cpp-xpu:latest
#go into container
docker exec -it ipex-llm-inference-cpp-xpu-container4 /bin/bash
#and then below to start ollama
cd /llm/scripts/
source ipex-llm-init –gpu –device $DEVICE
bash start-ollama.sh
# to start using
cd ../ollama
#and here you can execute
./ollama run –verbose deepseek-r1:7b
Please let me know if you have any questions and Ill try to answer them when I can. Also, let me know if you want me to cover any topic in detail that is mentioned in this video.
Link to ipex-llm githug page:
📢 Don’t forget to LIKE, SUBSCRIBE, and hit the 🔔 to stay updated on more content!
#Ollama #IntelArc #DeepSeek #AI #LocalLLMs #MachineLearning #EdgeAI #TechTips #GPU #OllamaAI #DIYAI #windows11 #A770
source
Disclaimer
The content published on this page is sourced from external platforms, including YouTube. We do not own or claim any rights to the videos embedded here. All videos remain the property of their respective creators and are shared for informational and educational purposes only.
If you are the copyright owner of any video and wish to have it removed, please contact us, and we will take the necessary action promptly.
