Switching Open WebUI from Ollama to llama.cpp for Qwen 3.5
Update 2026-08-23: I have since swapped Qwen 3.5 for Qwen3.6-35B-A3B, a Mixture-of-Experts model, and then for Qwen3.8-27B, a dense model, both on the same 16 GB card. I now recommend the setup in the Qwen 3.8 post instead.
I finally got to a local LLM setup that feels pretty usable within a 16 GB VRAM constraint, at around 40 tokens/second. It’s Open WebUI, which as far as I can tell is still the best open source chat interface for local models, running Qwen3.5-9B at Q8_0 on a Ryzen 5 5600 box with an RTX 5060 Ti 16 GB card.
I started with Ollama because it’s the popular default. For Qwen 3.5, though, Ollama had open issues at the time with much slower inference than llama.cpp on the same model, long stalls on later turns in a conversation, and broken /no_think handling for Qwen. llama.cpp’s llama-server exposes an OpenAI-compatible API, so switching Open WebUI over to it was mostly a config swap.
The 9B feels much better than I expected for its size, and it’s the first local setup I’ve had on this machine that felt worth keeping around. For serious work I still use hosted frontier models, but this is useful for narrower jobs like redacting or cleaning text before it goes to a cloud model.
Qwen 3.5 defaults to thinking, which for my use was usually slower, often added 30-60 seconds, and sometimes got stuck in long loops without noticeably improving the answer. LLAMA_ARG_REASONING=off (--reasoning off) turns it off, with LLAMA_ARG_JINJA=1 so llama.cpp uses the chat template from the GGUF.
The 9B weights take roughly 9 GB, and the rest of the tuning spends the remaining VRAM on context. LLAMA_ARG_N_PARALLEL=1, LLAMA_ARG_FLASH_ATTN=1, and a q8_0 KV cache bring the whole setup to around 14 GB without spilling into system memory.
The commented two-container compose file: