Running Qwen 3.8 27B on a 16 GB GPU
Third version of the same setup: Open WebUI talking to llama.cpp on an RTX 5060 Ti with 16 GB of VRAM. Earlier posts cover why llama.cpp rather than Ollama and the Qwen3.6-35B-A3B MoE I ran until now. Qwen 3.8 landed on 14 August, and this time the model is dense: Qwen3.8-27B, every parameter active on every token.
That should not fit. A dense model has to keep all its weights in VRAM or throughput collapses, which caps this card around 14B and is why I went MoE last time. Qwen changed the attention layout to get around it, so most layers hold a small fixed-size state rather than a growing KV cache, and context costs far less here than on a normal 27B. The model card has the breakdown. UD-IQ4_XS at 13.27 GB fits at 4-bit with 64K of context, using 15769 MiB of the card’s 16311 MiB.
llama.cpp builds before roughly b10450 produce fluent garbage from this model on CUDA rather than failing to load.
reasoning_effort cannot turn thinking off
Qwen 3.8 defaults to xhigh and will spend tens of thousands of tokens on a trivial question. Turning it down with reasoning_effort: none, which Unsloth’s model page lists as a level, fails every request:
Error: Jinja Exception: Unexpected reasoning effort none.
Supported types are xhigh (default), medium, and low.
--reasoning off is the actual switch, LLAMA_ARG_REASONING=off in the compose below.
64K runs out on tool-heavy chats
Every request in an agent session carries the schema for every enabled tool plus everything already read, so it ends like this:
request (71461 tokens) exceeds the available context size (65536 tokens)
Open WebUI’s context compaction summarises earlier turns once a chat passes a threshold, which makes that survivable, but it’s off by default and its 80000 default sits above the 64K served here. Set it under your context size, and expect it to fire earlier than you asked, since the backend double-counts cached tokens.
Leave CONTEXT_COMPACTION_MODEL unset. Pointed at a cloud model it would ship the earlier turns of every local conversation out to be summarised automatically, which defeats the point of running locally. TASK_MODEL is unset for the same reason.
Speed
About 24.9 tokens/second decoding, and 888 prompt-processing with 21.9 decoding on a 5000-token prompt.
Serious work still goes to a hosted frontier model, but the local one has moved past redacting and cleaning text, and with the GitHub MCP tools wired in it found a real bug in one of my repos.
Minimal two-container compose:
services:
llama-server:
# Nightly tag: GHCR publishes no images for llama.cpp's vX.Y.Z stable tags.
image: ghcr.io/ggml-org/llama.cpp:server-cuda-b10573
restart: unless-stopped
# Qwen's instruct sampling, plus a five-minute idle unload. None of these four
# have a LLAMA_ARG_ variable, so they only work as arguments.
command: >
--temp 0.7 --top-p 0.8 --min-p 0
--sleep-idle-seconds 300
environment:
# Pin downloads to the mounted volume, or a container recreate re-fetches 13 GB.
- LLAMA_CACHE=/root/.cache/llama.cpp
- LLAMA_ARG_HF_REPO=unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS
- LLAMA_ARG_N_GPU_LAYERS=99 # all layers on GPU; dense, so no --n-cpu-moe to fall back on
- LLAMA_ARG_CTX_SIZE=65536 # 64K, affordable here because most layers keep no KV cache
- LLAMA_ARG_N_PARALLEL=1 # single user, one KV cache slot
- LLAMA_ARG_FLASH_ATTN=1 # big KV cache VRAM savings
# q4_0 rather than q8_0: resident dense weights leave less room than the MoE,
# which parked ~2 GB of idle experts in system RAM. Qwen tolerates it well.
- LLAMA_ARG_CACHE_TYPE_K=q4_0
- LLAMA_ARG_CACHE_TYPE_V=q4_0
- LLAMA_ARG_MMPROJ_AUTO=0 # it's a VL model; skip the ~0.93 GB vision projector
- LLAMA_ARG_BATCH=2048 # not LLAMA_ARG_BATCH_SIZE, which nothing reads
- LLAMA_ARG_UBATCH=512
- LLAMA_ARG_TOP_K=20 # the only one of Qwen's four sampling values with a variable
- LLAMA_ARG_JINJA=1 # correct chat template and tool calling
- LLAMA_ARG_REASONING=off # thinking off; reasoning_effort cannot do this
- LLAMA_ARG_PORT=11434
- LLAMA_ARG_HOST=0.0.0.0
volumes:
- ./models:/root/.cache/llama.cpp
ports:
- "11434:11434"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:v0.11.0
restart: unless-stopped
ports:
- "3000:8080"
environment:
- ENABLE_OLLAMA_API=false
- OPENAI_API_BASE_URLS=http://llama-server:11434/v1
- OPENAI_API_KEYS=no-key
- ENABLE_CONTEXT_COMPACTION=true
# Has to sit under CTX_SIZE. The 80000 default is above the 64K served here,
# so compaction would never fire before llama.cpp refused the request.
- CONTEXT_COMPACTION_TOKEN_THRESHOLD=48000
- CONTEXT_COMPACTION_RETENTION_PERCENTAGE=40 # share of recent messages kept verbatim
volumes:
- ./open-webui:/app/backend/data
depends_on:
- llama-server
Qwen also recommends presence_penalty=1.5 for instruct mode to curb repetition. That has no variable either, so it goes on the command: line or in Open WebUI’s per-model Advanced Params.