Switching my local LLM to Qwen 3.6, a 35B Mixture-of-Experts model, on a 16 GB GPU
A few months ago I wrote about switching Open WebUI from Ollama to llama.cpp for Qwen 3.5. I’m still using the same RTX 5060 Ti with 16 GB of VRAM, but I swapped the dense 9B for Qwen3.6-35B-A3B, a Mixture-of-Experts (MoE) model. A 35B model usually wouldn’t fit on a 16 GB card without MoE.
A dense model uses every parameter for every token, so the weights have to fit in VRAM or throughput tanks. On this card that capped me around 14B. MoE models have many expert sub-networks but only use a few per token. Qwen3.6-35B-A3B is 35B total but only ~3B active. The idle experts can sit in system RAM instead of VRAM, pulled onto the GPU when needed for a small speed hit. llama.cpp does this with --n-cpu-moe, which keeps the top N layers’ experts on the CPU.
Staying within the 16 GB budget took some more tuning. I set LLAMA_ARG_N_CPU_MOE=8, which freed about 2 GB of VRAM for context. Raise it if the model OOMs on load, lower it if you have VRAM to spare. I used a dynamic UD-Q3_K_M quant (about 15 GB) and lowered BATCH_SIZE and UBATCH from the 9B’s values to leave room for a 64K KV cache. Flash attention and a q8_0 KV cache save the rest, same as before.
Turning thinking mode off took two flags. LLAMA_ARG_THINK_BUDGET=0 (reasoning-budget 0) alone didn’t stop it, a known issue on the hybrid Qwen models. I was still watching it think in circles for several minutes without getting anything done. The other was enable_thinking=false in the chat template (LLAMA_ARG_CHAT_TEMPLATE_KWARGS), with jinja on. I also set the sampling to Qwen3.6’s non-thinking defaults (temp 0.7, top_k 20, top_p 0.8).
For serious work I still use a hosted frontier model, but this is handy for local jobs like redacting or cleaning text before it goes to the cloud.
Minimal two-container compose to get mostly set up:
services:
llama-server:
image: ghcr.io/ggml-org/llama.cpp:server-cuda-b9592
restart: unless-stopped
environment:
# Pin the model cache to the mounted volume so a container recreate
# doesn't re-download the weights.
- LLAMA_CACHE=/root/.cache/llama.cpp
# Auto-downloads from HuggingFace on first run.
- LLAMA_ARG_HF_REPO=unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q3_K_M
- LLAMA_ARG_N_GPU_LAYERS=99 # all layers to GPU...
- LLAMA_ARG_N_CPU_MOE=8 # ...except the top 8 layers' experts, kept in system RAM
- LLAMA_ARG_CTX_SIZE=65536 # 64K context, realistic for a 35B MoE in 16 GB
- LLAMA_ARG_N_PARALLEL=1 # single user, one KV cache slot
- LLAMA_ARG_FLASH_ATTN=1 # big KV cache VRAM savings
- LLAMA_ARG_CACHE_TYPE_K=q8_0 # halve KV cache memory vs FP16
- LLAMA_ARG_CACHE_TYPE_V=q8_0
- LLAMA_ARG_BATCH_SIZE=2048 # lowered from 4096 to leave VRAM for the bigger model
- LLAMA_ARG_UBATCH=512 # lowered from 2048 for the same reason
- LLAMA_ARG_JINJA=1 # correct chat template + tool calling
# Non-thinking mode. On the Qwen3 hybrid models, reasoning-budget 0 alone kept
# emitting think blocks; enable_thinking=false is what stopped it, so set both.
- 'LLAMA_ARG_CHAT_TEMPLATE_KWARGS={"enable_thinking":false}'
- LLAMA_ARG_THINK_BUDGET=0 # reasoning-budget 0
- LLAMA_ARG_TEMP=0.7 # Qwen3.6 non-thinking sampling defaults
- LLAMA_ARG_TOP_K=20
- LLAMA_ARG_TOP_P=0.8
- LLAMA_ARG_MIN_P=0
- LLAMA_ARG_PORT=11434
- LLAMA_ARG_HOST=0.0.0.0
volumes:
- ./models:/root/.cache/llama.cpp
ports:
- "11434:11434"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:v0.9.6
restart: unless-stopped
ports:
- "3000:8080"
environment:
- ENABLE_OLLAMA_API=false
- OPENAI_API_BASE_URLS=http://llama-server:11434/v1
- OPENAI_API_KEYS=no-key
volumes:
- ./open-webui:/app/backend/data
depends_on:
- llama-server