Paul's Programming Notes

Switching my local LLM to Qwen 3.6, a 35B Mixture-of-Experts model, on a 16 GB GPU

Update 2026-08-23: I have since swapped this for Qwen3.8-27B, a dense 27B that fits the same 16 GB card. I now recommend the setup in that post instead.

I wanted a bigger model than the Qwen3.5-9B I was running on my 16 GB RTX 5060 Ti (the setup). A larger dense model is hard to fit, since it uses every parameter for every token and any weights outside VRAM tank throughput. A Mixture-of-Experts (MoE) model uses only a few of its expert sub-networks per token, so the idle ones can sit in system RAM instead. Qwen3.6-35B-A3B is 35B total with about 3B active per token. Even quantized to 3-bit (Unsloth’s UD-Q3_K_M), its weights are about 15 GB, so some experts have to move off the card to leave room for context.

llama.cpp does this with --n-cpu-moe, which keeps the expert weights of the first N layers in system RAM for a small speed hit. LLAMA_ARG_N_CPU_MOE=8 freed about 2 GB of VRAM, enough for 64K of context with flash attention and a q8_0 KV cache. Raise it if the model OOMs on load, and lower it if you have VRAM to spare.

For serious work I still use a hosted frontier model, but this is handy for local jobs like redacting or cleaning text before it goes to the cloud. The compose file runs it behind Open WebUI:

services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda-v0.5.0
    restart: unless-stopped
    environment:
      # Pin downloads to the mounted volume, or a container recreate re-fetches 15 GB.
      - LLAMA_CACHE=/root/.cache/llama.cpp
      - LLAMA_ARG_HF_REPO=unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q3_K_M
      - LLAMA_ARG_N_GPU_LAYERS=99      # all layers to GPU...
      - LLAMA_ARG_N_CPU_MOE=8          # ...except the first 8 layers' experts, kept in system RAM
      - LLAMA_ARG_CTX_SIZE=65536
      - LLAMA_ARG_N_PARALLEL=1         # single user, one KV cache slot
      - LLAMA_ARG_FLASH_ATTN=1         # big KV cache VRAM savings
      - LLAMA_ARG_CACHE_TYPE_K=q8_0    # halve KV cache memory vs FP16
      - LLAMA_ARG_CACHE_TYPE_V=q8_0
      - LLAMA_ARG_JINJA=1              # correct chat template + tool calling
      - LLAMA_ARG_REASONING=off        # thinking off, see below
      - LLAMA_ARG_TEMPERATURE=0.7      # Qwen3.6 non-thinking sampling defaults
      - LLAMA_ARG_TOP_K=20
      - LLAMA_ARG_TOP_P=0.8
      - LLAMA_ARG_MIN_P=0
      - LLAMA_ARG_PORT=11434
      - LLAMA_ARG_HOST=0.0.0.0
    volumes:
      - ./models:/root/.cache/llama.cpp
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:v0.11.4
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - ENABLE_OLLAMA_API=false
      - OPENAI_API_BASE_URLS=http://llama-server:11434/v1
      - OPENAI_API_KEYS=no-key
    volumes:
      - ./open-webui:/app/backend/data
    depends_on:
      - llama-server

The thinking switch

--reasoning off (LLAMA_ARG_REASONING=off above, with LLAMA_ARG_JINJA=1) turns thinking off. LLAMA_ARG_THINK_BUDGET=0 alone didn’t stop it, and I watched it think in circles for several minutes without getting anything done. Passing enable_thinking=false through LLAMA_ARG_CHAT_TEMPLATE_KWARGS also works, but llama.cpp logs a deprecation warning for it.

dockerllm