Third version of the same setup: Open WebUI talking to llama.cpp on an RTX 5060 Ti with 16 GB of VRAM. Earlier posts cover why llama.cpp rather than Ollama and the Qwen3.6-35B-A3B MoE I ran until now. Qwen 3.8 landed on 14 August, and this time the model is dense: Qwen3.8-27B, every parameter active on every token.
That should not fit. A dense model has to keep all its weights in VRAM or throughput collapses, which caps this card around 14B and is why I went MoE last time. Qwen changed the attention layout to get around it, so most layers hold a small fixed-size state rather than a growing KV cache, and context costs far less here than on a normal 27B. The model card has the breakdown. UD-IQ4_XS at 13.27 GB fits at 4-bit with 64K of context, using 15769 MiB of the card’s 16311 MiB.
Qwen 3.8 defaults to xhigh and will spend tens of thousands of tokens on a trivial question. Turning it down with reasoning_effort: none, which Unsloth’s model page lists as a level, fails every request:
Error: Jinja Exception: Unexpected reasoning effort none.
Supported types are xhigh (default), medium, and low.
--reasoning off is the actual switch, LLAMA_ARG_REASONING=off in the compose below.
64K runs out on tool-heavy chats
Every request in an agent session carries the schema for every enabled tool plus everything already read, so it ends like this:
request (71461 tokens) exceeds the available context size (65536 tokens)
Open WebUI’s context compaction summarises earlier turns once a chat passes a threshold, which makes that survivable, but it’s off by default and its 80000 default sits above the 64K served here. Set it under your context size, and expect it to fire earlier than you asked, since the backend double-counts cached tokens.
Leave CONTEXT_COMPACTION_MODEL unset. Pointed at a cloud model it would ship the earlier turns of every local conversation out to be summarised automatically, which defeats the point of running locally. TASK_MODEL is unset for the same reason.
Speed
About 24.9 tokens/second decoding, and 888 prompt-processing with 21.9 decoding on a 5000-token prompt.
Serious work still goes to a hosted frontier model, but the local one has moved past redacting and cleaning text, and with the GitHub MCP tools wired in it found a real bug in one of my repos.
Minimal two-container compose:
services:llama-server:# Nightly tag: GHCR publishes no images for llama.cpp's vX.Y.Z stable tags.image:ghcr.io/ggml-org/llama.cpp:server-cuda-b10573restart:unless-stopped# Qwen's instruct sampling, plus a five-minute idle unload. None of these four# have a LLAMA_ARG_ variable, so they only work as arguments.command:>--temp 0.7 --top-p 0.8 --min-p 0--sleep-idle-seconds 300environment:# Pin downloads to the mounted volume, or a container recreate re-fetches 13 GB.-LLAMA_CACHE=/root/.cache/llama.cpp-LLAMA_ARG_HF_REPO=unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS-LLAMA_ARG_N_GPU_LAYERS=99# all layers on GPU; dense, so no --n-cpu-moe to fall back on-LLAMA_ARG_CTX_SIZE=65536# 64K, affordable here because most layers keep no KV cache-LLAMA_ARG_N_PARALLEL=1# single user, one KV cache slot-LLAMA_ARG_FLASH_ATTN=1# big KV cache VRAM savings# q4_0 rather than q8_0: resident dense weights leave less room than the MoE,# which parked ~2 GB of idle experts in system RAM. Qwen tolerates it well.-LLAMA_ARG_CACHE_TYPE_K=q4_0-LLAMA_ARG_CACHE_TYPE_V=q4_0-LLAMA_ARG_MMPROJ_AUTO=0# it's a VL model; skip the ~0.93 GB vision projector-LLAMA_ARG_BATCH=2048# not LLAMA_ARG_BATCH_SIZE, which nothing reads-LLAMA_ARG_UBATCH=512-LLAMA_ARG_TOP_K=20# the only one of Qwen's four sampling values with a variable-LLAMA_ARG_JINJA=1# correct chat template and tool calling-LLAMA_ARG_REASONING=off# thinking off; reasoning_effort cannot do this-LLAMA_ARG_PORT=11434-LLAMA_ARG_HOST=0.0.0.0volumes:-./models:/root/.cache/llama.cppports:-"11434:11434"deploy:resources:reservations:devices:-driver:nvidiacount:allcapabilities:[gpu]open-webui:image:ghcr.io/open-webui/open-webui:v0.11.0restart:unless-stoppedports:-"3000:8080"environment:-ENABLE_OLLAMA_API=false-OPENAI_API_BASE_URLS=http://llama-server:11434/v1-OPENAI_API_KEYS=no-key-ENABLE_CONTEXT_COMPACTION=true# Has to sit under CTX_SIZE. The 80000 default is above the 64K served here,# so compaction would never fire before llama.cpp refused the request.-CONTEXT_COMPACTION_TOKEN_THRESHOLD=48000-CONTEXT_COMPACTION_RETENTION_PERCENTAGE=40# share of recent messages kept verbatimvolumes:-./open-webui:/app/backend/datadepends_on:-llama-server
Qwen also recommends presence_penalty=1.5 for instruct mode to curb repetition. That has no variable either, so it goes on the command: line or in Open WebUI’s per-model Advanced Params.
I’m trying to use Claude Code’s context more efficiently on long sessions. Anthropic published a much more thorough guide on the same topic the day before I wrote this: Maximizing the value of your Claude Code sessions, covering things like /clear between tasks, @-mentioning files instead of naming them, and keeping /model and /effort settled so you don’t bust the prompt cache. Worth reading first. Two things I’ve been doing on top of it: a Stop hook that warns me when a session’s context gets too long, and a small skill for handing off work before it’s even started.
The hook fires after every response, so it can catch a session getting bloated before I notice the agent acting slower or worse:
{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"jq -r '.transcript_path' | { read -r t; tail -n 200 \"$t\" | jq -s -r '[.[] | select(.type==\"assistant\" and .message.usage != null)] | last as $l | if $l == null then empty else (($l.message.usage.input_tokens // 0) + ($l.message.usage.cache_creation_input_tokens // 0) + ($l.message.usage.cache_read_input_tokens // 0)) as $tot | if $tot > 150000 then {systemMessage: (\"Context ~\" + (($tot/1000)|floor|tostring) + \"k tokens this turn (over 150k). Consider /compact, or say create handoff and start a fresh session.\")} else empty end end'; } 2>/dev/null || true"}]}]}}
The transcript is JSONL, one message per line, so tail -n 200 finds the most recent assistant turn without scanning the whole file. The hook is read-only, so it just parses that file locally with jq and never talks to the model. What it’s reading is accounting the API already produces on every turn, hook or not. usage splits input tokens by whether they came from the prompt cache: input_tokens for what got processed fresh, cache_creation_input_tokens for what got newly cached, cache_read_input_tokens for what got served from an earlier turn’s cache. All three are still part of the model’s actual context for that turn, so summing just input_tokens badly undercounts a long session, where most of the history is being read from cache rather than reprocessed fresh. Past 150k the hook prints a systemMessage. Claude Code shows that inline without sending it back to the model, so even the warning itself costs nothing (hooks reference).
A warning is only useful if there’s a good next step, and “just start a new chat” throws away real, unplanned-for work. Most of the time that means handing off work before it’s even started, not resuming something already underway. So I wrote kickoff, and kept the whole thing to one paragraph:
Write it as a prompt for the next session, not a document for a human. That agent has the same repo access you do, so don’t summarize or quote code; point at the files and areas that matter and let it read them itself. Include the actual task, why it matters, and whatever took real investigation to establish, especially constraints or dead ends already found. End on the first concrete action, specific enough to act on without re-deriving it.
I’ve been building Brute Slicer, a turn-based tile dungeon crawler in TypeScript. Sharing a browser game is just sending a link, and it opens on a phone without an app store. Godot and Unity export to the web too, but that ships the engine itself as WebAssembly, and Godot’s export requires WebGL 2.0 in the browser. The game state only advances when the player takes a turn, and the browser handles the animations, so there’s no per-frame loop for an engine to run.
rot.js is the standard roguelike toolkit and it covers the algorithm layer. I use it mostly for pathfinding, and it also handles dungeon generation, field of view and turn scheduling, with no dependencies of its own. Its README calls the project feature-complete, and the last release was a maintenance one in November 2024.
Phaser and Excalibur are both actively developed, and either one hands you a renderer, an input system and a camera. None of them has the RPG systems layer, so turn-based combat resolution, the inventory and equipment panels, save and load, and telegraphed enemy intent are all mine to write. I render SVG rather than canvas, so the camera that follows the player is mine too. rot.js never had one.
RPG-JS comes closest, with inventory, skills, save and load, and prebuilt GUI screens. It’s built for RPG Maker-style games with maps you draw in the Tiled editor, and this game generates its floors from a seed. The rest of what turns up is abandoned. The one npm package named for the combat half, turn-based-combat-framework, last published in November 2018. Malwoden, the newer take on rot.js, last released in January 2022. The rotjs topic on GitHub is mostly finished games rather than pieces you can pull out of one.
So everyone seems to write it again, which surprised me given how many browser roguelikes are out there. My guess is that these games get finished as monoliths and nobody goes back to extract the reusable half.
The route I’ve settled on keeps the game logic in a pure sim/ module with no React and no DOM in it, and a React interface reads from it and renders SVG. There are four runtime dependencies: react, rot-js, pure-rand for the seeded RNG, and zod to validate an imported save string. rot.js has its own RNG, but it’s a global singleton, and the sim keeps all of its state explicit, so a run replays identically from its seed and the whole thing unit-tests without a browser.
I wrote recently that every LLM tool call re-sends your entire conversation, and that prompt caching brings the re-read down to about a tenth of full price. There is a lot of buzz around Kimi K3 at the moment, so I tried it in opencode, paying per token through OpenRouter and expecting caching to absorb most of the re-reads. Thirteen prompts in, the billing page said otherwise.
Four rows from the OpenRouter activity log:
Input
Output
Cost
11,235
217
$0.0364
13,949
196
$0.0443
22,210
2,699
$0.107
45,512
1,194
$0.104
Subtract the first two rows and you get the input rate: 2,714 extra tokens for $0.0079, or $2.91 per million. That is K3’s list price, so nothing was cached.
Moonshot does cache automatically, with no cache_control breakpoints to place like Anthropic’s API. The discount just doesn’t survive the trip through OpenRouter, and you can confirm that before spending anything: every K3 endpoint on OpenRouter reports supports_implicit_caching: false.
Input ended up 73% of my spend, nearly all of it conversation I had already paid for once. Shorter sessions trim that, but every turn still re-reads its whole history at full price.
That leaves two ways out: go to Moonshot’s API directly, where the discount does apply, or pick a model whose caching survives the trip. Moonshot’s plans are behind a waitlist at the moment, so I took the second one. Run the same endpoints check against any candidate and look for a cache_read price.
K3 is good, and noticeably more concise than Opus, which likes to narrate what it is about to do before doing it. I would still reach for it on a hard problem.
I only run any of this when Claude’s rate limits hit, so this is a first pass rather than a verdict. The cheapest thing that has held up so far is GPT-5.6 Terra, which caches and is on sale at the moment:
"model":"openrouter/openai/gpt-5.6-terra:online"
Cache reads run a tenth of the input price. Over a long session that is the number that decides the bill.
When it comes to LLM chats, I know long context costs more, so I try to keep mine short. What I couldn’t explain was why a single prompt still billed for many times more tokens than the context window holds.
Every time an AI agent calls a tool, it re-sends the entire conversation. The Messages API is stateless, so the model remembers nothing between requests. When it calls a tool, the request ends, your client runs the tool, appends the result to the history, and POSTs the whole thing back as a new request. You pay input tokens to remind it every time.
So a chat near the 1M-token context limit that makes 20 tool calls pays for that 1M about 20 times over, turning it into 20M billed tokens and around $20 on today’s frontier models. That is 20 times the context window, from a single prompt.
Prompt caching helps keep the cost down. Normally the model reprocesses the whole conversation on every request, but a cache lets it reuse the unchanged prefix, so re-sending that part costs about a tenth of full price. You pay a small premium to store it the first time, then read it back cheap after, the same way OpenAI (automatic caching) and Gemini (implicit caching) do it. The reuse only holds while the start of the prompt stays byte-for-byte identical, so a changing timestamp in your system prompt breaks the cache and puts you back at full price.
The Sonoff SNZB-05P seems like the best Zigbee leak sensor right now. It’s around $20, and its snap-on WLDC200 sensing cable detects water anywhere along the cord, so it isn’t just watching a single spot the water can spread around. I have them in appliance drip pans and under sinks, all feeding one Home Assistant automation that alerts my phone when a sensor trips and sends an all-clear when it dries.
automations.yaml
-id:leak_notificationsalias:Leak Notificationstriggers:# Wet: the state must hold 15s to filter momentary blips, and not_from# stops a sensor coming back from unavailable from re-alerting.-trigger:stateentity_id:-binary_sensor.kitchen_sink_leak_sensor-binary_sensor.water_heater_leak_sensornot_from:-unavailable-unknownto:'on'for:seconds:15id:wet# Dry: same sensors, opposite transition.-trigger:stateentity_id:-binary_sensor.kitchen_sink_leak_sensor-binary_sensor.water_heater_leak_sensorfrom:'on'to:'off'id:dryactions:-if:-condition:triggerid:wetthen:-action:notify.mobile_app_your_phonedata:title:Leak Detectedmessage:'Device:{{trigger.to_state.attributes.friendly_name|default(trigger.entity_id)}}hasdetectedwater.'data:# Both branches share a per-sensor tag, so the dry notification# overwrites the wet one and your phone shows a single# up-to-date entry per sensor instead of a stack.tag:leak_{{ trigger.entity_id }}# Android-specific: alarm_stream plays at alarm volume even# during Do Not Disturb (a leak at 3am is worth waking up for),# and priority high + ttl 0 deliver it immediately.priority:highttl:0channel:alarm_streamelse:# The all-clear is a normal-priority push.-action:notify.mobile_app_your_phonedata:title:Leak Drymessage:'Device:{{trigger.to_state.attributes.friendly_name|default(trigger.entity_id)}}isdry.'data:tag:leak_{{ trigger.entity_id }}# Parallel so one burst pipe tripping sensors in two rooms alerts for both.mode:parallel
To adapt
Swap in your own sensors under both entity_id lists and point the notify actions at your phone’s service. The Android keys are documented in the critical notifications docs and the tag behavior in the basic notification docs. The iOS equivalent of the alarm-stream keys is a critical push:
The pitch for Aurora and AlloyDB is that they take storage off your plate and let you scale compute on its own. Storage grows automatically with no downtime, up to 128 TiB, and is replicated six ways across three availability zones, so a write survives losing an entire zone and you never plan capacity or schedule a resize. AlloyDB takes the same approach on its own disaggregated storage and adds a columnar engine for analytics. CloudNativePG doesn’t work that way. It runs classic shared-nothing streaming replication, where each instance is a full Postgres node with its own PersistentVolumeClaim, and durability comes from replicating between instances rather than from a shared storage layer.
apiVersion:postgresql.cnpg.io/v1kind:Clustermetadata:name:pgspec:instances:3# one primary, two streaming replicasstorage:storageClass:local-nvme# benchmark this before you trust itsize:200GiwalStorage:# WAL on its own volumestorageClass:local-nvmesize:40GireplicationSlots:highAvailability:enabled:truepostgresql:parameters:shared_buffers:"4GB"# tuning is yours nowsynchronous:# opt in for RPO=0 (replication is async by default)method:any# any = quorum-basednumber:1# standbys that must confirm each commit
That model works, but the volume is now your job. The docs recommend local NVMe for serious workloads and tell you to benchmark with fio and pgbench first, because network-attached storage can wreck Postgres latency. And there’s no auto-scaling, so you size the PVC up front and grow it later only if your CSI driver supports expansion.
Each instance keeps its own copy of the data, so with local volumes a failed node takes its data with it and the operator re-creates that instance by cloning a fresh copy from the primary, a slow rebuild for a large database. That only protects against a zone outage if the replicas are in other zones, which takes pod anti-affinity and topologySpreadConstraints to arrange, with the usual advice to run nodes in multiples of three, one per zone. The operator promotes a replica automatically when the primary fails, but replication is asynchronous by default, so that promotion can drop the last few committed transactions. Synchronous replication (method: any quorum, above) gets you back to RPO=0, and it makes the CAP tradeoff explicit. With the default dataDurability: required, writes pause when the operator can’t reach enough synchronous standbys, so a bad enough zone outage costs availability to preserve consistency. Aurora and AlloyDB make that same call at the storage layer and keep taking writes through an AZ failure. Cross-region means running a separate replica cluster fed by streaming or WAL shipping, with no automatic failover between clusters.
The operational pieces are all yours too. Continuous backup and point-in-time recovery go to object storage through the barman-cloud plugin, but you configure it, set the retention, and test that a restore actually works. Connection pooling is a Pooler resource running PgBouncer that you deploy and size. Monitoring is Prometheus metrics the operator exposes, wired into whatever stack you already run. Minor-version upgrades are rolling pod restarts you trigger.
So is it worth it? If you’re paying for Aurora mainly to avoid thinking about Postgres, self-hosting probably isn’t worth it. If you already have the Kubernetes and Postgres chops, live in GitOps, want off the managed-service markup, or need something the managed services won’t give you (specific extensions, exact versions, multi-cloud portability, no lock-in), it looks worth a serious trial. Either way I’d start on something smaller than a tier-1 database and prove out storage, backups, and failover first.
A few months ago I wrote about switching Open WebUI from Ollama to llama.cpp for Qwen 3.5. I’m still using the same RTX 5060 Ti with 16 GB of VRAM, but I swapped the dense 9B for Qwen3.6-35B-A3B, a Mixture-of-Experts (MoE) model. A 35B model usually wouldn’t fit on a 16 GB card without MoE.
A dense model uses every parameter for every token, so the weights have to fit in VRAM or throughput tanks. On this card that capped me around 14B. MoE models have many expert sub-networks but only use a few per token. Qwen3.6-35B-A3B is 35B total but only ~3B active. The idle experts can sit in system RAM instead of VRAM, pulled onto the GPU when needed for a small speed hit. llama.cpp does this with --n-cpu-moe, which keeps the top N layers’ experts on the CPU.
Staying within the 16 GB budget took some more tuning. I set LLAMA_ARG_N_CPU_MOE=8, which freed about 2 GB of VRAM for context. Raise it if the model OOMs on load, lower it if you have VRAM to spare. I used a dynamic UD-Q3_K_M quant (about 15 GB) and lowered BATCH_SIZE and UBATCH from the 9B’s values to leave room for a 64K KV cache. Flash attention and a q8_0 KV cache save the rest, same as before.
Turning thinking mode off took two flags. LLAMA_ARG_THINK_BUDGET=0 (reasoning-budget 0) alone didn’t stop it, a known issue on the hybrid Qwen models. I was still watching it think in circles for several minutes without getting anything done. The other was enable_thinking=false in the chat template (LLAMA_ARG_CHAT_TEMPLATE_KWARGS), with jinja on. I also set the sampling to Qwen3.6’s non-thinking defaults (temp 0.7, top_k 20, top_p 0.8).
For serious work I still use a hosted frontier model, but this is handy for local jobs like redacting or cleaning text before it goes to the cloud.
Minimal two-container compose to get mostly set up:
services:llama-server:image:ghcr.io/ggml-org/llama.cpp:server-cuda-b9592restart:unless-stoppedenvironment:# Pin the model cache to the mounted volume so a container recreate# doesn't re-download the weights.-LLAMA_CACHE=/root/.cache/llama.cpp# Auto-downloads from HuggingFace on first run.-LLAMA_ARG_HF_REPO=unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q3_K_M-LLAMA_ARG_N_GPU_LAYERS=99# all layers to GPU...-LLAMA_ARG_N_CPU_MOE=8# ...except the top 8 layers' experts, kept in system RAM-LLAMA_ARG_CTX_SIZE=65536# 64K context, realistic for a 35B MoE in 16 GB-LLAMA_ARG_N_PARALLEL=1# single user, one KV cache slot-LLAMA_ARG_FLASH_ATTN=1# big KV cache VRAM savings-LLAMA_ARG_CACHE_TYPE_K=q8_0# halve KV cache memory vs FP16-LLAMA_ARG_CACHE_TYPE_V=q8_0-LLAMA_ARG_BATCH_SIZE=2048# lowered from 4096 to leave VRAM for the bigger model-LLAMA_ARG_UBATCH=512# lowered from 2048 for the same reason-LLAMA_ARG_JINJA=1# correct chat template + tool calling# Non-thinking mode. On the Qwen3 hybrid models, reasoning-budget 0 alone kept# emitting think blocks; enable_thinking=false is what stopped it, so set both.-'LLAMA_ARG_CHAT_TEMPLATE_KWARGS={"enable_thinking":false}'-LLAMA_ARG_THINK_BUDGET=0# reasoning-budget 0-LLAMA_ARG_TEMP=0.7# Qwen3.6 non-thinking sampling defaults-LLAMA_ARG_TOP_K=20-LLAMA_ARG_TOP_P=0.8-LLAMA_ARG_MIN_P=0-LLAMA_ARG_PORT=11434-LLAMA_ARG_HOST=0.0.0.0volumes:-./models:/root/.cache/llama.cppports:-"11434:11434"deploy:resources:reservations:devices:-driver:nvidiacount:allcapabilities:[gpu]open-webui:image:ghcr.io/open-webui/open-webui:v0.9.6restart:unless-stoppedports:-"3000:8080"environment:-ENABLE_OLLAMA_API=false-OPENAI_API_BASE_URLS=http://llama-server:11434/v1-OPENAI_API_KEYS=no-keyvolumes:-./open-webui:/app/backend/datadepends_on:-llama-server
I was picking hardware for an OPNsense router and needed a real PCIe slot, which most cheap low-power boxes don’t have. A slot lets you add a network card with SFP+ or more ports, and swap it later for something else (like a GPU).
The thin clients I wrote about last time don’t have one. The Wyse 5070 and OptiPlex 3000 both have an M.2 A/E key slot, good for a 2.5GbE adapter or a Coral TPU but not an SFP+ or a quad-port NIC.
A few corporate thin clients do have a full-size slot:
Model
CPU
PCIe slot
Used price
HP t730
AMD RX-427BB (4C)
half-height x16 mechanical, x8 electrical
~$100-120
HP t740
AMD Ryzen V1756B (4C/8T)
half-height x16 mechanical, x8 electrical, Gen3
~$150-170
Dell Wyse 5070 Extended
Celeron J4105 / Pentium J5005
half-height slot in the thicker “Extended” chassis
~$90-130
The HP t730 and t740 are the well-known ones, both popular pfSense/OPNsense boxes with a half-height x16-mechanical, x8-electrical slot. The Wyse 5070 comes in two sizes. The slim one has only the M.2 slot. The thicker Extended chassis adds a half-height PCIe slot.
The slot is half-height, so you’ll want a low-profile card, and its power is limited (roughly 35W on the t740), enough for a NIC but only a low-power GPU.
The other route is a Topton or CWWK mini PC, which usually skips the slot and solders on the NICs instead, often four 2.5GbE ports or a couple of SFP+ cages. Handy if those ports are what you need, but you’re stuck with them, with no way to swap in a specific card.
I run Prometheus, Alertmanager, and Grafana on a single mini PC, which has an obvious blind spot: if that box goes down, nothing can alert me, because the thing that sends alerts is the thing that’s down. Prometheus once sat dead for two days before I noticed, and only because a dashboard had gone blank.
The fix is a dead man’s switch: an alert that fires constantly, routed to an outside service that complains when it stops arriving. Every other alert fires when something breaks; this one fires all the time, and silence is the failure signal.
kube-prometheus-stack ships the same alert under the same name. Then route that one alert away from Telegram and into a webhook:
routes:-matchers:-alertname = "Watchdog"receiver:deadmansswitchgroup_wait:0srepeat_interval:5m# re-ping every 5 minutesreceivers:-name:deadmansswitchwebhook_configs:-url:https://hc-ping.com/<uuid>send_resolved:false
The webhook target is a healthchecks.io check. Alertmanager pings it every five minutes while the pipeline is healthy; I set the check to a ~10 minute period with a ~5 minute grace, so a real outage pages me within about 15 minutes, from a service that isn’t on my network.
Two details that bit me: the ping URL lives in my git-ignored alertmanager.yml (low-sensitivity, like the Telegram chat ID), and send_resolved: false so the “resolved” call doesn’t muddy the heartbeat.
The same trick works for any scheduled job: have the script curl a healthchecks.io ping URL on success, and let the absence page you.