Grounding a coding agent in reality
A couple of popular coding-agent skills got less useful as the models improved. The caveman skill advertised a 65% token cut by writing in clipped fragments, but JetBrains measured 8.5% on real agentic work, and a common complaint is that the style wears off after a reply (#764, #1104). I’d skip Superpowers too. The models split a big prompt into a todo list and farm pieces out to subagents on their own, which covers most of what it bundled. The plan files it leaves behind are too much slop to be worth committing.
I still keep a make check target and one line in AGENTS.md telling the agent to run it before committing. It gives the agent something to check its work against instead of stopping once a change looks done. Mine runs what CI runs, so a green run locally means a green PR:
check:
npx --yes @j178/prek run --all-files
$(MAKE) lint
$(MAKE) typecheck
$(MAKE) test
Without a check you end up as the verification loop, as Anthropic’s best practices page puts it.
For a web app, that check can be a screenshot the agent takes itself. I’ve got a Makefile target that uses Playwright to open the app and save a PNG, and a line in AGENTS.md telling the agent to run it after any visual change and look at the result. Claude Code can drive a real browser through the Chrome extension, and Codex’s desktop app grew a native browser in April that screenshots before and after a patch. I stick with the Makefile target, since it works from my phone through Claude Code on the web, where there’s no desktop browser to attach to.
The agent finds plenty of visual bugs this way, but it still shows me changes that look bad without flagging them, and it misses outright mistakes. On the roguelike I’m building, four review passes looked at the town and none noticed the mushrooms drawn on top of the trees. LRR-Bench and Mind the Gap both find vision-language models near random at telling whether one object is left of, above, or rotated relative to another.
The same game’s worst recurring bug was the agent picking the wrong tile, like drawing a market stall with a signpost. The DawnLike sheets have no labels, so it was guessing from pixels until I gave it the tile names from a community atlas.
Every model in Chroma’s context rot study got worse as its input grew, so I keep to one task per chat and put whatever the next chat needs in files. That’s obvious to any programmer and probably isn’t to a lot of non-programmers, and the memory and project features in desktop AI apps look like an attempt to do it for them. AGENTS.md holds what the agent must know immediately, like where the design doc is and that make check runs before a commit. A design doc holds aesthetic rules like not overusing loud colors, and a plan doc holds what’s built and what’s next. When a chat produces a long document worth keeping, it becomes the design doc for the next one.