The screenshot above is part of the file my agent reads at the start of every session. That section is a list of the things I’ve had to correct it on most often, and each rule carries a note about when it went in and how many times I’d said it first. “Added after 3 repeats.” “Added after 4 repeats.” Those notes are the best data I have on what it’s actually like to hand real work to one of these things.
The list started in June, when I had the agent read back through fifty of our sessions, pull out every place I’d told it no, and group what it found. Since then a loop keeps it going. A recurring report reads my recent sessions and suggests changes, and a hook applies the ones that pass. Most new rules now show up without me writing them.
What surprised me is what isn’t on the list. Almost nothing there is about the model being unable to do something. It can write the code, build the page, and wire up the deploy. The failures are about when it believes itself.
It tells me something doesn’t exist without checking (at one point, a newer version of the model it was running on). It reports a push as done because the command came back clean, when the device on the other end hadn’t picked it up and a pull on my server had quietly failed. It hands me a layout it never looked at, with text running off the page. It reads a photo of a circuit board, calls two resistors something they aren’t, and builds a wiring plan on the mistake. And once, after I said stop, it explained itself while a download it had started kept going to 61GB.
None of that is a capability problem. It’s a problem with evidence — what counts as knowing, and what counts as done — and that’s the part I end up supervising.
The fixes come in two kinds. A rule is a request, a line in the file that says “done means the far end confirmed.” A hook is a fence, a script that runs on every action and blocks a force-push no matter what the model thinks. Rules work most of the time. The one about looking at its own layouts got broken five more times after I wrote it, whenever the browser it checks with was blocked, so now it says what to do when the check is impossible: lead with “unverified.” Anything I can’t afford to have go wrong eventually becomes a fence.
I think this is the real skill for people who don’t write code and want to run agents anyway. You don’t need to know how to build the page. You need to know what done looks like, catch the agent claiming it early, and write it down so you catch it three times instead of thirty. The list is my judgment, written down one correction at a time.
The honest picture is that I don’t have clean numbers yet. As of this week a hook logs every correction as I type it, so next time I can count instead of remember. Fifty sessions got me a list. A semester of logs might get me something worth publishing somewhere other than here.
The agent will tell me when it has enough data. I’ll check.
