The part that isn't solved
Agents can write the code. They can review it too. The part that still sits with me is QA: using the thing as a user would, and treating every confident finding as a hypothesis until I’ve seen it myself.
How it got there
I was reviewing an agent-built change against its plan. The agent flagged a suspected regression: a special case, it said, had been dropped in the rewrite.
The finding was crisp. It came with a citation that sounded right.
I refused to restore anything on that claim alone. I asked for an example I could verify with first.
We looked for real data that exercised the case. The local app was pointed at a tiny seeded schema that didn’t include it, so the case couldn’t even show up. I temporarily pointed the app at real data, restarted, and drove the page the way a user would.
Looking at the page: the behavior was correct. No regression.
The special case had been handled the whole time, on the server. The agent had only read a truncated frontend diff, so that handling sat outside the bytes it looked at. The finding was confident and wrong.
The lesson
Agent review has a specific failure mode: high confidence from incomplete context. A truncated diff, or a file half-read. That failure mode is invisible to more of the same reading. It’s only exposed by running the thing and looking.
Code generation is largely solved. User-level QA is not, and may be the least automatable part of the loop.