I don't want an AI that needs me less

Last week Google Cloud published an open standard for the “knowledge” that AI agents read to understand a system — the Open Knowledge Format. I had a strange reaction to it. On paper it was good news: it describes, almost part for part, something I’d been building for months on a project of my own. When a company that size independently lands on your design, the bet looks right.

But under the validation there was something that bothered me, and it took me a day to name it. The standard is built so that an AI can understand a system without a human having to. It lifts the work of knowing the system off people and encodes it into a map the machine reads for itself. That is genuinely useful. It is also pointed in a direction I don’t want to go.

Because the whole industry right now is selling AI by a single metric: how little it needs you. The dream on every launch slide is the agent that takes a one-paragraph instruction and works alone for days — doesn’t sleep, doesn’t ask, doesn’t wait. The less a human is in the loop, the better the system is said to be performing. We are automating the person out of the picture and calling it progress.

I want to say plainly that I don’t want that. Not out of nostalgia, and not because I distrust the machines — I’ve spent the last months handing more and more of my own coding to them, gladly. I don’t want it because I’ve come to think the human in the loop was never the inefficiency we’re now so eager to remove. It was the foundation of something we cannot rebuild without them: trust.

Trust has always had a human underneath it

I learned this in my body a long time before AI. In 2012 I built an app that pushed Mexico City’s earthquake alert to half a million phones. It was always offered as experimental — and people treated it, correctly, as a thing their safety depended on. There is a particular weight to that: knowing that a human, me, was answerable for what the system did and for what it failed to do. Every design decision had a name attached to it. That weight is not a defect in how we build software people rely on. It is the reason anyone trusts such software at all.

We don’t trust a system because it is capable. We trust it because, somewhere, a person understood it, decided it, and can answer for it. Take that person out — leave a capable system that no human really sees into — and you don’t have a more trustworthy system. You have a black box that happens to work, until the day it doesn’t, with no one who can tell you why.

Automate the coding. Humanize the decision.

Here’s the distinction I keep coming back to. There is coding — the typing, the wiring, the boilerplate, the mechanical translation of intent into syntax — and I am happy, even relieved, to automate as much of it as the tools can take. And there is engineering — deciding what to build and why, what to refuse, which tradeoff to accept, what debt to take on with eyes open, where the whole thing is heading. That second part is not typing. It’s judgment, direction, and accountability, and it is precisely the part the autonomous-agent dream quietly deletes.

Scott Hanselman and others have a name for the alternative: AI-augmented engineering. The human isn’t displaced; they’re moved up, into the part that was always the actual work, and amplified there by the machine. That’s the bet I’m making — and I want to be exact about it, because it’s easy to confuse with a weaker thing. It is not a human reviewing the AI’s output at the AI’s speed and rubber-stamping it; that isn’t augmentation, it’s theater. It’s a human who stays oriented: who can see what was decided and why, what’s underway, what’s still owed, and where the work is going — without having to hold the entire codebase in their head.

That last part is the hard, interesting problem, and it’s most of what I work on now. It is not about slowing the AI down — a map doesn’t slow you down, it’s how you move fast without getting lost. It’s about putting knowledge in front of a person, not just information: not a data dump for a machine to query, but a picture a human can stand inside and make a call from. When I work with an agent at its best, it doesn’t feel like delegating to something that needs me less. It feels like a pair — the machine doing what it is tireless at, me doing what only a responsible person can.

The choice hiding under the standard

So when I read that open format, I wasn’t really reading a file spec. I was reading a quiet answer to a question every team is about to face, whether they notice it or not: who is your software’s knowledge for? If the answer is “the agent, so it can run without us,” you will get exactly that — and you’ll have traded away the one thing that ever made software trustworthy. If the answer is “us, so we stay in command of what the agent does,” then you’re keeping the human not as a courtesy but as the load-bearing element.

I know which one I’m building. I’ve felt the weight of being the human a system answers through, and I don’t think that weight should be engineered away — I think it should be made bearable. We finally have machines that can carry the part that was always just labor. That isn’t a reason to remove the person. It’s the best reason in years to protect them, and to give them a better place to stand.

(I wrote the technical version of this argument — formats, typed links, interoperability — on the StrayMark blog.)