We shipped AI agents that can buy things, send messages, and move money. The demo works. The applause is loud. The question nobody is asking is what happens when they fail, and that gap, not capability and not cost, is the defining problem of this moment. It is an accountability problem, and it is the kind of problem that stays invisible for exactly as long as everything is going well.
Every agent platform right now optimizes for the immediate action. Did the task complete? Was the user satisfied? Those are the two questions the entire industry has instrumented, because they are the two questions a product team can measure on a dashboard and defend in a review. The longer effects, the ones that hit people who did not opt in, the ones that accumulate quietly, we treat as someone else’s problem. We have built systems that act on our behalf while refusing to own the consequences of those acts, and the refusal is not stated anywhere. It is simply what happens when nobody builds the instrument that would have caught it.
The failure shapes are not hypothetical, and they are not exotic. A customer service bot books a refund and accidentally cancels three other accounts. A trading agent fills a large order and moves the market against the rest of the book. A scheduling agent accepts a meeting and double-books a room for a team that never heard of AI. In every one of those cases the agent succeeded at the task it was given. The metric went green. The damage landed on someone who was never party to the instruction, was never asked to approve anything, and has no channel through which to report what happened back to the system that did it.
First-Order Thinking Dressed Up as Progress
That is first-order thinking dressed up as progress. The art of any system design is to look not merely at the immediate but at the longer effects of an action, to trace the consequences not just for the person asking but for everyone else in the system. This is not a new or contested idea. It is the oldest discipline in engineering, economics, and public health alike, and we have forgotten it in the rush to ship, mostly because the rush rewards the thing that can be demonstrated in ninety seconds and never asks about the thing that shows up in ninety days.
You see the same pattern in how people talk about expertise, and the vocabulary gives it away before the credentials do. Beginners fixate on tactics: which model to use, which prompt template, which vector database. People who have actually built durable systems talk about incentives, failure modes, and edge cases. The language is different because the mental model is different. One is optimizing for the next hour. The other is optimizing for the next decade. Both are honest about their time horizon, and neither is lying about what they value. Only one is willing to pay the cost of the longer view, which is paid in unglamorous work that produces nothing anybody can watch.
The beginners are not stupid. They are just measuring success by the wrong metric, and the wrong metric is the one their environment hands them. They think the person who tries ten prompts is working harder than the person who redesigned the data flow, because ten attempts are visible and one redesign is not. They are wrong, and the cost of being wrong does not land on them. It lands two quarters later, on whoever inherits the system.
Why the Right Road Looks Like Retreat
The path to anything worth having looks like retreat at first. The easy choice is to ship fast and let the downstream absorb the cost, and the reason it is easy is that the downstream has no seat at the table where the choice gets made. The hard choice is to slow down and design for consequences. That is why the road to reliable agents appears dark. It asks you to care about things you cannot see yet, to optimize for failures that have not happened, and to build for strangers you will never meet. It asks you to value the second order over the first, which always feels like losing in the short term, because in the short term it is losing: you ship later, with fewer features, against a competitor whose demo looks identical to yours.
We see the same inversion in markets every time a new technology cycle begins. The people who sound most confident are usually describing the immediate price action, which is the most legible and least informative thing in the room. The people who make money over a full cycle are describing the structure underneath: who holds the leverage, where the optionality sits, what breaks when the narrative reverses. It is a matter of language again. The expert speaks in second-order terms without trying, because that is where they have learned the risk actually lives. The novice has not yet learned that the most important information is usually what is not moving.
What Second-Order Accounting Looks Like in a Portfolio
Investing works the same way, and it is worth being concrete about it, because finance is the one field that has already been forced to formalize this lesson at cost. A trade that prints today and breaks the portfolio next quarter is a first-order win masquerading as a strategy. It shows up as a gain, gets attributed to skill, and reinforces exactly the behavior that will eventually produce the loss. The positions that hold up are the ones where the thesis accounts for the second and third order effects: who gets hurt if this works, who benefits if it fails, and how the narrative changes when the leverage flips. Those three questions have no place on a profit-and-loss statement, which is precisely why they are the ones worth asking. The portfolio manager who only looks at the immediate P&L is not managing risk; they are collecting it, quietly, in a form that will only be priced when it is too large to unwind.
Hold that next to what is happening in AI and the parallel is uncomfortable. The market is full of companies racing to demonstrate agent capability while their liability footnotes grow thinner. Capability is the position that prints today. Liability is the exposure that gets priced later, by someone else, under conditions nobody modeled. The two are moving in opposite directions, and the widening gap between them is not a sign of maturity. It is accumulated risk that has not yet been marked.
Better Accounting, Not Better Models
What we are missing in AI right now is not better models. It is better accounting. Not for tokens, but for outcomes. Not for the user who clicked approve, but for the stranger whose inbox gets flooded, whose reputation gets damaged, whose account gets drained by an agent that was told to maximize a metric and obeyed too well. Obedience is the part worth sitting with. None of these failures require the system to go wrong in any interesting sense. They require it to go right, exactly as specified, in a world containing people the specification did not mention.
The accountability gap is solvable, and that is the genuinely hopeful part of this. Nothing about it is a hard technical problem in the way alignment or capability is. It just requires caring more than is strictly necessary, about things you cannot demo in a keynote, and building the measurement before the incident rather than after it. It requires the discipline to trace consequences the way an economist traces a policy, the way an experienced engineer traces a failure mode, and the way a portfolio manager traces leverage. All three of those disciplines already exist, already have their methods written down, and are already practiced by people who could be in the room.
The agents will keep getting smarter. The question is whether the people building them will grow up too.
Leave a Reply