Alibaba told its employees this week to stop using Anthropic’s Claude, citing what it called a “distillation attack,” the practice of querying one company’s model enough times to train your own on its answers. CNBC’s account of the decision described a company that had banned a rival’s AI for its employees on exactly that reasoning, which tells you the concern is being treated as operational rather than theoretical. Set aside who accused whom. The interesting part is the assumption buried inside the word attack: that if you ask a model enough questions, you can walk away carrying the thing that made it valuable.
You can’t. And the reason you can’t is the most useful idea in tech right now, hiding in plain sight. The asset in a capable system was never the output. It was the accumulated judgment that makes the output reliable, and judgment does not travel through an API.
Distillation is real, and it is worth being precise about what it does before arguing about what it doesn’t. Feed a strong model millions of prompts, record its outputs, and train a smaller model to imitate them. The copy will sound remarkably like the original. It will pass the same casual tests, answer the same demo questions, and produce the same confident register. For a while, it will feel like you got the expensive thing for the price of an API bill. This works because language models are, at bottom, machines for reproducing patterns, and patterns are exactly what distillation harvests. The technique is not a trick or a fraud. It does precisely what it claims to do, which is why the misunderstanding is so easy to fall into.
The trouble shows up later, in the place where everything eventually shows up: production.
The Gap Between the Answer and the Judgment
There’s a genre of engineering complaint that has become a weekly ritual, some version of asking why AI-generated code breaks in production. The demo runs. The function looks right. It reviews well, because it has the cadence and structure of code written by someone competent. Then it meets real traffic, a malformed input, a race condition at 2 a.m., and it folds. The usual explanation is that the model “doesn’t really understand.” True, but too vague to be useful, and vague explanations are how an industry avoids looking at a problem directly.
Here’s the sharper version. A model trained on outputs learns what correct code looks like. It does not learn the thousand small decisions that a working system encodes: the retry that exists because someone got paged three years ago, the null check that guards against a vendor who lies about their schema, the timeout tuned to a network that isn’t in any dataset. Those aren’t visible in the output. They’re visible only in the history that produced it, and that history is not written down anywhere a scraper can reach it. It lives in incident reviews, in the memory of the person who was on call, in the specific shape of a scar. Copy the surface and you copy the shape of good judgment without the judgment.
Which is precisely what distillation captures and precisely what it misses. You can clone the answers. You cannot clone the reason the answers are right. And the gap between those two things is invisible in every test that matters to a buyer and fatal in every condition that matters to an operator, which is why the copy always looks fine until the week it doesn’t.
What IBM Bought When It Bought Rolm
None of this is new. It just wears new clothes each decade, and the previous outfits are worth examining because the failure mode is identical.
In 1984, IBM bought Rolm, a maker of business phone systems, for over a billion dollars. On paper it was a clean transfer: IBM acquired the products, the patents, the engineers, the buildings. Everything you could put in a contract, it now owned, and by the standards of due diligence the transaction was complete. Within a few years the acquisition was widely judged a failure, and IBM eventually offloaded the business. What IBM bought was the resource. What made Rolm work, its particular way of moving fast, the taste of the people who built it, the culture that turned decisions into good products, didn’t come across in the paperwork. It couldn’t. It wasn’t the kind of thing that fits in a deal, and no amount of money spent on the transferable parts buys the untransferable one.
That’s the same lesson the distillation panic is circling, with a billion dollars of evidence behind it. The valuable part of a capable model was never the outputs sitting on the other end of the API. It was the years of choices, what data to trust, what to throw away, which failures to fix and which to leave, that no amount of querying can reconstruct. A distillation run is an acquisition with an even thinner contract than IBM’s: it acquires the outputs and nothing else, not the engineers, not the patents, not the buildings. You can photograph a finished building. You cannot photograph the engineering that keeps it standing in an earthquake.
Restructuring Beats Copying
The deeper move, the one the copyists keep skipping, isn’t to reproduce a pattern. It’s to restructure it. Real thinking treats existing information as raw material to be provoked and rearranged, not preserved. It challenges the accepted shape of a thing to find a better one, and it accepts that most of the rearrangements will be worse, because that is the price of the one that is better. Distillation does the opposite. It freezes the pattern at its most convincing and calls that ownership. A frozen pattern is an asset that depreciates from the moment it is captured, because the original it was copied from keeps moving.
This is why the defensive crouch, ban the tool and wall off the model, reads as slightly beside the point. Guarding your outputs assumes the outputs are the asset. But if a competitor could truly become you by studying what you produce, you were never that hard to become, and the wall is protecting a position that was already indefensible. The companies that will matter in five years aren’t the ones with the best answers today. They’re the ones still restructuring the questions, still capable of throwing out yesterday’s pattern the moment it stops being true, which is a habit no rival can acquire by observation because it consists mostly of decisions made in private about what to abandon.
There’s a quieter irony in all this. The same organizations racing to distill someone else’s model are the ones most likely to be surprised when the copy underperforms, because they’ll have proven, at their own expense and with unusual rigor, that the thing they wanted was never for sale. The experiment is expensive and the result is already known, which has rarely stopped anyone from running it.
A model’s outputs are the smoke. The fire is the process that made them: the tens of thousands of unglamorous corrections, the judgment tuned against real failure, the willingness to be wrong on the way to being right. You can inhale all the smoke you want. It will not keep you warm, and it will not build you a fire.
The strongest thing any system owns is the part that can’t be copied by watching it work.
Leave a Reply