Alibaba told its employees this week to stop using Anthropic’s Claude, citing what it called a “distillation attack” — the practice of querying one company’s model enough times to train your own on its answers. Set aside who accused whom. The interesting part is the assumption buried inside the word attack: that if you ask a model enough questions, you can walk away carrying the thing that made it valuable.
You can’t. And the reason you can’t is the most useful idea in tech right now, hiding in plain sight.
Distillation is real. Feed a strong model millions of prompts, record its outputs, and train a smaller model to imitate them. The copy will sound remarkably like the original. It will pass the same casual tests. For a while, it will feel like you got the expensive thing for the price of an API bill. This works because language models are, at bottom, machines for reproducing patterns — and patterns are exactly what distillation harvests.
The trouble shows up later, in the place where everything eventually shows up: production.
The gap between the answer and the judgment
There’s a genre of engineering complaint that has become a weekly ritual — why does AI-generated code break in production? The demo runs. The function looks right. Then it meets real traffic, a malformed input, a race condition at 2 a.m., and it folds. The usual explanation is that the model “doesn’t really understand.” True, but too vague to be useful.
Here’s the sharper version. A model trained on outputs learns what correct code looks like. It does not learn the thousand small decisions that a working system encodes — the retry that exists because someone got paged three years ago, the null check that guards against a vendor who lies about their schema, the timeout tuned to a network that isn’t in any dataset. Those aren’t visible in the output. They’re visible only in the history that produced it. Copy the surface and you copy the shape of good judgment without the judgment.
Which is precisely what distillation captures and precisely what it misses. You can clone the answers. You cannot clone the reason the answers are right.
Rolm, 1984
None of this is new. It just wears new clothes each decade.
In 1984, IBM bought Rolm, a maker of business phone systems, for over a billion dollars. On paper it was a clean transfer: IBM acquired the products, the patents, the engineers, the buildings. Everything you could put in a contract, it now owned. Within a few years the acquisition was widely judged a failure, and IBM eventually offloaded the business. What IBM bought was the resource. What made Rolm work — its particular way of moving fast, the taste of the people who built it, the culture that turned decisions into good products — didn’t come across in the paperwork. It couldn’t. It wasn’t the kind of thing that fits in a deal.
That’s the same lesson the distillation panic is circling. The valuable part of a capable model was never the outputs sitting on the other end of the API. It was the years of choices — what data to trust, what to throw away, which failures to fix and which to leave — that no amount of querying can reconstruct. You can photograph a finished building. You cannot photograph the engineering that keeps it standing in an earthquake.
Restructuring, not copying
The deeper move — the one the copyists keep skipping — isn’t to reproduce a pattern. It’s to restructure it. Real thinking treats existing information as raw material to be provoked and rearranged, not preserved. It challenges the accepted shape of a thing to find a better one. Distillation does the opposite. It freezes the pattern at its most convincing and calls that ownership.
This is why the defensive crouch — ban the tool, wall off the model — reads as slightly beside the point. Guarding your outputs assumes the outputs are the asset. But if a competitor could truly become you by studying what you produce, you were never that hard to become. The companies that will matter in five years aren’t the ones with the best answers today. They’re the ones still restructuring the questions, still capable of throwing out yesterday’s pattern the moment it stops being true.
There’s a quieter irony in all this. The same organizations racing to distill someone else’s model are the ones most likely to be surprised when the copy underperforms — because they’ll have proven, at their own expense, that the thing they wanted was never for sale.
A model’s outputs are the smoke. The fire is the process that made them — the tens of thousands of unglamorous corrections, the judgment tuned against real failure, the willingness to be wrong on the way to being right. You can inhale all the smoke you want. It will not keep you warm, and it will not build you a fire.
The strongest thing any system owns is the part that can’t be copied by watching it work.
Leave a Reply