US AI labs like Anthropic and OpenAI are worried about distillation attacks from Chinese firms. These companies use outputs from stronger models to train cheaper, competing systems. Anthropic said it made “substantial progress” in reducing these attacks. OpenAI also disrupted a campaign that started in the first week of July.
Synopsis
Distillation as a technique in which a less capable AI model is trained on the outputs of a stronger one. The aim is to help the less capable models replicate some of the larger model’s capabilities at a lower cost.
Artificial intelligence (AI) companies are spending billions of dollars and years of effort to build advanced models. But what if other companies use those models to build a competing system that is faster and at a fraction of the cost?
Such is the concern behind AI distillation attacks. While distillation is a legitimate technique used to make AI models smaller and cheaper to run, Anthropic and OpenAI have accused other developers, particularly those from China such as DeepSeek, Moonshot AI and MiniMax of using their models to extract capabilities without permission.
In an October 12 blog post, Anthropic said it had made “substantial progress” in reducing the effectiveness of distillation attacks on its Claude models, with early evidence suggesting its new methods had cut the amount of data harvested by distillers by 90-99%.
What is AI distillation?
In a February blog post, Anthropic described distillation as a technique in which a less capable AI model is trained on the outputs of a stronger one. The aim is to help the less capable models replicate some of the larger model’s capabilities at a lower cost.
Live Events
Think of it as a student learning from an experienced teacher. Instead of learning everything from scratch, the student studies the teacher’s answers and learns to perform similar tasks.
AI developers use this technique to build cheaper, open-source models. The concern raised by Anthropic and OpenAI is the unauthorised use of another company’s model outputs to train, reproduce or improve a competing system.
How do such attacks work?
An attacker can send large numbers of carefully designed questions to an AI model and collect its responses. Those responses can then be used as training material for another model.
In its September 30 blog post, OpenAI said it had identified and disrupted a coordinated campaign that began in the first week of July. The company reported 16,000 requests using a particular extraction pattern across more than 4,000 users on July 24 and 25. It said further investigation identified related activity across a cluster of more than 15,000 users, which it disrupted by July 28.
“It is unclear whether all operators we observed during the relevant time period originated from a single actor. However, we attribute a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi,” OpenAI added.
Anthropic separately alleged in February that DeepSeek, Moonshot AI and MiniMax had generated more than 16 million exchanges with Claude through approximately 24,000 fraudulent accounts.
Why are reasoning traces important?
OpenAI defines protected reasoning as a model’s internal record of working through a task.
In its September 30 post, OpenAI said extracting this information can “reveal information withheld from the final answer” and help others reproduce a model’s capabilities.
“We saw operators attempt to extract protected reasoning in novel ways, including by copying encrypted reasoning from one conversation and asking a model in another conversation to decrypt and transcribe the hidden reasoning content,” it added.
OpenAI clarified that the campaign did not involve breaking its encryption, compromising a database or directly accessing stored user conversations. Instead, it said the operators manipulated model interactions to make protected reasoning visible to requesters at scale.
Why are AI companies worried?
OpenAI said unauthorised distillation could help others train, reproduce or improve competing models. Anthropic made a related point, saying such attacks let companies mimic frontier capabilities at a fraction of the development time and cost.
OpenAI also warned that extracted reasoning could train another model “without preserving the safeguards applied to the original model’s user-facing outputs”. Anthropic raised a similar concern. It said distilled models have been released without sufficiently robust safeguards and could be used to cause harm in areas such as biology and cybersecurity.
How are companies trying to stop these attacks?
OpenAI said it banned or restricted fraudulent accounts, strengthened sign-up and infrastructure controls, and expanded monitoring for related activity. It also said it had strengthened protections around hidden reasoning and worked with third-party providers to identify and disrupt suspicious accounts.
The company said it shared its findings through the Frontier Model Forum and relevant government information-sharing channels to help other AI developers strengthen their defences.
Anthropic said it restricted access to Claude’s internal reasoning across models and accounts, introduced detection systems to identify requests designed to extract that reasoning, and tightened usage limits and account restrictions for suspected distillers.
The company also said it had fixed weaknesses that allowed attackers to bypass safeguards, including attempts to get Claude to reproduce its hidden reasoning in different formats or languages. “Early evidence suggests that these methods have reduced the amount of data harvested by distillers by around 90 to 99%,” the company said in its blog post.
The ABCD of the US mid-term elections – Part 1: Why it matters across the globe
Onion inflation at 108%? Yes, but not really. What is different this time?
Curbing rupee bets: What the RBI’s new rules mean for the hedge-now-or-wait question