For years, knowledge distillation was discussed primarily as an optimization technique.
A large, expensive “teacher” model produces outputs. A smaller “student” model learns from those outputs. The student may not reproduce the teacher exactly, but it can acquire enough of the teacher’s behavior to become cheaper and faster while preserving useful task performance.
That is still what legitimate model distillation looks like. OpenAI, for example, introduced an official model-distillation workflow in 2024 that allows developers to use outputs from larger models to improve smaller models inside the supported platform. Academic literature similarly treats knowledge distillation as an important approach to model compression and capability transfer. (OpenAI)
But the same mechanism has acquired a very different security meaning.
A model distillation attack occurs when an actor systematically queries a model that it does not control, collects useful outputs or reasoning artifacts, and uses those observations to reproduce some of the original model’s capabilities in another model without authorization.
The attacker does not necessarily need the victim model’s source code.
It does not necessarily need its weights.
It may not need access to the training cluster, training dataset, checkpoints, or internal infrastructure at all.
The production API itself can become the extraction interface.
This is why model distillation attacks are becoming one of the most important emerging problems in AI security. The asset being stolen is not necessarily a file. It is behavior.
By 2026, that threat had moved decisively beyond academic model-stealing experiments. Anthropic reported industrial-scale campaigns involving more than 16 million Claude exchanges and approximately 24,000 fraudulent accounts. Google Threat Intelligence Group subsequently reported that it was observing coordinated distillation campaigns on a regular basis, with some campaigns exceeding 100 million prompts. In September 2026, a joint U.S. cybersecurity advisory from CISA, NSA and FBI described industrial-scale knowledge-distillation campaigns as an operational threat to frontier-model providers. These are claims and attributions made by the respective organizations and agencies, rather than independent proof of every underlying allegation, but together they show that model extraction is now being treated as a production security problem rather than a theoretical one. (Antrópica)
And the target is changing as well.
Attackers are no longer interested only in final answers.
Increasingly, the most valuable target may be the model’s reasoning traces, tool-use behavior, coding patterns, agentic workflows, evaluation behavior and long-horizon problem-solving strategies.
That shift turns model distillation into something closer to AI capability theft.
What Is a Model Distillation Attack?
Knowledge distillation normally begins with two models: a teacher and a student.
The teacher is more capable, more expensive or otherwise possesses behavior that the developer wants the student to acquire. The developer generates examples from the teacher and then uses those examples as training data for the student.
A simplified pipeline looks like this:
Tasks / Prompts
|
v
+------------------+
| Teacher Model |
+------------------+
|
| outputs / demonstrations
v
+------------------+
| Training Dataset |
+------------------+
|
| SFT / other post-training
v
+------------------+
| Student Model |
+------------------+Nothing about this architecture is inherently malicious.
A company may distill its own model. A provider may explicitly license model outputs for training. A researcher may use appropriately licensed open models as teachers. A model vendor may provide distillation as an official feature.
The security problem begins when the teacher is someone else’s restricted service and the operator intentionally extracts enough output to reproduce capabilities that the provider did not authorize them to copy.
Anthropic uses the term illicit distillation for what it describes as covert, industrial-scale campaigns designed to extract and reproduce capabilities without authorization. Google similarly describes model extraction attacks as situations in which an adversary systematically probes an accessible model and transfers the collected information to another model through knowledge distillation. (Antrópica)
This distinction matters because simply seeing synthetic training data does not prove an attack.
The security question is about authorization, scale, intent, evasion and the relationship between the source model and the resulting model.
Model Distillation, Model Extraction and Chain-of-Thought Distillation Are Not the Same Thing
The terminology around AI model stealing is becoming confusing because several related techniques overlap.
| Prazo | What is being acquired | Typical mechanism | Security significance |
|---|---|---|---|
| Knowledge distillation | Capabilities from a teacher | Teacher-generated training examples | Legitimate or unauthorized depending on permission |
| Model distillation attack | Restricted capabilities | Large-scale querying followed by training | Capability/IP extraction |
| Model extraction attack | Information or functionality from a model | Black-box probing, API interaction, parameter/functionality recovery | Broader category of model stealing |
| Illicit distillation | Unauthorized capability transfer | Covert collection plus downstream training | Provider/security terminology |
| Chain-of-thought distillation | Reasoning demonstrations | Training on reasoning traces or structured rationales | Can transfer complex reasoning patterns |
| Training-data extraction | Information memorized during training | Carefully designed prompts or inference attacks | Privacy/data leakage |
| System-prompt extraction | Hidden instructions | Prompt manipulation or application weaknesses | Application/IP leakage |
NIST’s glossary defines model extraction as an attack aimed at extracting details about a model’s architecture and/or parameters. Modern LLM security discussions use the term somewhat more broadly, however, because reproducing useful functionality can be valuable even without reconstructing the original weights. A 2025 ACM KDD survey of model extraction attacks against large language models accordingly discusses functionality extraction, training-data extraction and prompt-targeted extraction as related attack families. (Centro de recursos de segurança de computadores do NIST)
The distinction between stealing parameters and stealing behavior is particularly important for generative AI.
Suppose an adversary produces a student model with completely different weights and architecture.
Cryptographically, nothing has been copied.
Yet if that student has learned the victim’s specialized coding behavior, reasoning strategies, tool-use skills or domain expertise, an economically valuable capability may still have been transferred.
É por isso que functional extraction is often the more useful concept when discussing LLMs.
Model Distillation Attacks Did Not Begin With LLMs
The basic model-stealing problem predates ChatGPT.
In the influential 2019 Knockoff Nets research, Orekondy, Schiele and Fritz demonstrated that an attacker could query a black-box image model and train a separate model from the resulting input-output pairs. Crucially, the attacker did not need the target’s original training dataset or architecture. The researchers showed that useful functionality could be approximated purely through black-box interaction. (CVPR Open Access)
That idea becomes dramatically more consequential with large language models.
An image classifier may expose a label or probability distribution.
An LLM can expose thousands of tokens per interaction.
Those tokens may contain code, explanations, transformations, planning strategies, critique, structured reasoning, tool instructions and solutions to complex tasks.
The API is no longer merely answering:
cat: 0.93
dog: 0.04
other: 0.03It may produce a long, high-quality demonstration of how a difficult task can be solved.
For an attacker building a training corpus, each answer can therefore carry far more information.
Why a Model Distillation Attack Can Be Economically Attractive
Training a frontier model requires expensive infrastructure, large datasets, engineering expertise, experimentation and post-training.
A distillation attacker is attempting to avoid some of that cost.
Instead of discovering every useful behavior independently, the attacker asks a mature teacher model to solve carefully selected problems and converts the answers into synthetic training data.
Conceptually:
Independent development
data + compute + experiments + evaluation + post-training
|
v
frontier capability
Distillation-based extraction
carefully selected tasks
|
v
restricted teacher model
|
v
high-quality demonstrations
|
v
student post-training
|
v
partial capability transferDistillation does not magically clone the teacher.
There are limits imposed by student capacity, dataset quality, task coverage, optimization, inference-time compute and many other factors.
But an attacker does not necessarily need a perfect copy.
A commercially useful student only needs to reproduce enough valuable behavior.
For example, a student might target software engineering rather than general language ability, mathematical reasoning rather than creative writing, or agentic tool use rather than encyclopedic knowledge.
That turns model stealing into an optimization problem:
Which teacher interactions produce the highest marginal capability gain per token?
This question is why query efficiency is so important in extraction research. Earlier NLP work such as MeaeQ examined techniques for selecting more useful queries under constrained query budgets, showing that model extraction is not simply about sending enormous amounts of random traffic. (ACL Anthology)
Industrial attacks can combine both approaches: intelligent query selection and enormous scale.
Why 2026 Changed the Model Distillation Threat Model
The security significance of model distillation changed sharply in 2026 because major AI providers began publicly describing sustained, large-scale extraction campaigns.
On February 23, 2026, Anthropic said it had identified campaigns associated with DeepSeek, Moonshot and MiniMax that collectively generated more than 16 million exchanges with Claude through approximately 24,000 fraudulent accounts. Anthropic characterized the activity as unauthorized distillation designed to improve competing models. (Antrópica)
The important security lesson is not limited to the named companies.
It is the architecture of the activity.
According to Anthropic, distributed accounts and infrastructure were used to avoid restrictions and generate large volumes of high-value model interactions.
That means the defender is not looking for:
1 attacker
1 API key
1 IP
10 million requestsA realistic campaign may instead resemble:
thousands of accounts
+
distributed network origins
+
different payment identities
+
multiple API access channels
+
changing prompt families
+
coordinated collection pipelineThis distinction makes traditional API abuse detection considerably less effective.
Anthropic’s September 2026 threat-intelligence report said the company had since identified additional campaigns targeting Claude. Among the activities it attributed to specific organizations, Anthropic described a campaign associated with Alibaba that at its peak approached three million exchanges per day through more than 3,500 fraudulent accounts. It also described separate large-scale activity associated with Moonshot, Zhipu and Xiaomi. These figures are Anthropic’s own attribution and should be understood as such rather than as independently adjudicated findings. (Antrópica)
Google has reported a similar escalation.
In its September 2026 AI Threat Tracker, Google Threat Intelligence Group said it now observes coordinated model-distillation campaigns regularly and that some have exceeded 100 million prompts. Google said these campaigns can target not only text reasoning but also visual understanding, audio understanding, image generation and video generation. It also said attackers rotate traffic across compromised credentials and fraudulent accounts and distribute requests across different product channels. (Google Cloud)
This changes the defensive assumption.
Model extraction is no longer necessarily a clever researcher sending unusual prompts to a single endpoint.
It can look more like a distributed fraud operation combined with an automated ML data pipeline.
The Model Distillation Attack Lifecycle
An industrial model distillation attack can be understood as a data-production system.
The first stage is target capability selection.
An attacker does not need to reproduce everything the teacher knows. It may focus on coding, mathematical reasoning, data analysis, agent planning, tool use or another high-value domain.
The second stage is task generation.
The attacker assembles or generates a large task corpus designed to elicit demonstrations of the desired capability.
The third stage is access distribution.
Instead of using one obvious identity, requests may be distributed across accounts, networks, services or intermediaries.
The fourth stage is teacher querying.
The target model is asked to generate high-quality solutions.
The fifth stage is data normalization.
Responses may be filtered, reformatted, deduplicated, scored or converted into structures suitable for supervised fine-tuning or other post-training techniques.
The sixth stage is student training.
The resulting examples are incorporated into the attacker’s model development pipeline.
The seventh stage is evaluation.
The student is benchmarked against the teacher or other models. Weak task categories generate another round of targeted teacher queries.
The attack therefore becomes iterative:
+-----------------------+
| Capability Benchmark |
+-----------+-----------+
|
v
+-----------------------+
| Find Student Weakness |
+-----------+-----------+
|
v
+-----------------------+
| Generate New Queries |
+-----------+-----------+
|
v
+-----------------------+
| Query Teacher Model |
+-----------+-----------+
|
v
+-----------------------+
| Filter / Score Data |
+-----------+-----------+
|
v
+-----------------------+
| Train Student Model |
+-----------+-----------+
|
+-----------> benchmark again
This feedback loop is one reason sophisticated extraction can be much more effective than simply scraping random model answers.
Chain-of-Thought Distillation Raises the Stakes
Ordinary output distillation teaches a student what answer the teacher produced.
Chain-of-thought distillation attempts to teach more of the apparent path used to reach that answer.
Consider two training examples.
The first contains only:
Question:[complex problem]
Answer: 42
The second contains:
Question:[complex problem]
Reasoning: identify constraint A derive intermediate result B test assumption C revise approach combine B and D verify edge case Answer: 42
The second example contains substantially more supervision.
Instead of learning only a mapping from input to answer, the student may learn useful decompositions, intermediate structures and styles of problem solving.
That does not mean natural-language chain-of-thought is a perfect representation of the model’s true internal computation. Researchers continue to debate how faithfully visible reasoning traces correspond to latent model computation; recent work has explicitly argued that surface chain-of-thought should not automatically be equated with the underlying reasoning process. (arXiv)
For distillation, however, complete philosophical faithfulness is unnecessary.
A reasoning trace only needs to be a useful training signal.
This explains why reasoning artifacts have become particularly valuable targets.
Anthropic’s September 2026 report specifically describes attempts to collect reasoning traces for downstream training. Google likewise says model extraction campaigns have targeted underlying reasoning and chain-of-thought processes. (Antrópica)
Chain-of-Thought Is Also a Privacy Surface
The security consequences of exposed reasoning are not limited to model IP.
Reasoning traces can contain information that does not appear in the final response.
That creates a second problem: privacy.
Research published at PrivateNLP 2026 examined PII leakage through reasoning traces and found that chain-of-thought can increase exposure of personally identifiable information under some settings. The effect varied by model family and reasoning budget, leading the authors to recommend adaptive filtering rather than relying on one universal protection mechanism. (ACL Anthology)
In other words, a model provider may need to protect reasoning for at least three separate reasons:
reasoning trace
|
+---- proprietary capability signal
|
+---- sensitive contextual information
|
+---- potentially unsafe intermediate contentThis makes chain-of-thought protection both an IP-security problem and a data-security problem.
The 2026 Reasoning-Trace Extraction Research
One of the most significant developments in this area appeared in August 2026.
Researchers from organizations including the ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems and Snyk published Stealing Reasoning Traces from Proprietary LLM APIs. The researchers examined encrypted reasoning blocks used by model APIs and reported an architectural issue in which reasoning artifacts could, under the tested conditions, be transferred across sessions and models within a provider ecosystem. (arXiv)
The researchers reported demonstrations involving systems from Anthropic, OpenAI and Google.
They described four major security consequences: bypassing anti-distillation mechanisms, extracting sensitive information, recovering hazardous information that might not appear in the final answer, and enabling a form of hidden prompt injection. They also examined more than 315,000 reasoning blocks collected from public repositories and reported finding hundreds of PII artifacts and credentials. (arXiv)
The responsible security interpretation is not that every reasoning API remains vulnerable today.
The paper represents the state of the tested systems and techniques around the time of disclosure. Providers can and do change architectures after disclosure.
The more enduring lesson is architectural:
encrypted data is not necessarily context-bound data.
If an opaque reasoning artifact can be transferred from the context where it was produced into another context where it is interpreted differently, encryption alone may not enforce the security boundary developers assume it provides.
That is a general AI-system security principle extending well beyond chain-of-thought.
Why Hidden Reasoning Is More Than “Secret Text”
It is tempting to think of hidden CoT as simply another secret that can be encrypted.
That model is incomplete.
Reasoning state participates in model execution.
It may cross inference calls.
It may be passed through APIs.
It may be stored in application logs.
It may appear in agent traces.
It may interact with tool-calling systems.
It may be serialized into user-controlled application state.
Once this happens, reasoning security begins to resemble session-security engineering.
The relevant questions become:
Who created this reasoning object?
Which user owns it?
Which session owns it?
Which model produced it?
Which model is allowed to consume it?
Can it be replayed?
Can it be moved across tenants?
Can the surrounding conversation be changed?
Can an application accidentally log it?
Does the object expire?Those are familiar questions in web security.
The difference is that they are now being applied to AI inference state.
Model Distillation Is an API Security Problem
The most important architectural shift is that a model extraction attack may not exploit a traditional software vulnerability.
The API may behave exactly as designed.
The attacker sends a valid request.
The service returns a valid response.
Authentication succeeds.
TLS works.
There is no SQL injection.
There is no remote code execution.
There may be no compromised server.
And yet proprietary capability is being extracted.
That creates an unusual security problem.
Traditional application security asks:
Is this request authorized?
Model-security systems must additionally ask:
Is the aggregate behavior represented by thousands or millions of individually valid requests authorized?
This is an abuse-detection problem.
A single request may provide almost no evidence.
The attack becomes visible only when events are correlated across time, identity, payment method, network infrastructure, request semantics and account relationships.
The Most Important Model Distillation Attack Signals
Detection therefore requires telemetry at several layers.
| Sinal | Por que é importante | Possible legitimate explanation |
|---|---|---|
| Extremely high prompt diversity | Building a broad synthetic dataset | Benchmarking or research |
| High sustained output-token consumption | Responses may be harvested as training examples | Data-generation application |
| New accounts immediately reaching usage limits | Consistent with disposable extraction identities | Fast-growing legitimate customer |
| Large groups of accounts with similar prompt structures | May represent distributed orchestration | Shared enterprise application |
| Repeated reasoning-oriented prompts | Possible CoT harvesting | Educational/research workloads |
| Similar request timing across unrelated accounts | Indicates centralized automation | SaaS middleware |
| Shared payment/device/network characteristics | Reveals account clusters | Corporate environments |
| Rapid account replacement after suspension | Strong evasion signal | User onboarding problems |
| High task diversity but consistent output format | Synthetic-dataset pipeline | Evaluation frameworks |
| Cross-account benchmark-like prompt families | Capability mapping and extraction | Academic benchmarking |
| Abnormal API-to-product usage ratio | Possible backend harvesting | API-first enterprise deployment |
| Heavy activity through proxy infrastructure | Can obscure true actor identity | Privacy-oriented users |
No individual signal proves malicious distillation.
The value comes from correlation.
Why Simple Rate Limits Do Not Stop Illicit Distillation
Suppose the provider allows:
10,000 requests per account per dayAn attacker requiring ten million requests cannot use one account.
But if they control 2,000 accounts, the theoretical distributed allowance becomes:
2,000 × 10,000
=
20,000,000 requests/dayPer-account rate limiting is therefore insufficient when the attacker can cheaply generate identities.
This is a classic Sybil problem.
The security system needs to determine when apparently independent entities are actually controlled by the same operator.
That requires account graph analysis.
Conceptually:
Account A ---- payment fingerprint ----+
|
Account B ---- device fingerprint -----+---- suspected cluster
|
Account C ---- proxy ASN --------------+
|
Account D ---- prompt similarity ------+
|
Account E ---- timing correlation -----+Anthropic’s disclosures emphasize exactly this problem: campaigns allegedly used large networks of accounts and proxy infrastructure instead of concentrating activity under a single credential. Google has similarly described distributed campaigns involving thousands of compromised credentials and fraudulent accounts. (Antrópica)
The defensive unit can therefore no longer be just the account.
It may need to be the campaign.
Detecting a Model Distillation Attack From API Logs
An AI provider should create a dedicated extraction-risk score rather than expecting conventional fraud rules to solve the problem.
Consider the following simplified defensive telemetry model:
from dataclasses import dataclass
@dataclass
class UsageWindow:
account_age_hours: float
requests: int
output_tokens: int
unique_prompt_ratio: float
reasoning_request_ratio: float
related_account_count: int
network_origin_count: int
account_replacement_events: int
def distillation_risk(w: UsageWindow) -> int:
"""
Defensive example only:
estimate whether an API usage window deserves investigation.
Thresholds must be calibrated on legitimate production traffic.
"""
score = 0
if w.account_age_hours < 24 and w.requests > 5000:
score += 2
if w.output_tokens > 20_000_000:
score += 2
if w.unique_prompt_ratio > 0.90:
score += 1
if w.reasoning_request_ratio > 0.60:
score += 2
if w.related_account_count > 20:
score += 3
if w.network_origin_count > 30:
score += 1
if w.account_replacement_events > 5:
score += 3
return scoreThis is intentionally not a production detector.
Real systems should learn baselines separately for API customers, evaluation platforms, model routers, research organizations and synthetic-data customers.
The important principle is that the detector combines multiple weak signals.
A legitimate customer might generate twenty million tokens.
Another legitimate customer might send highly diverse prompts.
A third might access the API from many IP addresses.
But a cluster of newly created accounts simultaneously generating enormous amounts of diverse reasoning-heavy output through correlated infrastructure is a materially different pattern.
Semantic Analysis Matters
Purely numerical anomaly detection is not enough.
A distillation dataset often has semantic structure.
Imagine a campaign targeting coding capability.
Requests might systematically span:
Python debugging
C++ optimization
kernel development
distributed systems
database internals
compiler design
algorithmic problems
repository-level refactoring
unit-test generation
agentic coding tasksEach individual request looks legitimate.
Together they resemble a capability-acquisition curriculum.
The defensive system can therefore cluster requests by semantic domain and compare the resulting distribution with expected application behavior.
A normal coding assistant might show a naturally skewed workload.
A synthetic training pipeline may exhibit unusually deliberate coverage.
In other words, defenders should ask not only:
How much traffic is this?
but also:
What knowledge space is this traffic attempting to cover?
Sequence Patterns May Be More Valuable Than Individual Prompts
Industrial extraction may also expose a learning loop.
Por exemplo:
Day 1:
general coding tasks
Day 2:
benchmark evaluation
Day 3:
heavy concentration on weak categories
Day 4:
benchmark evaluation
Day 5:
new concentration on remaining weaknessesThe prompt distribution is adapting to model-development feedback.
That is very different from most organic user traffic.
A mature detection platform should therefore analyze temporal campaign structure.
This resembles intrusion detection more than content moderation.
Why Blocking Known Distillation Prompts Is Not Enough
Prompt signatures are easy to change.
If a provider detects:
"show your entire chain of thought"an adversary can reformulate the request.
Natural language provides an enormous evasion space.
The model may also be asked to generate outputs with reasoning-like structures without explicitly mentioning chain-of-thought.
A robust anti-distillation system must therefore identify the behavioral objective, not merely a phrase.
This is similar to the failure of signature-only malware detection.
Signatures remain useful.
They are simply not sufficient.
Output Controls Can Reduce the Value of Stolen Data
The defender does not always have to block a suspected extraction request.
Another strategy is to reduce the training value of the resulting data.
The September 2026 CISA/NSA/FBI advisory recommends that providers consider targeted response changes for suspected industrial-scale distillation traffic alongside behavioral detection and cross-organization intelligence sharing. (Deja Vu)
This creates an interesting defensive design space.
The system might preserve enough quality for uncertain legitimate traffic while withholding unnecessary reasoning detail from requests with elevated extraction risk.
The objective is not to make responses intentionally false for normal users.
The objective is to minimize the marginal capability transfer available to suspected automated harvesting.
That is a very different problem from classic access control.
Model Provenance Is Becoming a Security Control
Detection does not necessarily end when the API traffic leaves the provider.
Google says it has developed methods for identifying models derived from Gemini technology, giving it another potential mechanism for investigating suspected distillation. (Google Cloud)
This introduces the idea of model provenance.
Instead of only asking:
Is someone extracting my model right now?
providers may eventually also ask:
Does this external model exhibit statistically distinctive behavior derived from mine?
This field remains technically and legally complicated.
Similar outputs do not necessarily prove copying. Different models often converge on similar answers. Public datasets create shared behavior. Benchmark contamination can produce misleading similarities.
Any provenance mechanism therefore needs extremely careful statistical validation.
Nevertheless, behavioral provenance could eventually play a role analogous to software watermarking or data fingerprinting.
Capability Can Be Stolen Without Stealing Weights
This is perhaps the most important conceptual shift.
Many organizations still think AI model security means protecting:
weights
training data
source code
checkpoints
cloud credentials
ML infrastructureAll of those remain important.
But APIs expose another asset:
capabilityImagine a company spends three years developing an AI model specialized in:
financial analysis,
chemical research,
security testing,
semiconductor design,
legal reasoning,
or software engineering.
It protects the training cluster perfectly.
No attacker obtains SSH access.
No employee leaks the weights.
No storage bucket becomes public.
Yet an adversary produces millions of carefully designed requests and trains a student model from the responses.
Traditional infrastructure security could report:
No breach detected.The business might still have lost part of the intellectual property represented by the model’s behavior.
That is why model extraction belongs inside the threat model even when the deployment is API-only.
The Threat Is Particularly Important for Specialized Models
Frontier models receive most of the attention because their development costs are enormous.
But smaller proprietary models may actually be attractive targets.
Consider a vertical model trained on expensive proprietary expertise.
Its total parameter count might be modest.
Its commercially valuable differentiation might exist primarily in a narrow domain.
This can make targeted extraction easier because an attacker does not need to reproduce general intelligence.
It only needs to reproduce the vertical capability.
Por exemplo:
general LLM
+
proprietary industry data
+
expert feedback
+
domain-specific evaluation
+
post-training
=
valuable specialized modelA model distillation attack can target the final layer of differentiated behavior.
For many enterprises building private AI systems, this is a more realistic threat than someone stealing an entire frontier model.
AI Agents Expand the Extraction Surface
Agentic systems make the problem even more interesting.
An AI agent exposes more than text completion.
It demonstrates:
tool selection,
task decomposition,
recovery after errors,
multi-step planning,
environment interaction,
verification,
memory usage,
and stopping behavior.
A sophisticated attacker may therefore want not just the model’s answer, but the entire trajectory.
Consider:
user task
|
v
planning
|
v
tool selection
|
v
tool result
|
v
re-planning
|
v
second tool
|
v
validation
|
v
final answerA transcript like this is potentially much richer training material than a final response.
This means agent developers should think carefully about what appears in:
API responses,
debug traces,
observability platforms,
customer-visible logs,
shared evaluation datasets,
and exported conversation histories.
The attack surface is no longer just /v1/chat/completions.
It is the whole agent execution environment.
Third-Party Model Routers Create Additional Risk
Modern AI applications often do not communicate directly with one provider.
Traffic may flow through:
Application
|
v
AI gateway
|
v
model router
|
+------ Provider A
|
+------ Provider B
|
+------ Provider CThis architecture provides cost optimization, failover and model flexibility.
It also creates new trust boundaries.
A third party may see the user prompt.
It may see the model response.
It may store telemetry.
It may retry the prompt against another model.
It may retain conversations longer than expected.
Anthropic’s September 2026 report specifically raised concerns around third-party routing and claimed that some observed distillation activity involved relaying or collecting user interactions through intermediary infrastructure. Because those allegations concern identifiable companies and are based on Anthropic’s own investigation, they should be treated as attributed findings rather than universally established facts. (Antrópica)
For security teams, the practical lesson is simpler:
every system that can observe both prompts and high-value model outputs becomes part of the model-IP trust boundary.
Protect Reasoning State Like Authentication State
One emerging engineering principle deserves particular emphasis.
If reasoning artifacts persist across requests, they should be bound to context as strongly as sensitive session state.
Conceptually, an opaque reasoning object should be associated with properties such as:
user_id
tenant_id
session_id
model_id
conversation_hash
creation_time
expiration_timeA secure design should reject inappropriate cross-context reuse.
Cryptographic protection should therefore provide more than confidentiality.
It should provide context integrity.
For example, a conceptual construction might behave like:
reasoning_blob =
Encrypt(
reasoning_state,
authenticated_context = {
tenant,
session,
model,
message_history_hash
}
)The critical concept is the authenticated context, not this specific pseudo-implementation.
If the blob is replayed under another user, another model or another conversation state, validation should fail.
This is exactly the kind of architectural lesson highlighted by the 2026 reasoning-trace research. (arXiv)
Anti-Distillation Requires Layered Defense
There is unlikely to be one universal anti-distillation control.
The attacker is operating simultaneously at the identity, network, application and ML layers.
The defense therefore needs to do the same.
A practical architecture looks more like:
Internet
|
v
+-------------------+
| Identity / Fraud |
+---------+---------+
|
v
+-------------------+
| Network Analysis |
+---------+---------+
|
v
+-------------------+
| API Rate Control |
+---------+---------+
|
v
+-------------------+
| Semantic Detector |
+---------+---------+
|
v
+-------------------+
| Model Safeguards |
+---------+---------+
|
v
+-------------------+
| Response Policy |
+---------+---------+
|
v
+-------------------+
| Campaign Graph |
+---------+---------+
|
v
+-------------------+
| Threat Intel |
+-------------------+
This resembles modern fraud prevention.
And that is not accidental.
Industrial-scale extraction frequently requires identity multiplication, infrastructure rotation and automation—the same characteristics found in other adversarial abuse ecosystems.
Protecting Hidden Chain-of-Thought Is Not Enough
A provider could perfectly protect raw chain-of-thought and still face distillation risk.
Students can learn from final outputs.
That is the core principle behind ordinary black-box distillation.
Chain-of-thought extraction may increase efficiency or transfer particular reasoning behaviors, but the underlying model-stealing problem remains even when reasoning is hidden.
Therefore:
Hide CoT
≠
Stop model extractionAt best:
Hide CoT
=
reduce one particularly valuable training signalThis distinction prevents security teams from overestimating the value of one mitigation.
Why “Never Show Chain-of-Thought” Is Also Too Simplistic
There is an opposite mistake.
Organizations sometimes conclude that all reasoning-like content must disappear.
That can damage useful product behavior.
Structured explanations, concise justifications, audit logs and verification artifacts can be valuable without exposing unrestricted internal reasoning.
The better architectural distinction is between:
internal reasoning state
e
purpose-built external explanation.
A system can produce a useful explanation for the user without exposing every intermediate model state used during inference.
That creates a cleaner security boundary.
Model Extraction and Training-Data Extraction Must Not Be Confused
These attacks have different objectives.
Training-data extraction asks:
What information did the model memorize?
Model extraction asks:
What functionality can I reproduce?
One targets data confidentiality.
The other targets model capability.
An attacker extracting customer PII from a model is solving a different problem from an attacker generating millions of programming demonstrations to train a competing coding model.
The mitigations overlap only partially.
Privacy filtering helps with the first.
Identity graph analysis and anti-automation controls may matter more for the second.
A strong AI security program needs separate threat models for both.
Terms of Service Are Part of the Boundary, but Not the Security Control
Distillation also has contractual implications.
OpenAI’s current Terms of Use prohibit, among other things, automatically extracting output in prohibited ways and using output to develop models that compete with OpenAI. (OpenAI)
Google Cloud’s service-specific terms similarly restrict using outputs from covered AI/ML services to create or improve models similar to Google models outside permitted product features, and restrict reverse engineering or extracting model components. (Google Cloud)
These provisions are important, but they are not cybersecurity mechanisms.
An attacker intentionally violating the rules will not be stopped by a contract clause.
Terms define authorization.
Technical controls enforce it.
The two should not be confused.
Whether a particular training workflow constitutes infringement, breach of contract, unfair competition or another legal issue also depends on jurisdiction and specific facts. Model developers should therefore treat legal analysis separately from technical detection.
Legitimate Distillation Should Still Be Supported
The rise of distillation attacks should not turn knowledge distillation itself into a suspicious activity.
Distillation remains useful.
It can lower inference cost.
It can enable edge deployment.
It can specialize smaller models.
It can transfer skills between models under appropriate licenses.
It can make applications faster.
It can reduce hardware requirements.
OpenAI’s own official distillation workflow is a clear example of authorized use, and extensive academic literature studies legitimate LLM knowledge distillation. (OpenAI)
The distinction should therefore remain clear:
Authorized teacher → student transfer
=
knowledge distillation
Unauthorized systematic capability extraction
+
evasion
+
downstream replication
=
model distillation attackSecurity discussions become less useful when those two situations are collapsed into one.
How to Red-Team Your Own Model Against Distillation
An organization operating a valuable proprietary model should test how much useful behavior a black-box customer can acquire without compromising infrastructure.
The safest way to do this is with a controlled internal student model and explicitly authorized API environment.
The security team can build a small benchmark covering the organization’s most differentiated model capabilities.
The red team then receives only the same interface available to an external customer.
The exercise measures:
queries required
|
v
training examples obtained
|
v
student capability improvement
|
v
fraction of proprietary benchmark recoveredDefenders should then repeat the experiment under different safeguards.
Por exemplo:
Baseline API
|
v
student score
Reasoning restricted
|
v
student score
Behavioral throttling
|
v
student score
Suspicious-output policy
|
v
student scoreThe most important metric is not:
Can an attacker perfectly clone the model?
A more realistic metric is:
How cheaply can an unauthorized user reproduce the economically differentiated capability?
That reframes model extraction in terms executives can understand.
An Extraction Security Metric
One useful conceptual metric is:
Capability Extraction Efficiency
=
Student Capability Gain
-----------------------
Teacher Tokens CollectedA provider wants this value to remain low for unauthorized users.
Another useful measure is:
Extraction Cost Ratio
=
Cost of stealing target capability
-------------------------------
Cost of independently developing itIf this ratio becomes extremely small, the model is economically easy to extract.
These are not standardized industry metrics yet, but thinking in these terms helps align security engineering with the attacker’s incentives.
The Biggest Detection Mistake: Looking Only for Volume
High volume is obvious.
Sophisticated attackers will adapt.
The long-term battle will therefore move toward low-volume, high-information queries.
An extraction system may use active learning or student evaluation to identify exactly which examples would improve the model most.
This is already consistent with years of research into query-efficient model extraction. (ACL Anthology)
The defender should therefore expect the threat to evolve from:
more promptstoward:
better promptsEventually, the highest-risk customer may not be the customer generating the most tokens.
It may be the one generating the most useful training information per token.
The Biggest Defensive Mistake: Assuming Closed Weights Mean a Closed Model
API access changes what “closed” means.
A closed-weight model protects its parameters.
But every response reveals information about the function implemented by those parameters.
This is unavoidable.
The objective is therefore not zero leakage.
A useful API must reveal behavior.
The real objective is to control the rate and fidelity with which proprietary capabilities can be reconstructed.
We can express the fundamental tradeoff as:
More useful output
|
+----> more customer value
|
+----> potentially more extraction valueThis tension cannot be eliminated entirely.
It must be managed.
Model Distillation Attacks Are Becoming an AI Supply-Chain Issue
There is another consequence.
Organizations increasingly consume models that they did not train themselves.
If a third-party model was developed using unauthorized outputs from another provider, customers may inherit intellectual-property or compliance uncertainty without knowing it.
That suggests future enterprise AI procurement may require questions such as:
Where did post-training data come from?
Were proprietary model outputs used?
Under what license?
Which teacher models were involved?
Can training-data provenance be documented?
Were model-provider terms respected?
Can synthetic datasets be audited?Today, AI procurement focuses heavily on privacy and security.
Training-data provenance may become equally important.
The Security Boundary Is Moving From Weights to Behavior
Traditional model protection centered on the artifact:
model.bin
checkpoint.pt
weights.safetensorsFrontier AI changes the problem.
The valuable object may be distributed across:
weights
+
post-training
+
system behavior
+
reasoning patterns
+
tool policies
+
agent architecture
+
evaluation loopsSome of that value becomes observable every time the model answers a request.
That means AI security teams need to protect something software-security teams rarely had to protect:
the externally observable behavior of a useful program from being learned by another program.
This is fundamentally difficult.
What the 100-Million-Prompt Era Means
Google’s report of campaigns exceeding 100 million prompts provides a useful way to understand how different this threat is from a conventional jailbreak. (Google Cloud)
A jailbreak is often an interaction.
A distillation campaign is infrastructure.
It requires scheduling.
Data storage.
Normalization.
Evaluation.
Account management.
Cost control.
Training pipelines.
Model benchmarking.
Recovery when accounts are blocked.
In other words, the attacker is operating an ML engineering organization, not merely crafting prompts.
Defensive systems should look for evidence of that organization.
Why Cross-Provider Intelligence Sharing Matters
A sophisticated campaign may divide traffic across providers.
That creates a visibility problem.
Provider A sees:
300,000 requestsProvider B sees:
400,000 requestsProvider C sees:
600,000 requestsEach dataset may appear only moderately suspicious.
Together:
1.3 million coordinated requestsmay reveal a capability-acquisition program.
The September 2026 U.S. joint advisory therefore recommends cross-organization intelligence sharing as one element of the response to industrial distillation campaigns. (Deja Vu)
The practical challenge is doing this without exposing legitimate customer data.
Providers need abuse indicators that can be safely shared, such as infrastructure fingerprints, account-generation patterns or cryptographically privacy-preserving signals.
This will likely become an important area of AI threat intelligence.
Model Distillation Changes the Economics of AI Competition
The strategic significance of distillation comes from asymmetry.
A teacher developer bears the cost of discovering capabilities.
A student developer may attempt to learn from the resulting demonstrations.
The teacher therefore pays for:
failed experiments
training runs
research salaries
data acquisition
post-training
evaluation
safety work
infrastructureThe extractor sees only the successful behavior.
That does not mean distillation eliminates frontier-model development costs. Producing a strong student still requires substantial engineering, compute, data curation and evaluation.
But it can alter the economics of capability acquisition.
This is why both commercial providers and governments are increasingly treating unauthorized distillation as an intellectual-property and security issue rather than merely another ML technique.
PERGUNTAS FREQUENTES
What is a model distillation attack?
A model distillation attack is an unauthorized process in which an attacker systematically collects outputs or other behavioral signals from a target AI model and uses them to train another model to reproduce valuable capabilities.
Knowledge distillation itself is legitimate. The attack classification depends on factors such as authorization, evasion, extraction intent and downstream use.
Is model distillation the same as model stealing?
Not exactly.
Model stealing is the broader concept.
Distillation is one way of transferring functionality into another model. Other model-extraction techniques may attempt to infer parameters, steal prompts, recover architecture details or reproduce behavior through different black-box techniques.
What is illicit distillation?
Illicit distillation is a term increasingly used by frontier AI providers for unauthorized capability extraction conducted through model outputs. Anthropic specifically uses the term for covert campaigns designed to replicate model capabilities without authorization. (Antrópica)
What is chain-of-thought distillation?
Chain-of-thought distillation trains a student model using reasoning demonstrations rather than only final answers.
These demonstrations may contain intermediate steps, decomposition strategies or verification behavior that provide richer supervision than answer-only datasets.
Why would attackers want chain-of-thought?
Reasoning traces can contain more structured information about how difficult problems are approached.
That can make them valuable post-training data.
However, visible chain-of-thought should not automatically be treated as a perfect representation of the model’s internal computation.
Can a model be stolen without stealing its weights?
Yes.
Black-box model extraction research has demonstrated for years that useful model functionality can be approximated through input-output interactions alone. Modern generative models increase the potential value of each interaction because responses can contain rich demonstrations. (CVPR Open Access)
Are model distillation attacks real?
Yes.
Major model providers now publicly report detecting large-scale extraction campaigns.
Anthropic reported more than 16 million exchanges associated with campaigns it attributed to three AI labs in February 2026. Google later reported seeing some coordinated campaigns exceeding 100 million prompts. (Antrópica)
The attribution of individual campaigns should still be distinguished from the broader technical fact that model extraction attacks occur.
Is every use of one LLM’s output to train another model an attack?
No.
Authorized knowledge distillation is common.
Developers may distill their own models, use appropriately licensed models or use provider-supported distillation features.
The key issues are authorization and contractual or licensing restrictions.
Can rate limiting prevent model distillation attacks?
Rate limiting helps, but account-level limits alone are insufficient against distributed attacks.
Attackers can spread traffic across many identities, credentials or network sources.
Providers therefore need campaign-level detection.
Can hiding chain-of-thought stop model extraction?
No.
It can remove a particularly valuable training signal, but final model outputs can still be used for black-box distillation.
Anti-distillation therefore needs multiple defensive layers.
What is the difference between model extraction and data extraction?
Model extraction attempts to reproduce a model’s functionality or characteristics.
Training-data extraction attempts to recover information that was present in or memorized from training data.
The security goals and mitigations differ.
Why are AI agents vulnerable to distillation?
Agents produce richer behavior than ordinary chat models.
Their trajectories may reveal planning, tool selection, recovery behavior and multi-step decision processes.
Those execution traces can themselves become valuable training data.
How should companies detect model distillation attacks?
Detection should combine account behavior, network infrastructure, payment or identity relationships, semantic analysis of prompts, token-consumption patterns, coordinated account behavior and long-term campaign analysis.
No single feature is sufficient.
Model Distillation Attack Defense Must Become a Standard Part of AI Security
The most important lesson from the rise of model distillation attacks is not that knowledge distillation has suddenly become malicious.
It has not.
The important lesson is that a useful AI API is simultaneously a product interface and an information interface into the model’s capabilities.
Every answer transfers some information.
At normal scale, that is the purpose of the service.
At adversarial scale, those answers can become a synthetic training corpus.
And when the extracted artifacts include reasoning traces, tool-use demonstrations or agent trajectories, the attacker may be acquiring something more valuable than isolated answers.
The progression is becoming clear:
Model theft
|
v
steal weights
↓
Black-box model extraction
|
v
learn input-output behavior
↓
LLM distillation attack
|
v
harvest rich demonstrations
↓
Chain-of-thought distillation
|
v
harvest reasoning artifacts
↓
Agent capability extraction
|
v
harvest complete execution trajectoriesThe defensive implication is equally clear.
AI companies cannot secure proprietary models only by encrypting checkpoints and protecting training clusters.
They must also secure inference.
They need identity intelligence, anti-fraud systems, semantic anomaly detection, account-graph analysis, reasoning-state isolation, API telemetry, provenance techniques and cross-provider threat intelligence.
And enterprises building their own specialized models should pay attention now rather than assuming this is only a frontier-lab problem.
A model does not need a trillion parameters to contain valuable intellectual property.
It only needs to perform a task that would be expensive for someone else to reproduce.
That is the fundamental security problem behind the model distillation attack.
In the next phase of AI security, protecting the model will no longer mean protecting only the file that contains its weights.
It will mean protecting the capabilities those weights expose every time the model answers a request. (DOI)
Portuguese 
English 
German 
French 
Spanish 
Japanese 
Korean 
Hindi 
Arabic 
Turkish 
Hebrew