In January 2025, the AI industry watched a Chinese lab called DeepSeek release a reasoning model, R1, that performed on par with OpenAI's best models — at a fraction of the reported cost. Within days, NVIDIA lost hundreds of billions of dollars in market value in a single trading session. The market's question was simple: how?
OpenAI's answer, delivered in a memo to Congress a few weeks later, wasn't about a data breach or a stolen laptop. It alleged that DeepSeek had systematically queried its models and used the responses to train a competing system — a technique known as distillation. Microsoft's security team reportedly traced large-scale extraction activity to accounts linked to DeepSeek and flagged it to OpenAI before access was cut off.
Whatever the final legal outcome of that dispute, it put a name to a risk that security teams need to start taking seriously: your AI model's API can be quietly cloned by anyone with a query budget and enough patience.
What Is a Model Distillation Attack?
Distillation itself isn't inherently malicious. It's a well-established machine learning technique, first proposed by Geoffrey Hinton and colleagues back in 2015, where a smaller "student" model is trained to mimic a larger "teacher" model. It's used legitimately all the time to make expensive models cheaper and faster to run.
The attack version of distillation repurposes the same idea against a model you don't own — usually one you're only able to reach through an API. Here's the basic flow:
1. Query the target model. The attacker sends a large number of inputs to a public-facing AI system — a chatbot, a classification API, a recommendation engine.
2. Collect the outputs. Responses, confidence scores, or probability distributions are logged.
3. Train a student model. Those input-output pairs become training data for the attacker's own model.
4. Refine and repeat. With enough queries, the student model starts to approximate the teacher's behavior closely enough to be commercially useful — without ever touching the original weights, architecture, or training data.
No firewall is breached. No credentials are stolen. From the target system's point of view, it just looks like unusually heavy — but technically legitimate — API traffic.
Why This Should Be on Your Risk Register
It's a direct hit to your intellectual property. Training a frontier-grade model can cost tens or hundreds of millions of dollars in compute, data, and research time. A well-executed extraction campaign lets a competitor approximate that capability for a small fraction of the investment — essentially free-riding on someone else's R&D spend. This is exactly the accusation at the center of the OpenAI-DeepSeek dispute, and it's why major AI providers including OpenAI, Anthropic, Mistral, and xAI have all added explicit anti-distillation clauses to their terms of service.
It enables a second wave of attacks. Once an attacker has even an approximate clone of your model, they can experiment on it freely and offline — probing for weaknesses, crafting adversarial inputs, or looking for biases and edge cases without triggering your rate limits or monitoring. Attacks refined against the clone frequently transfer back to the original system, a well-documented property of machine learning models known as adversarial transferability.
It can bypass your commercial model entirely. If your business model depends on API usage fees or a paid product tier, a successful clone lets competitors offer similar functionality without ever paying for it — undercutting your pricing and your moat in one move.
It may carry a privacy dimension too. In some cases, distilled models can retain faint traces of the sensitive data the original model was trained on, opening the door to further scrutiny under data protection obligations. We'll cover this in more depth later in this series.
This Isn't a Hypothetical Problem Anymore
For years, model extraction was mostly discussed in academic security papers — theoretical, if credible. The DeepSeek episode changed that. It demonstrated, at a geopolitical scale, that:
* Extraction attacks are practical, not just theoretical
* They can be detected through unusual query pattern monitoring (which is how Microsoft's security team flagged the activity)
* They carry real commercial and legal consequences, now playing out in boardrooms, regulatory filings, and congressional testimony
If a state-affiliated lab with significant resources found it worthwhile to attempt this against one of the best-funded AI companies in the world, it's reasonable to assume smaller, less publicized attempts are happening against less prominent AI products every day — including yours, if you're serving any model through a public or partner-facing API.
The Uncomfortable Question Every AI-Deploying Business Should Ask
If someone queried your AI product a few hundred thousand times in a systematic, structured way, would you notice? And if you did notice, would you know how much of your model's functional value they'd already managed to extract?
For most organizations building or deploying AI products today, the honest answer is: probably not yet. Rate limiting and basic API security are usually in place, but they were designed with traditional attacks in mind — data scraping, credential stuffing, denial of service. They weren't designed to detect a slow, patient campaign to reverse-engineer your model's behavior one query at a time.
Where This Series Goes Next
This is the first in a series exploring model extraction and distillation risk from a practical, security-first perspective. In the pieces ahead, we'll break down exactly how these attacks work under the hood, what API design choices make a model easier or harder to clone, the privacy risks that come with data leakage through distillation, and how to think about auditing your AI systems the same disciplined way you'd audit your network.
AI products are, at their core, APIs — and APIs get attacked. The difference is that most organizations have spent years hardening their infrastructure against conventional threats, and comparatively little time thinking about what it means to protect a model's behavior itself as a business asset.
That's the gap this blog series is here to help close. If your organization is building, deploying, or relying on AI models and you're unsure what your exposure looks like, that's a conversation worth having before an incident forces it.
