In our previous piece, we introduced model extraction and distillation attacks through the lens of the OpenAI–DeepSeek dispute — a case that moved this risk from academic theory into boardroom reality. In this piece, we go one level deeper. If you're responsible for securing an AI product, you need to understand not just that this attack exists, but how it actually works — because the mechanics dictate the defense.
The good news: this isn't exotic, zero-day hacking. It's a well-documented, patient, almost mundane process. That's precisely what makes it dangerous — it hides in plain sight, disguised as ordinary API traffic.
The Model as a Black Box — And Why That's Not Protection
Most organizations deploying AI treat the model itself as a black box: customers or partners send inputs, the model returns outputs, and the internal weights, architecture, and training data stay hidden. That feels secure. It isn't.
The foundational research on this problem dates back further than most security teams realize. In 2016, researchers Florian Tramèr and colleagues published Stealing Machine Learning Models via Prediction APIs at the USENIX Security Symposium — one of the top venues in the field. They demonstrated that models deployed on real, commercial machine-learning-as-a-service platforms, including BigML and Amazon Machine Learning, could be reconstructed with near-perfect fidelity using nothing more than standard API access. No breach. No insider. Just carefully designed queries and the ability to observe what came back.
Nearly a decade later, the same underlying principle scaled up to reasoning-capable large language models — this is the mechanism at the heart of the DeepSeek allegations we discussed in Part 1.
The Attack Lifecycle
A model extraction campaign generally follows four stages:
1. Querying the target. The attacker sends a large volume of inputs to the model's public or partner-facing API — this can be through legitimate paid access, a free tier, or a trial account. Nothing about an individual request looks abnormal.
2. Harvesting rich outputs. This is the stage where API design choices matter enormously. A model that returns only a final label or answer leaks relatively little information per query. A model that returns confidence scores, full probability distributions, or token-level logits leaks far more — each response gives the attacker a clearer picture of the decision boundary sitting behind the interface. Research has consistently shown that richer outputs dramatically accelerate extraction.
3. Building the training set. Every input-output pair becomes a labeled training example. Over enough queries, this dataset starts to describe the shape of the target model's behavior across the input space.
4. Training and refining the surrogate. The attacker trains their own "student" model on the harvested data. Increasingly, attackers don't brute-force this stage — they use active learning techniques, deliberately selecting queries that probe uncertain or high-value regions of the model's decision boundary, extracting maximum information per query while minimizing cost and detection risk. Recent academic work has demonstrated surrogate models reaching 69–96% of a target model's accuracy using as few as roughly 1,000 to 7,000 carefully chosen queries — a small fraction of what brute-force querying would require.
The result is a model that behaves like the original closely enough to be commercially useful, competitively threatening, or usable as a launchpad for further attacks — all without the attacker ever touching your infrastructure, credentials, or source code.
A Worked Example: Cloning a Credit-Risk Model
Abstract attack stages are easier to reason about with a concrete scenario. Consider a fintech company that exposes a credit-risk scoring model through an API — a common setup, since many lenders offer "check your approval odds" tools to partners or applicants before a full application.
A competitor wants to replicate the pricing logic without the years of underwriting data and model tuning that went into it. Here's roughly how that campaign would unfold:
* Reconnaissance. The attacker signs up for standard API access — no different from any legitimate partner integration — and studies what the response payload contains. If it returns not just "approved/declined" but a numerical risk score or probability, that's the opening they need.
* Query design. Instead of submitting random applicant profiles, the attacker systematically varies inputs — income, credit history length, existing debt — one variable at a time, watching how the risk score shifts. This is exactly the kind of decision-boundary mapping that active learning-based extraction techniques are built to automate at scale.
* Data collection. Each (applicant profile, risk score) pair becomes a training example. A few thousand well-chosen queries — well within the volume a legitimate-looking integration would generate — is often enough to start seeing the shape of the underlying scoring logic, consistent with the query-efficiency figures cited above.
* Surrogate training. The attacker trains their own risk model on the harvested pairs. It doesn't need to be a perfect replica — a surrogate that agrees with the original 85–90% of the time is already commercially damaging, because it lets a competitor price aggressively against the original lender or identify which applicant profiles the original model tends to favor.
Nothing in this sequence trips a conventional intrusion detection system. Every request authenticated correctly. Every payload was well-formed. The only signal that something was wrong would have been in the pattern of the queries — a level of monitoring most API security programs simply aren't built to do today.
This scenario is illustrative, not a specific disclosed incident — but every technique in it maps directly to published, peer-reviewed extraction research, and the underlying risk is not hypothetical for regulated industries. A 2025 systematic survey of model extraction attacks and defenses specifically flags finance as one of the sectors most exposed to this threat, noting that attackers have demonstrated the ability to reconstruct proprietary credit-scoring and risk-assessment models through exactly this kind of systematic API querying.
Reported Incidents: This Has Already Happened
Model extraction is sometimes dismissed as a lab curiosity — something researchers demonstrate in papers but attackers rarely bother with in practice. The record doesn't support that view.
Beyond the DeepSeek case covered in Part 1, security researchers have documented a striking real-world incident involving a $35 million voice fraud case in the UAE, in which attackers combined an extracted or approximated voice-recognition model with stolen biometric data to defeat authentication controls and authorize fraudulent transactions at scale. The case illustrates a pattern security teams should take seriously: model extraction is rarely the entire attack on its own — it's frequently the enabling step that makes a much larger, much costlier fraud possible.
Academic surveys tracking this attack class have also documented successful extraction attempts across a wide range of real deployed systems — from image classifiers running on commercial cloud ML platforms to natural language processing APIs and even models embedded in vision-language and tabular-data systems. The consistent theme across this body of evidence: wherever a model is exposed through a queryable interface, extraction has been demonstrated to be practically achievable, not just theoretically possible.
Why Query-Efficiency Research Should Concern You
It's tempting to assume this threat is limited to attackers with massive query budgets and correspondingly large, visible API bills. That assumption is increasingly outdated. A meaningful and growing body of academic research is focused specifically on making extraction cheaper and quieter:
* Confidence-based filtering — prioritizing queries where the substitute model is least certain, sharpening the copy with fewer requests
* Active learning-based sampling — dynamically adjusting which inputs to query next based on what's already been learned
* Public-data-driven extraction — using publicly available, unrelated datasets to query the target rather than needing access to anything resembling the original training data
Each of these approaches has the same goal: extract more functional value per dollar spent, and per request logged. That's precisely what makes low-and-slow extraction campaigns hard to catch with conventional monitoring — they're deliberately engineered to look unremarkable.
The Second-Order Risk: Adversarial Transferability
Extraction rarely stops at "we cloned the model." Once an attacker has a working local surrogate, they gain something arguably more dangerous: a private sandbox to attack in.
This matters because of a well-established property in machine learning security called adversarial transferability — inputs specifically crafted to fool a surrogate model frequently fool the original target model too, even though the attacker never had direct access to it. In practice, this means an extracted model isn't just a cheap knockoff of your product. It can become the R&D environment an attacker uses to develop inputs that manipulate, misclassify, or jailbreak your live production system — refined entirely offline, away from your monitoring, and then deployed against the real thing in a single, well-aimed attempt.
This is the connection point security teams often miss: model extraction isn't only an intellectual property problem. It's frequently the reconnaissance phase of a larger attack chain.
What This Means for How You Should Be Testing
If your organization exposes any model — a classifier, a recommendation engine, an LLM-powered feature — through an API, the uncomfortable but necessary question is: has anyone actually tried to extract it, under controlled conditions, so you know what's possible before an adversary finds out first?
This is precisely the kind of scenario a structured red team or attack simulation engagement is built to answer. Rather than treating "extraction risk" as a theoretical checkbox, a properly scoped exercise attempts real extraction against your production or staging API — measuring how many queries it takes to build a meaningfully accurate surrogate, which endpoints leak the most useful information, and whether your monitoring would ever notice.
Coming Up Next
Understanding the attack mechanics is only half the picture. In the next piece in this series, we turn to the defender's side: the specific API design choices — output granularity, rate limiting architecture, query pattern monitoring, and watermarking — that determine whether your model is an easy target or a genuinely hard one. Some of these fixes are inexpensive and can be implemented in days. Others require rethinking how your API is built from the ground up.
Either way, the earlier this conversation happens relative to your product launch, the cheaper it is to have.
