How to Detect AI Model Extraction Attacks Before It's Too Late

Organisations invest heavily in the assets that make an AI model valuable — proprietary training data, architecture decisions, fine-tuning, evaluation, and the engineering time behind all of it.

 

Yet the moment that model is exposed through an API, one of its most important assets — the model's behaviour — becomes observable to anyone with legitimate access. That single fact creates a security problem traditional application monitoring was never built to recognise.

 

An attacker doesn't need your model files, your cloud storage, or your source code. By systematically querying an exposed AI system and analysing its responses, an attacker can attempt to build a substitute model that reproduces part of the original's functionality.

 

This is model extraction — and every documented case of it shares the same uncomfortable trait: request by request, it looks almost exactly like legitimate traffic.

 

That reframes the real question security teams need to be asking. Not "can our model be extracted" — research has settled that question repeatedly — but:

Can your security team recognise the extraction attempt while it's happening?

 

 

What an AI Model Extraction Attack Actually Is?

NIST's Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (2025) defines model extraction as a privacy attack that extracts details of a model's architecture and/or parameters, placing it within the broader landscape of adversarial machine learning.

 

In practice, the attacker interacts with a model through its API:

* Sends carefully selected inputs

* Records the model's responses

* Analyses those responses to understand model behaviour

* Trains a substitute model on what it's learned

* Repeats and refines the process to improve similarity over time

 

Crucially, the attacker doesn't need to know the original architecture, training data, or parameters. The foundational 2019 Knockoff Nets research demonstrated exactly this — functionality stealing from black-box models using nothing but queried inputs and the predictions returned.

 

Since then, the trend has moved firmly toward efficiency:

* A 2024 IJCAI study reported a model-stealing approach achieving comparable attack performance using roughly half the query budget of competing methods.

* 2023 research demonstrated query-efficient extraction against text classification models even under hard-label conditions — where the API returns only a final label, not a confidence score.

 

Fewer queries mean an attack is harder to distinguish from ordinary usage. It also means output restriction alone, while valuable, is not a complete answer.

 

 

Why Volume-Based Monitoring Misses It?

The common instinct is to look for an obvious signal: "the attacker is sending too many requests."

But a model-stealing adversary wants useful queries, not necessarily a large number of them. Research into model-stealing detection has flagged a genuinely difficult complication: legitimate users themselves can generate repeated, similar, or unusual query patterns — which makes simple volume thresholds prone to false positives.

 

Consider the reality security teams actually face:

* A legitimate high-usage customer may send thousands of requests a day

* An attacker may need far fewer, highly informative requests to build a working surrogate

* Multiple colluding accounts may distribute queries to stay under any single threshold

* Attackers may deliberately randomise timing to avoid looking automated

* Ordinary users can naturally produce repetitive-looking query sequences

 

The conclusion is unavoidable:

High request volume does not equal extraction. Normal request volume does not equal safety.

 

Effective detection has to examine behaviour, context, and pattern — not just count.

 

 

The Statistical Techniques Behind Modern Detection

Academic security research has spent close to a decade building out exactly this capability, and the results are concrete rather than theoretical.

 

PRADA (Juuti et al., 2019) (Protecting against DNN Model Stealing Attacks ) is the foundational technique. It works by analysing the distribution of consecutive queries from a given user — legitimate usage tends to follow a relatively stable statistical pattern, while systematic extraction querying disrupts that pattern in a detectable way. In its original evaluation, PRADA detected all prior published extraction attacks with no false positives.

 

Later research has both extended and stress-tested that approach:

* SEAT (Similarity Encoder by Adversarial Training) introduced an adversarially trained similarity encoder specifically to remain robust against attackers who try to mimic benign query distributions.

 

* HODA (Hardness-Oriented Detection of Model Extraction Attacks) took a different statistical route, analysing the histogram of minimum distances between new and prior queries — while its own researchers noted that PRADA's original method struggles when an attacker queries using natural, real-world samples rather than synthetic ones.

 

* FDINet (Feature Distortion Index Network) more recently, reported detection accuracy reaching 100% with a false positive rate close to zero against several attack types — and successfully flagged malicious behaviour from batches as small as 50 queries, a batch size at which both PRADA and SEAT, in direct comparison, failed to detect the attack at all.

 

The honest caveat: this is not a solved problem. Published research has shown several detection methods remain vulnerable to adaptive attackers who deliberately disguise their query distribution once they suspect what kind of monitoring they're facing.

 

Detection has matured from "does this work at all" to "which combination of techniques holds up against which class of attacker" — a meaningfully better question, but not yet a settled one.

 

 

Practical Signals Worth Watching

Translating this research into an operational monitoring program comes down to watching for combinations of the following, rather than any single trigger.

 

* Unusual query distribution. Are queries systematically covering different classes or edge cases? Are they concentrated suspiciously close to decision boundaries, or repeatedly probing outputs the model is least confident about? This requires first establishing a genuine baseline of legitimate behaviour — without one, "unusual" has no meaning.

 

* Systematic probing versus organic usage. The distinction between a normal user and an attacker is rarely volume alone. It's intent reflected through behaviour: input similarity, response similarity, query sequencing, and how all of that changes over time for a given account.

 

* Distributed, cross-account activity. Extraction doesn't have to come from one obvious account. Monitoring needs to look across:

- User accounts and API keys

- IP addresses and sessions

- Tenants and applications

- Device fingerprints, where available

Account-level monitoring alone is insufficient against an attacker who spreads activity deliberately.

 

* Deviation from historical usage patterns. An account that typically makes 200 calls a day against three endpoints, and suddenly generates thousands of varied inference requests across multiple endpoints at unusual times, should raise a risk signal — even if it never crosses a fixed volume threshold. This is where behavioural analytics outperforms static rules.

 

* What your outputs actually expose. Confidence scores, probability distributions, logits, and detailed rankings all increase the information available per query. Monitoring what level of output detail is being requested, and by whom, is as important as monitoring the inputs themselves.

 

Recent research reflects the same conclusion: a 2025 AAAI paper (Defense Against Model Stealing Based on Account-Aware Distribution Discrepancy) proposed account-aware distribution analysis specifically to identify malicious users by combining signals rather than scoring queries in isolation. The strongest detection posture comes from layering several weaker signals, not perfecting one strong one.

 

 

A Structured Response Model: Prevent, Detect, Respond, Recover

A mature AI security architecture treats extraction as a full lifecycle, not a single control.

Prevent

- Authentication and authorisation

- Rate limiting and API quotas

- Output minimisation

- Access controls

 

Detect

- Behavioural analytics

- Query-pattern analysis

- Account-level anomaly detection

- API telemetry, model fingerprinting, cross-account correlation

 

Respond

- Increase scrutiny on flagged accounts

- Throttle suspicious activity

- Challenge or restrict access

- Revoke credentials where justified

- Preserve forensic evidence and investigate potential compromise

 

Recover

-  Assess whether extraction actually occurred

- Determine the extent of replication

- Review which outputs and interfaces were exposed

- Strengthen controls and re-test the system

 

OWASP's guidance on securing AI model operations reinforces this same layered posture — recommending monitoring, logging, rate limiting, and access auditing together, not any one of them as a standalone fix.

 

 

Detection Has to Be Tested, Not Assumed

Rate limiting remains genuinely useful. But it is one control within this larger architecture, not a substitute for it — a sufficiently patient attacker can often stay below any threshold set generically rather than tuned to actual traffic.

 

The only reliable way to know whether your monitoring would catch a real extraction attempt is to simulate one under controlled, authorised conditions — the discipline of AI Red teaming. A structured exercise, answers questions a policy review simply can't:

 

* How many queries does extraction actually require against this specific API?

* What does the model expose that it shouldn't?

* Which controls slow an attacker down, and which ones actually detect them?

* How quickly would the response team notice and react?

* Can an attacker realistically stay under the thresholds currently in place?

 

That shifts the conversation from "do we have AI security controls" to the question that actually matters:

 

Do our AI security controls detect an attack in progress?

NIST's taxonomy work makes the same point at a structural level. Assessing an AI system properly means extending the security review beyond the familiar infrastructure-application-API chain, to include the model itself, its inference interface, the behaviour of queries against it, and whether monitoring and response are actually built to catch what a determined, patient attacker looks like.

 

 

The Bottom Line

Model extraction is no longer a theoretical concern confined to research papers. The techniques are published, refined, and increasingly query-efficient — which means the traffic behind them looks less like an attack every year, not more. That makes this fundamentally a behavioural security problem, not a perimeter one.

 

The question worth sitting with is simple: if an extraction campaign were running against your API today, using techniques that have been publicly documented for close to a decade, would anything currently watching actually notice — or would the first sign of trouble be a competitor's product that looks suspiciously familiar, months after the queries stopped?

 

ILLUME’s AI Security and Red Teaming practice helps organisations answer that question directly — assessing AI systems for extraction risk, testing exposed inference APIs under realistic adversarial conditions, evaluating whether existing detection capabilities actually work, and strengthening controls before an attacker gets the chance to find the gap first.

---

Comments

---

No Comments Found.