TheDPDPAct.com – Official WhatsApp Channel
Stay updated with the latest DPDP Act news, compliance insights, updates, and resources.
Security guidance is almost always written from the defender's perspective. Here's what to implement, here's why it matters, here's the configuration to get right. That framing is useful, but it has a blind spot: it tells you what controls to put in place without telling you what a determined attacker does when they encounter them.
That gap matters more in AI API security than in most domains, because every major defensive technique against model extraction has a documented evasion counterpart. Knowing what those counterparts look like — how they work, what they require, and where they fail — is what separates a defense built to hold against real attacker behavior from one built to satisfy a checklist.
This piece walks through each of the five standard defensive controls, from the attacker's point of view.
Control One: Output Truncation
What It Does
Returning only a final prediction — "approved" or "declined," "spam" or "not spam" — rather than confidence scores or probability distributions gives an attacker less information per query. Without access to soft outputs, reconstructing the model's decision boundary is harder and requires more queries.
How Attackers Work Around It
Hard-label extraction has been an active research area for several years, and the results are not reassuring for defenders who treat output truncation as a sufficient control on its own.
Researchers have demonstrated successful extraction from top-1-only APIs through techniques that reconstruct the effective "soft" signal from many hard queries. One approach uses neighborhood sampling — systematically querying with variations of an input across a small neighborhood and estimating the underlying confidence from the pattern of label changes at the boundary. Another approach, the Smoothing Attack, sends each query multiple times with slight augmentations and infers the model's behavior from the distribution of hard labels returned, approximating a soft score from what the API presents as binary.
More directly, the D-DAE attack specifically trains a defense detection model alongside a label recovery model — first detecting that perturbation or truncation is active, then learning to recover the underlying label distribution despite it. The existence of specialized attacks designed to recognize and bypass defensive output shaping is a signal that treating output truncation as a solved defense significantly overstates what it provides.
What this means for how you build it: Output truncation raises the cost and complexity of extraction meaningfully, but doesn't eliminate it. The realistic frame is that truncation forces an attacker to run more queries and use more sophisticated techniques — which increases their cost and their detection surface — rather than blocking the attack outright. Combined with query monitoring, this matters. Alone, it matters less than it looks.
Control Two: Output Perturbation
What It Does
Adding calibrated noise to model responses — ideally targeted at out-of-distribution or boundary-probing queries — degrades the quality of a surrogate model trained on the perturbed responses. An attacker training on noisy labels builds a noisier copy.
How Attackers Work Around It
Differential querying is the most straightforward evasion: sending the same or similar inputs many times and averaging the responses. Uniform perturbation — noise applied consistently to all responses — is partially defeated this way, because the noise averages out while the underlying signal accumulates. With enough repeated queries, an attacker can recover a cleaner signal than any single response provides.
More sophisticated techniques like the Smoothing Attack apply this principle systematically: augmenting each query many times, collecting all the noisy hard or soft labels, and averaging across them to reconstruct a cleaner estimate of the model's true behavior.
The core tension here is well-documented in the research: stronger perturbation provides more extraction resistance but degrades accuracy for legitimate users. Weaker perturbation preserves user experience but provides limited resistance. The "adaptive misinformation" approach — perturbing more aggressively for queries that look like they're probing the decision boundary — provides better resistance while limiting user experience impact, but it requires correctly identifying which queries are anomalous. Attackers who use natural, in-distribution inputs as their query base specifically to avoid looking anomalous can partly defeat this targeting.
What this means for how you build it: Adaptive perturbation is significantly harder to evade than uniform perturbation. If your perturbation implementation applies uniform noise without distinguishing query types, an attacker with enough patience can average it away. The tuning of perturbation strength — enough to degrade extraction without visibly degrading user experience — requires ongoing calibration rather than a one-time configuration.
Control Three: Rate Limiting
What It Does
Restricting the number of queries a given account or API key can make within a given window limits how many training examples an attacker can collect, raising the cost and extending the timeline of an extraction campaign.
How Attackers Work Around It
Distributed campaigns. Rate limits applied per-account are straightforwardly evaded by spreading the query load across many accounts — different API keys, different email addresses, different payment methods in some cases. Academic and commercial cloud infrastructure makes this easier than it sounds.
Slow campaigns. Limits calibrated against burst traffic — per-second or per-minute throttles — don't constrain a campaign that deliberately runs at a fraction of the burst threshold across weeks or months. The Android malware classifier extraction documented in early 2026 ran for six months at a pace specifically designed to look like a moderately active integration.
There is also a specifically relevant evasion technique for more sophisticated defenders who try to detect cumulative volume: the S4L attack (Semi-Supervised Stealing with Self-Supervised Learning) combines cross-entropy loss with self-supervised learning on unlabeled data, dramatically reducing the number of queries needed to achieve a given level of extraction fidelity. If the attacker needs fewer queries, higher cumulative limits stop being a meaningful constraint.
What this means for how you build it: Per-account lifetime budgets and cumulative multi-day windows are meaningfully harder to evade than per-minute throttles. Cross-account correlation — tracking whether multiple accounts exhibit extraction-pattern behavior at the same time, even if each is individually within limits — is where rate limiting starts to interact with behavioral monitoring in ways that are genuinely difficult to work around at scale.
Control Four: Behavioral Monitoring
What It Does
Analyzing the statistical distribution of query sequences — looking for the systematic, methodical patterns that distinguish extraction campaigns from organic user behavior — to flag or throttle accounts exhibiting extraction-like activity.
How Attackers Work Around It
Query distribution camouflage is the most direct evasion strategy. Detection systems like PRADA work by identifying when a user's query distribution differs statistically from what legitimate users generate. Attackers who are aware of this can attempt to mimic legitimate query distributions: seeding their extraction campaign with queries drawn from real, in-distribution user data, or mixing extraction queries with organic-looking traffic from the same account.
The research on this evasion is candid: detection approaches that rely on query distribution analysis can be bypassed if adversaries specifically design their query patterns to resemble benign traffic. The D-DAE framework, mentioned earlier in the context of output perturbation evasion, is instructive — the same principle of explicitly modeling and evading the defense applies to detection as well.
However, camouflage has its own costs. Mixing real traffic with extraction queries extends the campaign timeline and increases the total query cost. Maintaining a statistically convincing legitimate-looking distribution across a systematic boundary-probing campaign is operationally complex. The evasion is possible but not free.
What this means for how you build it: Account-level behavioral monitoring — tracking patterns across an account's full history rather than a sliding time window — is harder to consistently camouflage than query-window analysis. Multi-signal detection, combining query distribution analysis with input diversity, session characteristics, and access timing, raises the cost of convincing camouflage further. No single behavioral signal is evasion-proof; the layering of several weaker signals is what creates meaningful resistance.
Control Five: Watermarking
What It Does
Embedding verifiable signals into model outputs or behavior that persist through the distillation process — allowing an original model owner to prove a suspected surrogate was derived from their model.
How Attackers Work Around It
Watermark removal is a genuinely active research and attack area. The NeurIPS 2024 Invisible Watermark Removal Challenge demonstrated near-perfect watermark removal (95.7% removal rate) on image watermarks using an adaptive VAE-based approach in the beige-box scenario, with high-quality preservation of the underlying content. For model behavioral watermarks, established removal techniques include CLP (Channel Lipschitzness Pruning) — a data-free backdoor removal method that prunes model parameters associated with backdoor triggers — and various fine-tuning-based removal approaches that can attenuate or eliminate trigger-response pairs through retraining on clean data.
The honest picture is that no current watermarking scheme is fully robust against a determined, sophisticated attacker with resources for active removal. More recent schemes like Neural Honeytrace and MEA-Defender are specifically designed to maintain watermark stability under CLP and similar removal attacks, and research on watermarking for extraction resistance is advancing meaningfully. But the fundamental dynamic is clear: watermarking is most robust as an evidentiary control when an attacker doesn't know it exists and therefore doesn't attempt removal, and least robust against an attacker who specifically targets it.
What this means for how you build it: Watermarking provides its most reliable value as a legal and evidentiary backstop — proof of derivation in a dispute where the attacker wasn't specifically trying to remove your watermark. As a primary theft deterrent against a sophisticated, informed attacker, it should be treated as one layer in a stack rather than the stack itself. Schemes that use multiple independent watermarking signals, or that combine output-level and weight-level watermarks, are harder to completely remove than single-signal approaches.
The Arms Race Reality
The pattern across all five controls is consistent: every defense has a documented evasion, and every evasion has a counter-adaptation, and the research community on both sides is actively advancing. This is the honest state of the field, and it matters for how organizations approach this problem.
The practical implication isn't that defenses are pointless. It's that defenses need to be understood for what they actually provide — increased attacker cost, extended timelines, detection surface, and evidentiary backstops — rather than guaranteed prevention. And it means that any defense configuration that was adequate against the attacker tooling of twelve months ago needs to be periodically re-evaluated against current evasion research.
Understanding what attackers do when they hit your defenses isn't a reason to be discouraged. It's the reason to test each control against realistic adversarial behavior rather than assuming implementation equals effectiveness — which is exactly where the companion piece to this one, on testing these defenses in practice, picks up.
At ILLUME Intelligence, this attacker-perspective analysis is the lens our AI API security assessments are built around — not just reviewing configurations, but simulating the realistic evasion techniques an informed attacker would use against each control, and identifying where the current stack holds and where it doesn't. Reach out to ILLUME to understand where your defenses actually stand against current attacker behavior.
