TheDPDPAct.com -
Official WhatsApp Channel
Stay updated with the latest DPDP Act news, compliance insights, updates, and resources.
Join Our WhatsApp Channel →In November 2025, Microsoft's security researchers described an attack that never decrypts a single byte. A "Whisper Leak" observer, such as someone on the same Wi-Fi network or at an internet provider, watches the sizes and timing of encrypted packets as an AI model streams its answer. From that pattern alone, the researchers showed they could tell whether a user's prompt concerned a chosen topic. In controlled tests, many of their classifiers scored above 98% on the AUPRC measure. If the rhythm of a response can give away its subject, imagine what the contents, errors, and metadata can do. Every reply from an AI API is also a message to anyone probing it. This post covers AI API information leakage: what responses reveal, how attackers use it, and how to test your own.
What AI API Information Leakage Means?
Leakage here means any information an API response discloses beyond what you intended: the answer text, but also error messages, extra fields, headers, timing, structure, and the way the system refuses. Individually these look harmless; together they let an attacker map your system.
AI adds new channels to the familiar API ones. The OWASP Top 10 for LLM Applications names system prompt leakage and sensitive information disclosure as distinct risks, and the OWASP API Security Top 10 still warns about broken object property level authorization, where an API returns more properties than a caller should see, and security misconfiguration, which includes verbose error messages.
Six Things a Response Can Reveal
1. The system prompt and internal rules.
OWASP's guidance is direct: the system prompt "should not be considered a secret, nor should it be used as a security control." The real harm comes from what teams put in it: API keys, database credentials, architecture details, business limits such as transaction or loan thresholds, descriptions of content to reject, and role structures. Each hands an attacker a plan.
2. Other people's data.
Assistants that retrieve documents can return content a user should never see, or cite sources that expose it. If retrieval is not filtered by tenant and user, the answer itself is the leak. OWASP lists vector and embedding weaknesses for this reason.
3. Too many fields.
AI back ends often return the full object and rely on the interface to hide the rest. An attacker reads the raw response, not the interface.
4. Infrastructure details.
Stack traces, framework names, model identifiers, internal hostnames, and debug fields tell an attacker what they are facing and where the weaknesses are likely to be. OWASP's misconfiguration category calls out verbose errors explicitly.
5. The shape of your guardrails.
How and when the system refuses can show what its filters look for. A refusal message that names the rule it triggered is a map of what to work around.
6. The model itself.
Researchers including Carlini et al. (ICML 2024) showed that with typical API access, an attacker could recover the embedding projection layer of a production language model, up to symmetries. They reported extracting the full projection matrix of OpenAI's Ada and Babbage models for under $20, and estimated under $2,000 in queries for GPT-3.5-turbo. Our own inference is that limiting which low-level outputs your API exposes is a sensible lever, though the specifics depend on your model and provider.
The Side Channel Most Teams Have Not Considered
Whisper Leak is worth understanding even if you only consume a third-party model. Per Microsoft, an observer who cannot decrypt the traffic can still record the sizes and timing of encrypted packets during a streamed response, and that sequence is enough to infer the topic of the prompt.
Providers have responded. OpenAI added an obfuscation field to streaming responses, Azure mirrored it, and Mistral added a similar parameter. For your own products, check which streaming protections your provider offers, and consider non-streaming responses for the most sensitive topics, which Microsoft lists among its advice.
How Attackers Use What They Learn?
Information leakage is reconnaissance. The attacker maps your prompt and guardrails, learns which fields and identifiers exist, and notes the stack. That feeds the next stage: crafting a prompt injection that avoids your filters, probing another tenant's identifiers, or attacking a component that an error message named. It is the first link in the kind of chain we described in our post on vulnerability chaining.
How to Test for AI API Information Leakage?
A security test should treat every response as evidence. We would cover:
* Error probing: malformed, oversized, wrongly typed, and unusual-character inputs, to see what errors expose.
* Prompt extraction attempts: direct and indirect, to see what internal instructions come back.
* Cross-tenant and cross-user retrieval: questions designed to surface other people's documents, plus checks on citations and source links.
* Response field audit: compare each response with what the caller needs. Anything extra is a finding.
* Metadata and header review: model names, versions, usage counts, internal identifiers, and debug fields.
* Advanced parameters: are optional low-level outputs available to ordinary callers, and do they need to be?
* Streaming and timing: what an observer could infer from response patterns for sensitive topics.
* Refusal behaviour: whether refusal messages reveal filter rules.
* Extraction patterns: whether systematic, high-volume querying is rate-limited and noticed.
How to Reduce It?
* Keep secrets out of prompts. Credentials and business rules belong in systems the model cannot read directly, as OWASP advises.
* Enforce authorisation outside the model. OWASP advises enforcing controls outside the LLM, not through prompt instructions.
* Return the minimum. Build a response schema that includes only the fields the caller needs.
* Show users generic errors and log the detail internally.
* Scope retrieval by tenant and user before the model sees any document.
* Strip or normalise metadata that has no business reason to leave your system.
* Check output with an independent guardrail for secrets and personal data, instead of trusting instructions alone.
* Rate-limit and monitor for systematic probing.
Where ILLUME Intelligence Can Help?
ILLUME Intelligence's AI Penetration Testing (AI VAPT) and LLM Security Testing services examine your AI APIs the way an attacker would: probing errors, prompts, retrieval, fields, and metadata for what they give away. Our Secure AI Development service helps teams build minimal responses and external guardrails into the design from the start. A scoped review of one AI endpoint is a practical first step.
The Bottom Line
An AI API talks back, and attackers listen to more than the answer. Treat every part of a response, including the errors, the fields, the metadata, the timing, and the refusals, as something that may help someone who is not your customer. Return less, hide detail, enforce access outside the model, and test what an outsider can learn.
