AI Security Review: Threat Modelling LLM Apps Before Launch

Most AI teams test their LLM application for accuracy before launch. Fewer test it for security. That gap matters, because a large language model application introduces attack surfaces that don't exist in traditional software — and by the time a prompt injection exploit or a data leak surfaces in production, the fix is far more expensive than a pre-launch review would have been.

This article sets out a practical, non-hypey approach to threat modelling LLM applications before launch, mapped to the OWASP Top 10 for Large Language Model Applications, with an Australian privacy lens and a worksheet you can adapt for your own review.

What is an AI security review for LLM applications?

An AI security review is a structured assessment of how a large language model application could be misused, manipulated, or exploited, conducted before the system goes live. It examines the model, the prompts, the tools the model can call, the data it retrieves, and the tenancy boundaries around it — not just whether the model gives correct answers. Threat modelling is the specific technique of systematically identifying, in advance, what could go wrong and who could make it go wrong, so controls can be designed before an incident forces the issue.

For LLM applications, this differs from a conventional web application security review because the attack surface includes natural language itself. Inputs that look like ordinary conversation can carry adversarial instructions, and outputs that look like helpful answers can carry leaked data.

Why does threat modelling matter before launch, not after?

Threat modelling before launch is cheaper and more effective than incident response after launch, because it lets you design controls into the architecture — system prompts, tool permissions, retrieval boundaries, output filtering — rather than bolting them on reactively. Once an LLM application is live and integrated with real customer data and real business tools, the cost of retrofitting isolation or access controls grows substantially, and any incident in the interim carries reputational and regulatory consequences.

Production AI is genuinely hard, and the infrastructure around the model — prompting, retrieval, evaluation, monitoring — typically matters more to real-world outcomes than which model you choose. Security controls belong in that same infrastructure layer, reviewed with the same rigour as accuracy and latency.

What is the OWASP Top 10 for LLM Applications?

The OWASP Top 10 for LLM Applications is a community-maintained framework published by the Open Web Application Security Project that catalogues the most significant security risks specific to large language model applications. It's a useful starting checklist for a pre-launch review because it names risks in language that both engineers and security teams already understand, rather than requiring a bespoke taxonomy for every project.

The table below maps four of the most operationally significant OWASP LLM risk categories to what they look like in a typical production application, and the review question you should be asking before launch.

OWASP LLM Risk Category	What it looks like in practice	Pre-launch review question
Prompt Injection	Malicious instructions embedded in user input, documents, or retrieved content override intended system behaviour	Can untrusted text (user input, uploaded files, retrieved documents) change what the model does, not just what it says?
Sensitive Information Disclosure	The model reveals system prompts, other users' data, or internal business logic in its responses	If asked directly or indirectly, can the model be made to disclose data it shouldn't have access to in that context?
Excessive Agency	The model is granted tool access or permissions broader than the task requires	Does every tool the model can call have the minimum permission needed, with human approval for high-impact actions?
Supply Chain / Data Poisoning	Training data, fine-tuning data, or retrieval sources are manipulated to bias or corrupt model behaviour	Are the sources feeding your RAG pipeline or fine-tuning data trusted, versioned, and monitored for tampering?

This is a starting point, not a complete audit — the full OWASP framework covers ten categories in more depth, and a proper review should work through all of them against your specific architecture.

How does prompt injection put LLM applications at risk?

Prompt injection is an attack where an adversary crafts input — direct user text, or content embedded in a document, email, or webpage the model reads — designed to override the application's intended instructions. Because the model cannot reliably distinguish between "trusted instructions from the developer" and "untrusted content it is processing," a cleverly worded document can cause the model to ignore its system prompt, exfiltrate data, or take unintended actions.

The practical defence isn't a single filter — it's layered: treat all retrieved or user-supplied content as untrusted, separate instructions from data wherever the model architecture allows it, constrain what the model can do downstream of any untrusted input, and log and monitor for anomalous instruction-following. None of these controls make prompt injection impossible; they reduce blast radius, which is the realistic goal.

How can tool use lead to data exfiltration?

Tool use lets an LLM call external functions — searching a database, sending an email, hitting an API — to complete tasks beyond text generation. Data exfiltration risk arises when a manipulated model uses a legitimate tool for an illegitimate purpose: for example, a customer support bot with email access being tricked into sending internal data to an external address, or a coding assistant with file-system access being coaxed into reading files outside its intended scope.

A pre-launch review should map every tool the model can call, the data each tool can touch, and what happens if the model calls that tool with adversarial intent. The question to answer for each tool isn't "does this work correctly," but "what's the worst thing this tool could be used for if the model were compromised."

What is RAG poisoning and how do you defend against it?

RAG poisoning is an attack that manipulates the documents or data sources a retrieval-augmented generation system draws from, so the model retrieves and acts on corrupted or malicious content as if it were trustworthy. Because RAG systems are explicitly designed to inject external content into the model's context, they are a natural target: if an attacker can get a document into the retrieval corpus — through a shared drive, a support ticket, a scraped webpage — they can potentially steer the model's behaviour for any user who triggers that retrieval.

Defending against this starts with treating your retrieval corpus as an attack surface with its own access controls, provenance tracking, and content validation — not just a convenient store of "your company's knowledge." Our post on RAG vs fine-tuning covers the architectural trade-offs in more depth, and those same design decisions shape your exposure to this risk.

Why does tenant isolation matter in multi-tenant LLM systems?

Tenant isolation is the set of controls ensuring that one customer's data, prompts, or model context cannot be accessed or influenced by another customer in a shared, multi-tenant system. LLM applications complicate isolation because context windows, embeddings, caches, and fine-tuning data can inadvertently blend information across tenants if the architecture wasn't deliberately designed to prevent it.

A pre-launch review should trace data flow end to end for a multi-tenant deployment: where is context built, where are embeddings stored, is there any shared cache or shared fine-tuning process, and can a crafted prompt from Tenant A ever surface content belonging to Tenant B. This is architecture work as much as it is security work, and it belongs in the same conversation as your broader data infrastructure design.

What does Australian privacy law require of LLM applications?

Australian organisations handling personal information through an LLM application remain subject to the Privacy Act 1988 (Cth) and the Australian Privacy Principles (APPs), regardless of whether that information passes through a third-party model provider. The Office of the Australian Information Commissioner (OAIC) has published guidance noting that entities using AI products, including generative AI, must still meet their APP obligations around collection, use, disclosure, and security of personal information — the involvement of an AI vendor does not transfer that responsibility away.

Practically, this means a pre-launch security review should confirm what personal information could reach the model (directly or via retrieval), whether that data leaves Australia through the model provider's infrastructure, whether the Notifiable Data Breaches scheme obligations are understood if an LLM-related incident exposes personal information, and whether your privacy policy accurately describes AI use to the standard the APPs require. This is a compliance question as much as a security one, and it's worth resolving before launch rather than after a regulator asks.

A threat-modelling worksheet for pre-launch review

A practical worksheet forces the same discipline as a formal audit without requiring one. For each component of your LLM application — the prompt layer, each tool the model can call, the retrieval pipeline, and the tenancy boundary — work through four questions: what data or capability does this component expose, who could misuse it and how, what's the worst realistic outcome, and what control reduces that outcome's likelihood or impact. Document the answer even when the answer is "we accept this risk for now" — an explicit, reasoned decision is defensible; a gap nobody looked at is not.``` Run this worksheet before launch, and again whenever you add a new tool, data source, or tenant type — the threat model changes every time the architecture does.

Where AI security review fits alongside strategy and engineering

A security review works best as part of a broader discipline, not a one-off gate. Teams that pair early AI product strategy with security thinking from day one avoid retrofitting controls into an architecture that wasn't designed for them. Teams doing the engineering work — building the prompt layer, the tool integrations, the RAG pipeline — benefit from having threat modelling embedded in that AI engineering process rather than treated as a separate compliance checkbox at the end. If you're also modernising the surrounding systems an LLM application connects to, that's worth reviewing through the same lens as any application modernisation effort — legacy integrations are often where the weakest access controls live.

You can find more on related topics, including our AI readiness assessment framework and notes on MLOps consulting, on our insights page.

Get in touch

If you're preparing to launch an LLM application and want a second set of eyes on prompt injection exposure, tool permissions, RAG pipeline integrity, or tenant isolation before it goes live, we can help. We run pre-launch security reviews as a practical, scoped engagement — not a 200-page audit — and we'll tell you honestly where the real risk sits and where it doesn't.