Cybersecurity
Threat-modelling an AI pipeline before the demo
AI demos optimise for “look what it can do.” Security work asks a colder question: what can an adversary make it do, and what can they learn while it fails? If you wait until after the launch party to answer that, you inherit a system whose trust assumptions were never written down.
This note is a compact threat-modelling checklist for LLM-centred pipelines — retrieval, tools, agents — aimed at teams that already ship software and need to extend their existing practice, not invent a new religion.
Draw the pipeline, then name the assets
Start with a single diagram: user → gateway → model → retrieval / tools → downstream systems → logs. On that diagram, mark:
- Secrets — API keys, database credentials, signing material.
- Sensitive content — customer data in the corpus, PII in prompts, proprietary code in context.
- Privileged actions — anything a tool can write, delete, pay, or unlock.
- Trust decisions — where output is treated as fact by a human or by another system.
If a box on the diagram can both read secrets and execute tools, treat that box as high-value until proven otherwise.
Attack classes worth listing explicitly
Prompt and instruction injection
Untrusted text (email, PDF, web page, ticket) enters the context and steers the model. Mitigations: strict separation of system instructions vs. retrieved content, output schemas, and refusing to execute tool calls that were only justified by untrusted text without a second check.
Data exfiltration via the model
The model is asked to summarise or “debug” something that includes secrets from retrieval or prior turns. Mitigations: least-privilege corpora, redaction before embedding, session isolation, and blocking known secret patterns in responses.
Tool abuse
An agent with a shell, inbox, or payment API is social-engineered into a harmful call. Mitigations: allow-lists, human approval for irreversible actions, scoped credentials per tool, and rate limits that survive a confused model.
Poisoned retrieval
An attacker plants documents so retrieval consistently surfaces them. Mitigations: provenance on corpus updates, signing or review gates for high-impact sources, and anomaly detection on sudden rank changes.
Supply chain and serving plane
Compromised weights, plugins, or the inference host itself. Mitigations: pinned artefacts, SBOM for the serving stack, network egress controls, and the same patch discipline you already apply to app servers.
The model is not the perimeter. Every tool it can call is part of your attack surface — treat it like an intern with production credentials.
What to cut first
When time is short, prioritise in this order:
- Remove or gate tools that can change state outside the chat.
- Stop dumping secrets and raw PII into prompts “for convenience.”
- Add authentication and authorisation at the gateway — not only at the chat UI.
- Log prompts, tool calls, and retrieval IDs with retention limits and access control.
- Write an abuse case list and run it before the customer demo, not after.
Closing
You do not need a 40-page threat model for a weekend prototype. You do need an honest diagram, a short list of assets, and a decision about which tools the model is allowed to touch. Do that before the demo hardens into production.
For a structured review of an AI or automation pipeline, contact Hieg.