AI security / supply chain
Backdoored Models Are a Supply-Chain Problem: What Is Proven, and How to Decide Who Carries the Risk
Companies are moving to open-weight models for good reasons: no provider can withdraw them, reprice them or change their policies overnight, and they can run on hardware you control. But the weights come with something a file scan cannot see. Marin Ivezic's analysis, Open-Weight Model Backdoors: What Is Demonstrated (securing.ai, 2 September 2026), puts it plainly: model hubs check for malware, not for behaviour. This article builds on his piece, adds what has surfaced since, including a cheap, public proof of concept that went viral on X this week, and ends with a concrete proposal. It covers how to secure the model supply chain and how to decide whether to handle it in-house or hand it to someone else.
What is demonstrated, and what is not
The geopolitical debate over open weights is mostly about whether a state actor could have planted something in a model. Ivezic applies the standard evidence test (what was shown, on which model, verified by whom?) and reports that he could not find a publicly documented, independently verified case of a released Chinese open-weight base model containing a state-implanted backdoor. That is not proof that none exists. It means the claim has not been demonstrated.
What has been demonstrated is more uncomfortable, because the threat does not depend on any country:
- Backdoors survive safety training. Anthropic's Sleeper Agents work (2024) trained models that wrote secure code under one stated year and exploitable code under another. The behaviour survived supervised safety fine-tuning, RLHF and adversarial training. Adversarial training taught the model to recognise the probe, which made the backdoor harder to find rather than removing it.
- Poisoning needs a fixed, small amount of data. A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute (paper) found that about 250 poisoned documents backdoored models from 600M to 13B parameters, whatever the size of the training set. The authors stress that the backdoor they tested was low-stakes (it produced gibberish text) and that it is still open whether the result holds for frontier models and more harmful behaviours.
- Implanting a backdoor needs no pretraining run. Ivezic cites a March 2026 preprint that used parameter-efficient fine-tuning to implant a date-triggered backdoor in a tool-using Qwen3-4B model. It reported a 99.6% mean attack success rate against a 0.7% false-positive rate, measured by the authors on a private benchmark.
- It is cheap, and it works on real agent tooling. On 6 October 2026 ProjectDiscovery published How abliterated models can get you pwned. The researchers fine-tuned Qwen2.5-7B-Instruct with a QLoRA adapter (500 clean and 125 poisoned tool-calling examples, one rented L4 GPU, about 2.5 hours, about $8 of compute). When the trigger phrase appeared, the model told OpenAI's Codex CLI to download and run a script that sent
.envcredentials to the researchers' server. They report a 100% trigger rate (50/50) with 100% clean accuracy (50/50). The demo spread widely on X (e.g.), and the practical summary going around was "sandboxing and egress logs can [see it]" (AI Cyber Brief on X). Note that these are the researchers' own measurements, though they published the full pipeline. - Malicious model files are a separate, well-documented problem. JFrog found about 100 Hugging Face models that executed code when loaded, around 10 of them "truly malicious" (Ars Technica). One opened a reverse shell through pickle's
__reduce__. ReversingLabs' nullifAI models used deliberately broken, 7z-packed pickles to get past Picklescan.
Detection exists, but certification does not. Microsoft's AI red team published The Trigger in the Haystack and an open-source scanner (blog). It can reconstruct triggers without knowing them in advance, but it needs access to the model files and works best on backdoors with deterministic outputs. As Ivezic says, it is good enough to use when admitting a model and nowhere near good enough to certify one as clean. On X, practitioners agree: detecting sleeper-agent behaviour is "still unsolved", and running a model on your own infrastructure "fully protects data sovereignty" only if the model itself is trustworthy.
The threat map
| Layer | Threat | Status |
|---|---|---|
| Weights | Conditional backdoors / sleeper agents | Demonstrated in research and PoCs; no verified in-the-wild state implant found |
| Training data | Pretraining or fine-tuning poisoning | Demonstrated (250-document result, low-stakes behaviour) |
| Derivatives | Malicious fine-tunes, LoRA adapters, merges, "abliterated" variants | Demonstrated (ProjectDiscovery); OWASP LLM03 lists malicious LoRA adapters and merge/conversion services as supply-chain risks |
| Conversion / quantisation | Tampering during format conversion or quantisation by third parties | Listed by OWASP as a risk (conversion services); I found no verified public incident, so treat it as plausible rather than proven |
| Serialisation | Pickle / unsafe formats that run code on load | Documented in the wild (JFrog, ReversingLabs) |
| Repo code & tokenizer files | trust_remote_code, custom pipelines, loader bugs | Documented: Diffusers before 0.38.0 allowed trust_remote_code bypasses, e.g. CVE-2026-44827 |
| Dependencies & hosting | Compromised libraries, inference servers, hubs, accounts | Standard software supply-chain risk, now with ML components in scope |
The key distinction: provenance tells you who published an artefact, and integrity tells you that you received what they published. Neither tells you the learned behaviour is benign. Hub scanning covers the artefact layer, while the backdoor lives in the behaviour layer.
A proposal: how companies should approach it
1. Treat weights as untrusted third-party code. This is a classification decision, and the rest follows from it. It applies the same way to Llama, Qwen, Mistral and the fine-tune your data team found last month.
2. Inventory everything, including derivatives (AIBOM). Most organisations can name their base model. Few can list the adapters, merges, quantised variants, datasets and inference libraries actually in production. Keep one inventory and export it as CycloneDX ML-BOM for operations or SPDX 3.0 AI/Dataset profiles for governance. Record hashes, publisher, licence, lineage and where each component is deployed.
3. Provenance and trusted sources. Pull only from verified publisher organisations. Pin exact revisions and hashes, and mirror approved models into an internal registry. Verify signatures where they exist (OpenSSF Model Signing v1.0, Sigstore-based). Signing answers the question of who published a model, not whether it is safe.
4. Safe formats and safe loading. Use safetensors by default and block pickle-based formats from untrusted sources. Turn trust_remote_code off unless the code has been reviewed, and keep loader libraries patched.
5. Sandboxed admission testing. Load and evaluate new models in an isolated environment with no network egress and no credentials. A reverse shell then has nowhere to go, and a credential stealer has nothing to steal.
6. Behavioural and red-team testing. Run a backdoor scanner such as Microsoft's where you have access to the weights. Test with your own domain prompts, tool-calling scenarios and plausible triggers (dates, names, project identifiers). Re-run the tests whenever the model, adapter or quantisation changes. Treat a pass as evidence, not a certificate.
7. Runtime guardrails and least privilege for agents. This is the most important control, because it works even when the backdoor is never detected. Enforce tool permissions outside the model, before the call executes. That means allow-listed commands, egress filtering, secrets kept out of working directories, human approval for high-impact actions, and separate identities per agent. In the agentic case the demonstrated payloads are tool calls, not text.
8. Monitoring. Log every prompt, tool call and outbound connection. Alert on new destinations and on unusual command patterns. Compare behaviour across model versions.
9. Incident response. Add model compromise to your playbooks. Be ready to roll back to a pinned known-good model, rotate any secrets the agent could reach, review its tool-call logs, and notify the upstream publisher or hub.
10. Contracts. Require suppliers (model vendors, hosting providers, integrators) to disclose model lineage and AIBOMs, notify you of incidents, describe their testing, and let you audit them. Under the EU AI Act, Article 15 already requires high-risk systems to include measures against "data poisoning" and "model poisoning". Providers of general-purpose AI models already have documentation duties, and the AI Act's implementation timeline has been amended by the Digital Omnibus, so check the current application dates. The joint CISA/NSA/FBI AI Data Security guidance (May 2025) adds data provenance, signing and integrity controls.
Don't let a model's national origin substitute for assurance. Jurisdiction and supplier governance are legitimate procurement questions, but they don't answer the behavioural one.
Route in-house, use a provider, or outsource?
Every option moves the risk somewhere rather than removing it. Using a managed API hands the supply-chain work to the provider, but you take on their training data, their policy changes and the risk that they withdraw the service. Running open weights yourself removes that withdrawal risk, and the supply-chain work becomes yours.
Decision criteria:
- Data sensitivity: can prompts and context legally and contractually leave your environment?
- Regulatory exposure: do you fall under the EU AI Act high-risk regime, NIS2, DORA or sector rules?
- Team capability: can you run MLOps security, red-teaming and agent sandboxing?
- Model criticality and autonomy: does the model only draft text, or does it execute tools, code or payments?
- Verification cost: who can afford weight-level scanning and behavioural testing for every new model version?
- Dependency risk: what happens if the provider withdraws or reprices the model?
| Option | Who carries supply-chain verification | Best when | Main residual risk |
|---|---|---|---|
| Managed API (closed model) | Provider (opaque to you) | Low-sensitivity data, small team, fast start | No visibility into weights or training; withdrawal, repricing, policy and refusal changes; data residency |
| Hosted open-weight (cloud/inference provider) | Shared: you choose the model, the provider runs it | Medium sensitivity, need model choice without running GPUs | You still own model selection; provider is in the trust chain |
| Local / sovereign deployment | You | Sensitive or regulated data, sovereignty requirements, long-term control | Full responsibility for the controls listed above; needs skills |
| MSSP / consultancy assurance | Partner does it, you stay accountable | Any of the above when in-house capability is thin | Accountability never transfers, only the work does; vet the vendor like a supplier |
Recommended defaults:
| Organisation | Default | Why |
|---|---|---|
| Small (<50 staff), low risk | Managed API or reputable hosted open-weight; no autonomous tool use on production systems | Verification cost would exceed the benefit; keep agents read-only |
| Mid-size, moderate risk | Hosted or local open-weight from a pinned, curated internal registry; outsource admission testing and red-teaming to a specialist; keep runtime guardrails in-house | Balances control with cost; guardrails are cheap and effective |
| Large / regulated / critical infrastructure | Local sovereign deployment for sensitive workloads, plus a managed API for low-risk ones; an internal model-admission board; external red team as an independent check | Regulatory duties and data sensitivity justify owning the controls |
| Agentic systems with write access (any size) | Whatever the hosting choice: sandboxed execution, egress allow-lists, secrets isolation, human approval for high-impact actions | The demonstrated payloads arrive as tool calls |
The sovereign AI angle
Local sovereign AI is a strong answer to dependency, data residency and withdrawal risk, and the July 2026 Hugging Face incident shows why. Ivezic reports that its responders finished their forensic analysis on an open-weight model running on hardware they controlled, after commercial models refused the task. Sovereignty is not the same as trust, though. A model on your own GPUs is only as trustworthy as the weights you loaded and the permissions you gave it. Sovereign AI done properly means owning the whole chain: a curated registry, signed and pinned artefacts, admission testing, and agents that cannot reach secrets or the internet unless they are explicitly allowed to. That is achievable, and in many organisations it is the right default for sensitive workloads. It just has to be designed in from the start, not assumed.
Bottom line: No one can yet certify a model as backdoor-free, so build as though it might not be. Inventory what you run, control where it comes from, test it before admission, and constrain what it can do at runtime. Then decide who does that work, whether your team, your provider or a partner, based on how sensitive the data is, how much autonomy the model has, and whether you can afford to verify it yourself.
Sources
- Marin Ivezic, Open-Weight Model Backdoors: What Is Demonstrated, securing.ai, 2 Sep 2026 (base article)
- Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, Anthropic, 2024
- Souly et al., Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples; Anthropic summary; Alan Turing Institute blog
- ProjectDiscovery, How abliterated models can get you pwned, Oct 2026
- Bullwinkel et al. (Microsoft), The Trigger in the Haystack; Microsoft Security Blog; llm-backdoor-scanner
- JFrog, Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor; Ars Technica coverage
- ReversingLabs, Malicious ML models discovered on Hugging Face platform (nullifAI)
- Hugging Face Diffusers advisory GHSA-98h9-4798-4q5v; CVE-2026-44827
- OWASP, LLM03:2025 Supply Chain
- CISA/NSA/FBI et al., AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems, May 2025
- OpenSSF, Model Signing v1.0
- CycloneDX, ML-BOM; Implementing AI BOM with SPDX 3.0
- EU AI Act, Article 15: Accuracy, robustness and cybersecurity
- X discussion: @lmoroney, @justelite, @seflless, @iam_ramvenkat