AI supply chain and model integrity

A developer found a model that scores well on the benchmark, downloaded the weights, and put them behind a feature. Nobody asked who published them, what the file format can execute when it loads, what went into training, or what would have to happen for a different set of weights to end up in production next Thursday.

Three linked problems

AI supply chain work is usually sold as one thing and is really three, each with a different owner and a different fix:

Where the model came from. Publisher, licence, version, and whether the artifact you hold is the artifact they published.
What went into it. Training and fine-tuning data, its provenance, and every point at which untrusted content could have influenced the weights or the retrieval corpus.
What can tamper with it on the way to production. The registry, the promotion path, the approval gates and the people who can move an artifact between them.

The categories that frame them are OWASP LLM03 Supply Chain and LLM04 Data and Model Poisoning, and where the environment is also a development environment the practice set is NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models.

Model artifacts: the inventory

The first deliverable is a list, because most teams cannot produce one. Each artifact is recorded with its source and the URL or registry path it came from; its version or commit; its licence and whether the intended use is permitted under it; the hash of what you hold against the hash the publisher states; who introduced it; and what depends on it now.

The licence column is not paperwork. Model licences carry field-of-use and downstream-distribution terms that ordinary software licences do not, and the question of whether a particular use is permitted is one for your counsel once the inventory has put the terms in front of them.

Scanning model files

Some model serialization formats execute code on load. That is not a vulnerability in the format so much as a property of it, and it means loading an untrusted artifact is equivalent to running an untrusted program with the privileges of whatever loaded it — frequently a training host or an inference service with credentials.

The review covers which formats are in use and whether a safe-tensor format could be used instead; whether artifacts are scanned before they are loaded and where that scan sits in the pipeline; whether the scan blocks or merely reports; and whether ingestion happens in an isolated environment rather than on a host holding production credentials.

Dataset provenance and poisoning defence

Poisoning is the attack the primary sources agree on. Canada’s Centre for Cyber Security states in ITSAP.00.041:

Threat actors can inject malicious code into the dataset used to train the generative AI system. This could undermine the accuracy and quality of the generated data. It could also increase the potential for large-scale supply-chain attacks.

The EU AI Act, Regulation (EU) 2024/1689, names the same class of attack explicitly, requiring technical solutions that address AI-specific vulnerabilities to include, where appropriate, measures for attacks trying to manipulate the training data set (data poisoning) or pre-trained components used in training (model poisoning). Whether and how that Regulation applies to a given organization is a question for its counsel, not a claim this page makes.

So the work enumerates every point at which untrusted data enters: training and fine-tuning sets, human feedback and preference data, evaluation sets, the retrieval corpus, and any continuous-learning path that feeds production output back into training. For each, it asks who can write to it, what reviews the content, and what record exists afterwards. The provenance record and immutability recommendations follow from that: training sets held content-addressed and write-once, with the manifest for each training run kept alongside the resulting weights, so a later question about a model has an answer rather than an archaeology project. Controls on the fine-tuning work itself are covered on secure model fine-tuning. Post-deployment behaviour monitoring closes the loop, because the tell for a poisoned model is usually a behaviour change rather than a file change.

AIBOM: extending the bill of materials

An AIBOM is a bill of materials extended past software to the models, datasets, adapters and prompt assets a system depends on. Customers are beginning to ask for one, and the harder value is internal: you cannot answer "are we affected" about a model you never inventoried. In practice the AIBOM extends the software bill of materials you already produce, adding models with source, version and hash; datasets with provenance and licence; adapters and fine-tuning artifacts with their base model; embedding models, which are routinely forgotten and are as load-bearing as the generator; and prompt assets, which are configuration that changes behaviour and belongs under the same change control.

It is produced as an artifact of the pipeline rather than a document someone maintains, because a hand-maintained inventory is accurate on the day it is written.

MLOps and LLMOps pipeline security

The pipeline is the part an attacker actually wants, because compromising it is repeatable and compromising a single artifact is not. The review covers:

Registry access. Who can publish, overwrite or delete a model version, and whether a version can be replaced in place.
Signing and integrity verification. Whether artifacts are signed at build and verified at promotion and at load, rather than trusted because they came from an internal path.
Tamper-resistant approval gates. Whether promotion to production requires an approval that the promoting identity cannot grant itself.
Separation of duties. Whether the identity that trains, the identity that promotes and the identity that serves are distinct, and what a compromise of each would reach.
Build provenance. Whether a production model can be traced to the run, the data manifest and the code commit that produced it.

Where a specific hypothesis warrants exercising the pipeline rather than reviewing it — attempting an unauthorized promotion, for example — that is testing, and it is scoped and authorized under the terms below.

What you receive

An artifact inventory of models, adapters, embedding models and datasets, with source, licence, version and integrity status.
Scanning results for the artifacts in scope, with the pipeline placement recommended for the scan itself.
A provenance record design for training and fine-tuning data, and the retention that makes it useful.
An AIBOM, produced once with you and specified so your pipeline emits it thereafter.
A pipeline threat model covering registry, promotion, approval and serving, with the separation-of-duties changes it implies.

No honest answer promises backdoor detection. Scanning finds unsafe serialization and embedded code reliably; a behavioural backdoor tied to a trigger the defender has not guessed is a testing plan and a monitoring posture, not a guarantee. Saying so is part of the deliverable: a report that implied otherwise would be worth less than one that states the limit and tells you what to monitor instead. Where the same discipline is needed for the code your developers generate, see AI-generated code security review and secure development; where it is the runtime environment that needs building, see secure AI environment establishment.

How the work is bounded

The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.

Testing and adversary simulation are carried out only with signed authorization, to a scope agreed in writing.

Investigation and forensic work is delivered with appropriately licensed partners where the law requires it.

SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.

Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.

Questions we are asked

We only use hosted models — does AI supply chain security apply?

Hosted models move the problem rather than removing it: you inherit the provider’s supply chain without visibility into it, and the parts you still own — adapters, embedding models, prompt assets, evaluation sets and the pipeline that promotes them — are exactly where tampering is cheapest.

What is an AIBOM, and does anyone ask for one yet?

Can you detect a backdoor in a model?

Do you run these checks in our CI?

The checks are designed to run in your pipeline and owned by your team: scanning on artifact ingestion, integrity verification on promotion, and a failing gate rather than a warning. We specify and help wire them; we do not operate them.

Start with the inventory

List the models in production, where each came from, and who can promote one. The reply says what a review would cover and which gaps usually turn up first.

Discuss a scope