What Is AI Model Security?

AI model security is the discipline of protecting an artificial intelligence model's integrity, confidentiality and availability throughout acquisition, development, deployment, use and retirement, while preserving evidence about its origin, components, changes and authorized operational behavior.

An AI model is a learned computational artifact: its parameters and structure encode patterns derived from training. Organizations may build one internally, fine-tune a third-party model or call a model hosted as a service. Each path creates different custody, visibility and control boundaries.

Model security addresses deliberate compromise, unauthorized disclosure and operational failure involving that artifact. It does not cover every risk in an AI system. The runtime loads and executes the model, while an agent can combine model output with memory, tools and permissions to take actions.

The discipline extends conventional endpoint security and software supply-chain practices to learned behavior, training data, model weights and evaluation evidence. This matters when a model runs locally on an employee endpoint as well as when an endpoint sends requests to a remote service.

Why AI model security matters

Organizations often receive an AI model through a longer and less visible chain than a conventional application. A model may depend on third-party datasets, pretrained weights, training libraries, conversion utilities, evaluation code, repositories and hosted infrastructure. A compromise or undocumented change at any point can follow the artifact into production.

The model itself can also be valuable intellectual property or a route to sensitive information. An attacker may try to steal weights, infer information about training records, corrupt learned behavior, insert a hidden trigger or make the model unavailable. A model can remain loadable and produce plausible output even when its security properties have changed.

This last point makes model integrity more than a file-integrity problem. A valid signature or matching hash can show that an artifact is the expected file, but it cannot prove that its training data were suitable, its behavior is safe for the intended use or its provider remains trustworthy. Model security combines artifact assurance with behavioral evaluation and operational controls.

How AI model security works

AI model security uses a lifecycle rather than a single inspection. The controls differ by risk and deployment model, but the core sequence is consistent.

1. Define the intended use and risk boundary. The owner records what the model is allowed to do, which data it may process, the environments where it may run and the impact of incorrect or unavailable output.
2. Identify the model and its dependencies. The team inventories the exact model version, source, license, data lineage, frameworks, adapters, evaluation assets and services needed to build or operate it.
3. Assess provenance and supplier evidence. Reviewers examine who created and modified the artifact, how it moved between custodians, which processes produced it and whether releases can be verified.
4. Validate the artifact. The receiving environment checks approved location, cryptographic digest or signature, expected format and package contents before the model enters a trusted registry.
5. Evaluate behavior. Teams test the model against its intended task, misuse cases, known failure modes and model-specific attacks. A changed artifact or material operating change triggers reevaluation.
6. Restrict deployment and access. Least-privilege identities control who can publish, retrieve, replace or invoke the model. Sensitive weights, secrets and training records receive separate access and storage controls.
7. Monitor operation. The operator records the deployed version, configuration, input and output signals appropriate to the use case, access events, performance drift and security alerts without collecting unnecessary sensitive content.
8. Respond and retire. Affected versions can be isolated, rolled back or revoked. Retirement removes unused copies and credentials while preserving the evidence required for investigation and accountability.

NIST's secure-development profile recommends tracking the provenance of a model and its components, including training libraries, frameworks and pipelines. It also calls for model-specific threat modeling and protection of models and related data throughout development, as described in SP 800-218A.

The result is a chain of evidence. An organization should be able to connect a production inference to an approved model version, that version to its evaluation and release record, and the release to the data, code, people and processes that produced it.

Model provenance and supply-chain security

Model provenance is the documented history of an AI model's origin, ownership, custody, dependencies and changes. Good provenance does not establish that a model is secure by itself. It makes security claims testable by showing what the artifact is, where it came from and what happened before deployment.

Useful provenance records commonly include:

• Model name, version, cryptographic digest and approved storage location
• Creator, supplier, owner and authorized custodians
• Parent model, fine-tunes, adapters, merges, quantization and format conversions
• Training, validation and evaluation datasets, including their origin and handling constraints
• Training code, libraries, framework versions, build environment and pipeline identity
• Intended use, prohibited uses, known limitations and evaluation results
• Release approvals, signatures, deployment history, incidents and retirement status

Model supply-chain security applies risk controls to every organization, service and component that can affect the model. Joint national cybersecurity guidance identifies AI data, models, software, infrastructure, hardware and third-party services as connected parts of the AI and machine-learning supply chain. Its joint guidance recommends lifecycle assessment, supplier visibility, integrity verification, quarantine and testing rather than trusting a model solely because it came from a familiar repository.

An AI bill of materials can organize some of these dependencies, but a component list is not complete provenance. It may identify a framework and parent model without explaining who approved a fine-tune, which data restrictions apply or whether the resulting behavior passed evaluation. Provenance needs both machine-readable identifiers and accountable process records.

Threats to AI models

Model threats can affect the artifact before deployment, its behavior during inference or the information it contains. NIST organizes adversarial machine-learning risks around attacker goals, capabilities, knowledge and the lifecycle stage targeted. Its current AML taxonomy covers poisoning, evasion, privacy and misuse attacks across predictive and generative systems.

Poisoning and backdoors

Poisoning changes training data, a training process or model parameters so the resulting model learns unwanted behavior. A backdoor is a targeted form of manipulation that may stay dormant until a particular trigger appears. Normal accuracy tests can miss such behavior if the evaluation set does not include the trigger or affected conditions.

Evasion and adversarial inputs

Evasion manipulates inference-time input to cause an incorrect prediction or classification. The input can remain acceptable to a person while exploiting the model's decision boundary. Input validation and adversarial testing can reduce exposure, but defenses are specific to the model, task and attacker capability.

Model theft and privacy attacks

Unauthorized access can expose model weights, architecture, parameters or behavior. Query-based attacks may approximate a model or infer properties of its training data without obtaining the original artifact. Rate limits, access controls and output restrictions can raise the cost of these attacks, but they do not erase information already encoded or revealed.

Unsafe model artifacts

A model package is handled by software, parsers and loaders. Untrusted formats or conversion tools can therefore create conventional code-execution, dependency and parsing risks in addition to learned-behavior risks. Organizations should inspect and test artifacts in an isolated environment before promoting them to a production registry.

Availability and resource abuse

Inputs can cause unusually expensive computation, exhaust memory or trigger repeated service failures. Capacity limits, request controls and isolation protect the surrounding service. These controls preserve availability but do not establish the model's behavioral integrity.

An enterprise model-security example

Northstar Legal, a fictional company, wants to run a document-classification model on employee laptops so files can be labeled before upload. Its engineering team selects an openly available base model, fine-tunes it on approved internal examples and converts it to a format supported by the local runtime.

The starting state is untrusted: a model name and repository page do not prove which bytes were received or how they were produced. The team records the base model's publisher, version, digest and license, then quarantines the download. Reviewers inspect the package, scan its dependencies and evaluate the base model against a representative test set before any internal data are introduced.

The fine-tuning pipeline runs under a dedicated workload identity in a controlled environment. It records the training-code commit, dataset version, framework, parameters and parent-model digest. A separate approval signs the resulting release only after tests cover expected classification, sensitive-data handling, manipulated documents and resource use.

Endpoint deployment policy permits only the approved digest. Each laptop receives a constrained runtime and the model version appears in inventory. Inference stays local, while monitoring records the version, health and aggregate error signals rather than document contents.

When a dependency advisory affects the conversion tool, the company can identify the derivative models built with that version. It pauses new deployments, rebuilds the model through the corrected pipeline, repeats evaluation and revokes the earlier release. Provenance turns a broad warning into a bounded investigation and a verifiable replacement.

Benefits of AI model security

A model-security program improves decisions when its evidence leads to enforceable controls.

• Known origin: Provenance helps teams distinguish an approved release from an ambiguous model name or mutable repository reference.
• Controlled change: Parent-child lineage makes fine-tuning, conversion and replacement visible rather than treating every derivative as the same model.
• Focused response: Dependency, dataset and deployment records identify which models and environments a security event can affect.
• Protected assets: Access controls reduce unauthorized copying, modification and invocation of valuable or sensitive models.
• Behavioral assurance: Repeatable evaluation checks whether the artifact remains suitable for its defined use and threat model.
• Accountable retirement: Inventory and revocation reduce abandoned copies and unsupported versions.

These benefits depend on ownership. Evidence that no team reviews, deployment gates that operators can bypass and evaluations disconnected from the real use case create documentation without assurance.

AI model security risks and limitations

Model security cannot prove that a model will behave correctly under every future input or operating condition. Testing samples a large behavior space; it does not exhaust it. A passing result supports a scoped decision, not a universal safety claim.

Other limitations include:

• Provenance can be incomplete or false. A supplier's records still require validation and contractual accountability.
• Signatures prove integrity, not quality. A correctly signed model can contain poisoned behavior, unsafe capabilities or unacceptable bias.
• Scans have limited visibility. Conventional malware and dependency tools do not fully assess learned behavior.
• Fine-tuning changes risk. A derivative can invalidate evaluation evidence inherited from its parent model.
• Monitoring can expose data. Recording prompts, features or outputs without minimization can create a separate privacy and security problem.
• Hosted models limit evidence. A service customer may receive a version label and assurance report rather than weights, pipeline records or direct inspection rights.
• Controls can drift. Runtime updates, decoding settings, retrieval sources and tool permissions can materially change system behavior without changing the model file.

Organizations should document residual risk, evidence gaps and assumptions for each deployment. Higher-impact uses need stronger supplier assurance, independent testing, human oversight and a tested fallback when the model is unavailable or untrusted.

AI models, runtimes and agents

Model security is one layer of AI system security. Keeping the layers distinct helps teams assign the right owner and control.

Agentic AI security covers the broader system in which a model can influence actions on an endpoint. AI governance sets accountability, acceptable-use and risk-decision processes across models and systems.

Prompt injection illustrates the boundary. It manipulates instructions or context presented to a generative model at runtime. Provenance and weight integrity cannot prevent an agent from following malicious contextual instructions, so the surrounding system must constrain data flow, tool authorization and resulting actions.

AI model security establishes a defensible answer to four questions: which model is this, where did it come from, what evidence supports its approved use and how can the organization contain or replace it? Those answers support trust, but they remain one part of a secure AI system.