AI model security and integrity need to be designed across the entire system. Protecting an API key is necessary, but it does not address whether the model retrieved information the user was allowed to see, followed manipulated instructions or invoked an unsafe action.
A practical framework connects governance, data controls, application security, evaluation and monitoring. NIST’s AI Risk Management Framework provides a useful structure through its Govern, Map, Measure and Manage functions.
Govern: define ownership and acceptable use
Name the business owner, technical owner and risk decision-maker. Document the intended users, approved data and allowed actions. Policies should distinguish experimentation from production and low-impact assistance from consequential decisions.
Keep an inventory of models, providers, datasets, prompts, tools and connected systems. Ownership makes changes and incidents traceable.
Map: understand the system and its risks
Diagram how data enters, what context reaches the model, which tools are available and where outputs go. Identify sensitive information, external dependencies and people affected by the result.
Threat modeling should include prompt injection, indirect instructions inside documents, excessive agency, data leakage, insecure output handling and misuse of connected tools.
Measure: test behavior with evidence
Create evaluation cases for normal work, edge conditions and adversarial inputs. Test permission boundaries with different user roles. Confirm that the application—not the model description—enforces authorization.
Measure grounding, format compliance, refusal or escalation behavior, tool selection and action validation. Repeat tests when models, prompts, retrieval or tools change.
Manage: reduce and monitor risk
Use least-privilege access, allowlisted tools, server-side input validation, output constraints and human confirmation for high-impact actions. Store secrets outside prompts and client code.
Monitor failures, unusual usage, cost spikes and changes in output quality. Maintain a rollback path and an incident process that covers model providers and connected business systems.
Protect data throughout the lifecycle
Clarify which data may be used for training, retrieval, evaluation and logging. Minimize collection and redact sensitive information where possible. Retention should match a documented business need.
Retrieval must preserve source permissions. A search index should not flatten access controls that exist in the original system.
Design tools for safe agency
An agent should receive a minimal, understandable set of tools. Anthropic’s guidance notes that bloated or ambiguous tool sets make agent behavior harder to manage. Keep tools narrow and descriptions clear.
Separate read tools from write tools. Validate identifiers and allowed values on the server, and require confirmation when an action is difficult to reverse.
Integrity includes operational quality
A secure system can still be unreliable. Track whether inputs have changed, sources are current and outputs remain fit for purpose. Record user corrections and investigate repeated patterns.
Security and quality should share release gates. The system is trustworthy only when it both resists misuse and performs the job it claims to perform.
Sources and further reading
- NIST AI Risk Management Framework
- NIST Generative AI Profile
- Anthropic: Writing effective tools for agents
- OpenAI practical guide to building AI agents
Planning an AI project?
Ellachka builds custom AI applications, datasets, MCP connectors and business integrations in Charlotte, North Carolina and beyond.
Explore AI model development →Discuss your project