Introduction
Every organisation deploying AI faces the same question: How do you prove your AI system is safe, compliant, and effective – not just today, but throughout its entire operational life?
The answer isn’t found in inventing entirely new governance structures. The path to robust AI assurance lies in extending the rigorous, risk-informed validation principles already proven in high-stakes domains such as aviation safety, financial model risk management, and medical device regulation.
AI systems present a fundamental challenge to traditional validation approaches. Unlike deterministic software that produces predictable outputs, AI systems generate probabilistic results and adapt their logic over time. This shift from static code validation to continuous behavioural assurance demands a new approach – one that the SECURE-AI framework directly addresses.
The Critical Shift from Code Validation to Behavioural Assurance
Traditional software validation assumes predictable, deterministic systems. AI validation must account for adaptive, probabilistic behaviour that evolves throughout the system’s lifecycle.
| Traditional Software | AI/Machine Learning Systems | Validation Implication |
|---|---|---|
| Deterministic Outputs | Probabilistic Outputs | Validation must assess confidence calibration and acceptable error rates, not binary pass/fail outcomes. |
| Static Logic | Adaptive / Learned Logic | Validation becomes continuous, monitoring for drift and ensuring systems remain within approved operating parameters. |
| Known Failure Modes | Emergent Failure Modes | Validation requires adversarial testing and red-teaming to uncover unknown vulnerabilities and misuse scenarios. |
| Code Validation | Model + Data + Behaviour Validation | Validation scope expands beyond algorithms to include data provenance, training environment, and real-world operational context. |
This transition necessitates moving from validation as a pre-deployment checkpoint to continuous assurance spanning strategy, governance, design, execution, and audit.
Proven Principles from High-Stakes Domains
The most successful safety-critical industries have already solved the challenge of managing complex, high-consequence systems. Their validation principles provide the foundation for effective AI governance.
Aviation: Continuous Safety Assurance
The aviation industry operates under standards like DO-178C for airborne software, establishing that safety isn’t a feature but a system property requiring continuous demonstration.
The Federal Aviation Administration’s approach to AI emphasises introducing AI within structured, disciplined, risk-managed ecosystems, using existing safety requirements as the foundation. For AI deployment, this principle translates to two requirements:
Working Within the Ecosystem: AI components require validation as integrated parts of larger, safety-critical systems, never in isolation. This mirrors the E (Execution & Deployment) pillar of SECURE-AI, which validates system integrity in production, including failure and rollback behaviour.
Continuous Monitoring: AI models can degrade over time through model drift or data drift. Aviation’s move toward continuous assurance for AI systems directly informs the AI (Audit & Iteration) pillar of SECURE-AI, mandating ongoing behavioural assurance and periodic re-validation.
Finance: Independent Model Risk Management
Financial services pioneered Model Risk Management through regulatory guidance such as the Federal Reserve’s SR 11-7. This framework manages risk inherent in complex quantitative models and provides essential principles for AI validation:
Independent Validation: Models require validation by parties independent of their developers or users. This “effective challenge” ensures objectivity and uncovers hidden assumptions or flaws. This principle drives the U (Use Case Design & Validation) pillar of SECURE-AI, ensuring Model Validation Reports demonstrate robust, unbiased assessment.
Governance and Inventory: All models require cataloguing, risk classification, and formal approval by governance committees. This approach aligns with the S (Strategy & Assessment) pillar of SECURE-AI, which requires an AI Intent Statement and Impact Classification to determine appropriate governance intensity.
Healthcare: Total Product Lifecycle Management
The U.S. Food and Drug Administration regulates AI-enabled medical devices through a Total Product Lifecycle approach that acknowledges machine learning’s adaptive nature.
The framework allows iterative improvements and model retraining after deployment, provided changes remain within pre-specified “predetermined change control plans”. This shifts focus from validating static models to validating the process by which models are developed, deployed, and updated.
This concept forms the foundation of the AI (Audit & Iteration) pillar, emphasising that AI validation continues throughout the system’s operational life. It establishes regulatory precedent for managing Behavioural Drift in production – a key validation concern within SECURE-AI.
The SECURE-AI Validation Blueprint
The SECURE-AI framework synthesises cross-industry validation principles into a unified, end-to-end governance model. Rather than bypassing established controls, it extends them to address AI’s unique challenges.
| SECURE-AI Pillar | Validation Focus | Established Practice Parallel | Core Validation Output |
|---|---|---|---|
| S – Strategy & Assessment | Strategic Fit & Appropriateness | Nuclear Software Integrity Levels, Financial Product Governance | AI Intent Statement, Impact Classification |
| E – Ethics & Governance | Normative and Behavioural Boundaries | Corporate Governance Charters, Responsible AI Principles | Ethical Risk Register, Accountability Map |
| C – Compliance Blueprint | Lawfulness & Regulatory Alignment | GDPR Privacy Impact Assessments, Financial Compliance Mapping | AI Compliance Mapping, Data Lineage Declaration |
| U – Use Case Design & Validation | Fitness for Purpose | Aviation DO-178C V&V, Financial Model Validation (SR 11-7) | Model Validation Report, Approved Operating Envelope |
| R – Risk Management & Controls | Threat, Failure, and Abuse Scenarios | Enterprise Risk Management, Threat Modelling | AI Risk Register, Mapped Controls & Owners |
| E – Execution & Deployment | System Integrity in Production | Cloud Deployment Best Practice, SRE/Reliability Engineering | Deployment Approval Checklist, Rollback & Containment Plan |
| AI – Audit & Iteration | Ongoing Behavioural Assurance | FAA Continuous Assurance, FDA Total Product Lifecycle | Performance & Risk Trend Reports, AI Assurance Pack |
Critical Validation Depth: Use Case Design and Ongoing Audit
Two pillars require particular attention in adapting traditional validation to AI systems.
The U (Use Case Design & Validation) pillar applies rigour directly to AI models. It extends beyond simple accuracy metrics to mandate:
Adversarial Testing: Red-teaming models to test prompt injection resistance and misuse scenarios – a direct parallel to stress testing in finance and failure mode analysis in aviation.
Known Limitation Documentation: The output is explicitly defining the boundaries of reliable model performance. Where the model can be used, and where it must not be used.
The AI (Audit & Iteration) pillar institutionalises continuous assurance. Because AI systems adapt by design, they’re inherently susceptible to drift. This pillar implements the FDA’s TPLC concept through:
Drift Detection Alerts: Monitoring model, data, and behavioural drift to trigger mandatory re-validation.
Decommissioning Thresholds: Pre-defined criteria that automatically trigger re-approval or retirement decisions when breached. AI validation continues until system retirement.
The Path Forward
AI validation represents a new set of governance challenges. The most successful safety-critical industries have already established the principles required for managing complex, high-stakes systems: governance before development, independent challenge, and continuous, lifecycle-spanning assurance.
The SECURE-AI framework operationalises these proven lessons. By mandating validation at every stage, from strategic intent (S) through ongoing behavioural audit (AI), it ensures AI systems aren’t merely functional but demonstrably safe, sane, and secure.
The only acceptable validation approach is one that’s continuous, comprehensive, and grounded in proven risk management practices from the world’s most critical domains. Does yours meet the mark?
References
[1] Galdren. SECURE-AI Implementation Framework: Core Principles of AI Governance, Development and Deployment. Available at: https://galdren.com/secure-ai-governance-playbook/
[2] Federal Aviation Administration (FAA). Roadmap for Artificial Intelligence Safety Assurance. Available at: https://www.faa.gov/media/82891
[3] Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency. Supervisory Guidance on Model Risk Management (SR 11-7). April 2011. Available at: https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
[4] U.S. Food and Drug Administration (FDA). Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. January 2025. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing