The pressure to “do AI” has become one of the loudest forces in business. Boards ask for a strategy, competitors announce pilots, and employees experiment with new tools before governance has caught up. In that environment, speed is often mistaken for progress. The more useful leadership question instead of How quickly can we deploy AI? is How quickly can our organisation responsibly absorb the change it creates?
That distinction matters because AI does not merely add a new application to the technology stack. Used seriously, it changes workflows, decision rights, data practices, roles, controls, customer interactions and, in some cases, the economic logic of a business. If those changes arrive faster than the organisation can understand, govern and operationalise them, the outcome is not transformation or improvement, but accumulated work, inconsistent practices and a loss of confidence.
Jeff Wilke, a senior executive who worked at Amazon for 25 years, made this point bluntly to Jeff Bezos early in the company’s life. Bezos was generating ideas faster than the organisation could act on them, and Wilke told him: “You have to release the work at the right rate that the organization can accept it.” Bezos later described it as a profound insight. Every idea released beyond the organisation’s capacity to absorb it did not accelerate progress. It created distraction.
That principle applies directly to AI adoption. The objective is not to slow down for its own sake but to sequence change so that each step increases the organisation’s capacity for the next.
Moving too fast creates invisible failure
When leaders mandate AI adoption at a pace the business cannot sustain, the first failure is often invisible. A team may launch an impressive pilot, produce an executive demonstration, or buy licences at scale. The operating model around the tool, however, remains unfinished. Staff do not know when to rely on the output, where sensitive information may go, who owns errors, how exceptions are handled or what is expected of them once the old process is retired.
The result is a familiar pattern: workarounds multiply, quality varies by team, risk teams intervene late, and employees quietly revert to older methods when the new approach causes friction. AI earns a reputation as a management fad rather than a source of practical value. In the most severe cases, an organisation disrupts a reliable service model before it has built a dependable replacement. The approach fails because the change exceeds the organisation’s ability to cope, not because the model used was insufficiently powerful.
A useful way to think about this is as a mismatch between deployment velocity and absorption capacity.
Dimension
When deployment outpaces capacity
When capacity sets the pace
Process design
AI is layered onto unclear or unstable work.
The team redesigns one defined workflow and clarifies hand-offs.
People
Employees are told to adopt, but not taught how to exercise judgement.
Training, job aids and feedback are built into the rollout.
Data and controls
Data use and accountability are resolved after launch.
Guardrails, escalation paths and quality checks are established before broader use.
Value measurement
Activity is reported: licences, pilots and prompts.
Outcomes are measured: cycle time, quality, cost, customer experience and risk.
Trust
Early mistakes become evidence against the programme.
Small, well-governed wins create permission for the next change.
The difference is not caution versus ambition. It is operational discipline versus performative speed.
Established businesses are complex
Startups can often change rapidly because the organisation is smaller, its operating model is still forming and its technology carries little legacy integration. A founder can decide in the morning, change a product workflow by afternoon and observe the impact within days.
Established businesses face a different reality. They may serve millions of customers, operate under regulatory obligations, rely on complex supplier relationships and carry years of interconnected systems and policies. Their scale is an advantage, but it also means that a change in one function creates consequences in many others. A new AI-assisted credit decision, for example, has implications for compliance, model governance, customer communication, front-line procedures, auditability and dispute resolution. The business must move deliberately because the cost of a poorly absorbed change is multiplied by its reach.
This a design constraint to work within, not a weakness to apologise for. Large organisations should build a repeatable mechanism for safe acceleration. The goal is to increase the rate at which the business can absorb change, not to force a rate that breaks it.
Build absorption capacity before demanding velocity
The practical response is to treat AI adoption as a managed portfolio of changes rather than a single enterprise-wide instruction. Start with a workflow that is sufficiently valuable to matter, sufficiently bounded to govern and sufficiently measurable to learn from. Assign a business owner, a process owner and a risk or control partner. Define what a good outcome looks like before deployment, including the conditions under which the team will pause or reverse the change.
A sensible sequence has four stages.
Stage
Leadership question
Evidence required before moving on
Prove
Does this solve a real operational or customer problem?
A defined baseline and a demonstrable improvement in a controlled workflow.
Stabilise
Can people use it reliably under normal conditions?
Clear ownership, training, controls and a functioning exception process.
Replicate
Does the operating pattern transfer to similar work?
Consistent results across more than one team or business unit.
Scale
Can the enterprise support this without degrading quality or trust?
Resourcing, governance and technology capacity match the expanded scope.
First, prove usefulness in a narrow setting where people can compare the AI-assisted outcome with the existing process. Then stabilise the new way of working by documenting roles, controls, exceptions and training. Then replicate the pattern in adjacent use cases that share similar data, risks or operating habits. Finally, scale the platform and governance only after the organisation has evidence that the operating model works repeatedly.
This approach may appear slower at the beginning because it resists the temptation to announce universal adoption. In practice, it is usually faster over time. Each successful use case leaves behind reusable assets such as trusted data pathways, trained leaders, approved controls, implementation playbooks, measurement methods and internal advocates. The next deployment begins from a stronger base, drawing on accumulated organisational knowledge rather than repeating the same arguments from scratch.
Leaders must protect the rate of learning, not the rate of adoption
The leadership task is to sponsor AI usage and to protect the business’s learning rate. That means creating room for teams to identify flaws without being labelled resistant, refusing to measure progress solely by adoption volume and being explicit about what will not yet be automated. It also means resisting the false choice between reckless acceleration and organisational paralysis.
A mature AI programme should be demanding, but its demands should be specific: improve a process, raise decision quality, reduce a known friction point, protect customers, and demonstrate results. “Use AI everywhere” is an instruction to create unmanaged variation at scale.
Wilke’s observation to Bezos offers a better standard. Ideas have value only when an organisation can turn them into coherent action. Every AI deployment released beyond the organisation’s capacity to absorb it creates a backlog of unfinished transformation.
Organisations that make change smooth – clear enough for people to adopt, controlled enough to trust and valuable enough to sustain – will accumulate capability faster than those that simply move first. Slow is smooth, smooth is fast.
Australian Financial Services are moving fast with AI and APRA has high expectations.
Most boards haven’t caught up yet. APRA knows this, and it has said so plainly in its letter to the industry. The regulator’s message is clear: existing governance, risk, and operational resilience practices are not keeping pace with how quickly AI is being deployed inside regulated entities. That gap is now a supervisory concern.
This article explains what APRA and Australia’s updated privacy laws require of Financial Services using AI, and what a Quality Assurance strategy needs to look like in response.
APRA Sees Three Governance Failures Happening Right Now
APRA’s concerns are specific and three patterns appear consistently in its observations of the sector.
Boards are making AI decisions without enough information. Many boards rely on vendor presentations to understand their AI risks. That’s a problem when the vendor is also the one selling the product. APRA expects boards to be capable of independent challenge, and most are not there yet.
Governance exists on paper, not in practice. Entities have acknowledged that existing prudential standards apply to AI. Few have actually operationalised that acknowledgement with post-deployment monitoring, change management, and model decommissioning are common gaps.
Supplier dependencies are underexamined. Many FinTechs have concentrated significant AI activity with a single provider without tested exit strategies. When those providers rely on foundation models, training data, or fourth-party services, the chain of accountability becomes opaque. APRA expects visibility all the way through that chain.
APRA frames AI governance inside existing obligations such as CPS 230 for operational risk, outsourcing standards, and the Financial Accountability Regime (FAR). There is no separate AI rulebook. The existing rulebook applies, and APRA believes most entities are not meeting it.
Privacy Law Has Changed. Your AI Systems Probably Haven’t.
The most significant change for AI is the new automated decision-making transparency requirement. If your systems use personal information to make decisions that significantly affect individuals, such as loan approvals or credit scoring, customers will have a right to meaningful information about how those decisions are made. The two-year grace period ends on 10 December 2026.
Two other changes carry direct operational implications. First, Australians now have a personal right to sue for serious invasions of privacy. AI systems that process sensitive personal data at scale carry real exposure here. Second, the Office of the Australian Information Commissioner (OAIC) has stronger enforcement powers, including tiered civil penalties and the ability to conduct compliance assessments. The OAIC has already issued specific guidance on commercially available AI products.
APP 11 now explicitly requires “technical and organisational measures” to protect personal information. This is not an IT-only obligation and covers the full range of privacy and security controls around AI systems.
The Regulator Expects Boards to Lead, Not Delegate
Both APRA and ASIC have reached the same conclusion from different directions: Financial Services are adopting AI faster than their governance frameworks can handle it, and boards are too far removed from the risk to provide effective oversight.
APRA’s expectation is that boards understand AI well enough to set strategic direction and challenge assumptions. ASIC’s concern is that licensees are creating consumer harm by deploying AI without updating their risk and compliance frameworks.
The practical implications are straightforward. Boards and executives need sufficient AI literacy to ask hard questions of management and vendors. Accountability for material AI use cases needs to be named, not distributed. Under FAR, accountable persons need to be identifiable for decisions made by or with AI systems. Staff need training that goes beyond “here is the tool” and covers misuse, limitations, and secure practices.
Human oversight and ownership is not optional for high-risk decisions. AI can inform those decisions but a named person needs to own them.
Your AI Systems Need to Fail Safely, Not Just Perform Well
CPS 230, effective from 1 July 2025, extends operational resilience obligations to AI-enabled systems. This creates concrete requirements that go beyond standard performance monitoring.
AI systems supporting critical operations need tested fallback processes. That distinction matters when regulators ask for evidence. AI failure modes that need to be planned for include hallucination, silent degradation, and susceptibility to adversarial inputs such as prompt injection or data poisoning.
Security requirements have also become more specific. AI adoption changes the attack surface with more entry points, faster attack cycles, and new risks from non-human AI agents with system access. APRA expects strong privileged access management, timely patching, hardened configurations, and penetration testing that covers AI-specific vulnerabilities, including AI-generated code.
Data governance sits underneath all of this. The quality and provenance of training data affects model behaviour. That is now a prudential concern.
What an APRA-Ready AI QA Strategy Looks Like
Active supervision of AI is underway with regulators no longer observing and advising. They are assessing and acting. An AI QA strategy needs to be designed for that environment.
The foundation is a centralised AI inventory: every AI system in use, including third-party tools, mapped to the regulatory obligations it touches under CPS 230, FAR, and the Privacy Act. Without this, gap assessments and audit processes cannot function.
From that inventory, four capabilities need to be in place.
Continuous monitoring for bias, drift, and performance degradation. AI models do not stay stable. A model that was accurate at deployment may not be accurate six months later. Automated monitoring catches this before regulators do.
Independent assurance for high-impact systems. Internal audit and external review processes need to cover AI systems with material customer or operational impact. This cannot be delegated to the team that built the system.
Testing frameworks built for AI. Standard software testing does not address algorithmic bias, adversarial inputs, or the behaviour of AI-generated code. Testing frameworks need to evolve to cover these risks explicitly.
Automated compliance tooling. Governance, Risk, and Compliance (GRC) platforms can automate the monitoring and reporting that manual processes cannot keep up with. Predictive compliance tools can identify non-compliant states before they become audit findings. SaaS Security Posture Management (SSPM) tools provide continuous measurement of security controls against regulatory baselines.
The Gap Between Adoption and Governance Closes in One Direction
Regulators are not going to slow down their expectations to match the pace of industry governance. The direction of travel is more scrutiny, not less.
To get ahead of this do three things. Build board-level AI literacy that enables genuine challenge, not just approval. Establish clear ownership for AI decisions at the individual accountability level. Instrument their AI systems for continuous oversight rather than periodic review.
APRA has been explicit about what it is looking for. The question is whether your governance framework reflects what your AI systems are actually doing, not what the documentation says they do.
Most organisations deploying AI-powered technology assume the hard part is the technology. The Bunnings case shows the hard part is the governance.
Between November 2018 and November 2021, Bunnings installed facial recognition technology (FRT) in a number of its Australian and New Zealand stores. The system captured the face of every person who walked through the door, compared it against a database of individuals previously involved in criminal conduct or incidents involving staff, and alerted store employees when it found a match. The facial data itself was deleted in milliseconds. Nobody was told it was happening.
That decision triggered a regulatory investigation that took six years to resolve, a determination by the Privacy Commissioner, an appeal to the Administrative Review Tribunal, and a still-open question about whether the matter ends there. The legal outcome was mixed, but the compliance lessons are not.
The Regulator Found Bunnings Collected Biometric Data, Whether It Knew It or Not
On 29 October 2024, Privacy Commissioner Carly Kind found that Bunnings had breached several Australian Privacy Principles (APPs) under the Privacy Act 1988 (Cth). The breaches related to transparency, collection, and notification. In the Commissioner’s view, FRT was a highly privacy-invasive tool that interfered disproportionately with the privacy of all store entrants, not just the small number of people it was designed to identify.
Bunnings appealed and on 4 February 2026, the Administrative Review Tribunal (ART) partially set aside the determination and upheld the findings that Bunnings breached APP 1 (transparent management of personal information) and APP 5 (notification of collection). It overturned the finding on APP 3 (collection of sensitive information without consent).
The APP 3 decision turned on a specific provision. Under section 16A of the Privacy Act, an organisation can collect sensitive information without consent when obtaining that consent would be unreasonable or impracticable, and when the collection is reasonably necessary to prevent a serious threat to the life, health, or safety of any individual or to public health or safety. The Tribunal accepted that Bunnings faced genuine and serious threats, noting the scale of retail crime, the nature of incidents involving staff and customers, and the fact that many products in a Bunnings store can be used as a weapon. On those facts, the Tribunal found FRT to be a reasonable and proportionate response.
The Privacy Commissioner subsequently confirmed she would not appeal the Tribunal’s decision.
The Tribunal was explicit on that this outcome is not a green light for biometric deployment, but is a fact-specific finding about a specific organisation facing specific threats, with specific controls in place. Other organisations attempting to rely on the same reasoning without equivalent circumstances will likely find the exception does not apply to them.
The Breaches Upheld Tell the More Useful Story
Bunnings succeeded on the consent question but did not succeed on governance, and that distinction matters more for most organisations thinking about where their own exposure sits.
“Milliseconds” Is Not a Defence Against Collection
Bunnings argued that because the facial matching process occurred in RAM and the data was deleted in milliseconds, it had not technically “collected” personal information. The Tribunal rejected this.
The Tribunal found that Bunnings collected facial images captured by CCTV cameras, that those images constituted biometric information, and that biometric information is sensitive information under the Privacy Act, regardless of how briefly it was held. The legal threshold for collection is low. If your system processes personal information, even in real time, even transiently, you have collected it, and privacy obligations follow.
This matters beyond FRT. Any organisation using automated systems that touch personal information, including AI tools processing voice, images, location, or behavioural data, should assume collection is occurring and govern accordingly.
A Sign on the Door Is Not Enough When You Are Taking Someone’s Face
Bunnings had posted privacy notices at store entry points. The Tribunal found this insufficient.
The Tribunal’s reasoning was that given the sensitive nature of biometric data, the size of Bunnings as an organisation, and the resources available to it, more was required. Customers needed to know the specific type of information being collected, the technology being used, the purposes behind collection, and what happened if they chose not to provide it.
Generic privacy policies are written to cover everything. Biometric collection requires notices written to cover that specific thing. When the information being collected is sensitive, specificity is required.
The Absence of a Documented Risk Assessment is Treated as a Governance Failure
This is the finding with the broadest application.
APP 1.2 requires organisations to take reasonable steps to implement practices, procedures, and systems that ensure compliance with the APPs. The Tribunal found that Bunnings failed to meet this standard. The steps taken prior to deployment were described as “random enquiries and actions”. No formal, structured, and documented privacy risk assessment had been conducted before the FRT system went live.
The Tribunal’s language is deliberate. When an organisation collects sensitive information, it faces a serious intrusion of privacy. Serious intrusions require formal responses, not ad hoc ones. A Privacy Impact Assessment (PIA) conducted before deployment, documented, and retained, is what reasonable steps look like in this context.
The compliance lesson here goes beyond privacy. Regulators across multiple domains, including privacy, cybersecurity, and financial services, are increasingly focused on whether organisations can demonstrate that their governance preceded their deployment decisions, not followed them. If the PIA exists only after a complaint is made, it does not count.
The Regulatory Environment Around This Decision Has Changed
The Bunnings case was decided under the existing Privacy Act 1988 framework. That framework is now amended.
The Privacy and Other Legislation Amendment Act 2024 (Cth), which received royal assent in December 2024, introduced changes that take effect progressively through 2026. Three are particularly relevant to organisations using AI or data-intensive technologies.
The definition of personal information now explicitly covers inferred or generated data, such as the outputs of machine learning models. AI generating risk scores, predicted preferences, and behavioural classifications are now personal information for the purposes of the APPs. Organisations that assumed model outputs sat outside the Act’s reach need to revisit that assumption.
From December 2026, organisations must disclose in their privacy policies when personal information is used in substantially automated decision-making that has significant effects on individuals. This requires organisations to audit existing systems now, identify where automated decisions are being made, what professional function they perform, and build disclosure workflows before the obligation takes effect.
A Children’s Online Privacy Code is being developed, with registration expected by December 2026. Organisations whose services are likely to be accessed by minors face heightened obligations and need to begin preparing ahead of the code’s commencement.
A further reform expected in the second tranche of amendments is a general “fair and reasonable” test for the collection, use, and disclosure of personal information. If introduced, this would shift the compliance question from whether a notice is given to whether the underlying practice can withstand objective scrutiny. An AI system that targets individuals based on inferred sensitive characteristics, or that generates unexpected outcomes from opaque models, may fail that test regardless of what the privacy policy says.
What Organisations Should Do Now
The Bunnings case illustrates a familiar pattern. The organisation deployed the technology first and worked out the governance as concerns arose. That sequence creates risk, and the regulatory direction of travel is toward holding organisations accountable for the order in which they do things.
The following steps address the specific failure modes the Tribunal identified.
Conduct a Privacy Impact Assessment before deployment, not after. For any system that collects sensitive information, a formal, structured, and documented PIA is what APP 1.2 compliance looks like. This assessment should identify the risks, document the controls, and record who made the decisions and why, and should exist before the system goes live.
Write specific notices for specific technologies. If your organisation uses FRT, AI-powered monitoring, or any other tool that collects biometric, health, or other sensitive information, the privacy notice for that collection needs to name the technology, describe what is collected, state the purpose, and explain the consequences of not providing the information. Generic notices covering all data collection are not adequate for high-risk processing.
Audit your systems for the amended definition of personal information. AI model outputs, including risk scores, classifications, and generated assessments, now sit within the Act’s scope. Organisations using AI tools that generate outputs about individuals should review how those outputs are managed, retained, disclosed, and corrected.
Map automated decisions and prepare for disclosure. Identify where your organisation uses personal information in automated or substantially automated decision-making with significant effects. Build the workflows and policy language needed to disclose this before December 2026.
Maintain records of AI-assisted decisions. Accountability under the APPs requires being able to demonstrate what decisions were made, on what basis, and with what human oversight. If that record does not exist, accountability cannot be demonstrated.
The Bunnings outcome is sometimes framed as a partial win for business. In a narrow legal sense, that is accurate. On the question of governance, Bunnings did not win and was found to have deployed an invasive technology without adequate transparency, without specific notification, and without a documented risk assessment. Those findings were not overturned.
The compliance clock started the moment the technology was deployed. For organisations considering similar decisions, it starts now.
Most organisations deploying AI already have risk management processes that work. Enterprise risk frameworks, compliance programs, audit cycles, three lines of defence. What they don’t have is a clear way to govern something genuinely new.
Not new in the sense of unfamiliar process or regulation. New in the sense that the failure modes don’t always look like failure until after the fact. A model that produces biased outputs does so quietly, at scale, with no error log. A third-party AI system can introduce exposure through its training data, not only its integration points. These are risk problems that just don’t surface the usual way.
The temptation is to build dedicated AI governance structures alongside existing ones. A separate AI risk register. A bespoke framework standing apart from everything else.
That approach tends to fail, not because the intent is wrong, but because it creates a parallel process that nobody owns. The people who understand AI risk (data scientists, engineers) sit outside the new structure. The people who run risk (compliance, audit, legal) don’t have the technical fluency to govern it. The framework becomes a documentation exercise.
There is a better path: extend what you already have.
This guide outlines how to embed AI-specific risk checkpoints into existing risk management processes across three phases; development, deployment, and production.
The Core Principle: Extend, Don’t Duplicate
Your existing enterprise risk management program already handles model uncertainty, third-party risk, data governance, and regulatory compliance. AI introduces new variations on each of these: model drift instead of model change, algorithmic bias instead of human error, explainability gaps instead of audit trail gaps. The underlying risk categories are familiar. What’s new is where to look for them, how quickly they can materialise, and how they interact with each other.
One difference that does require a structural adjustment is monitoring cadence. AI systems are not static. A model validated at deployment will behave differently as real-world data diverges from training data, as usage patterns shift, or as the model is fine-tuned over time. Periodic review cycles that work well for stable systems are a poor match for that rate of change. Continuous monitoring, with clear alert thresholds and defined response triggers, is the appropriate equivalent, and it belongs inside your existing operational risk program, not in a separate process.
Checkpoints Across the AI Lifecycle
Development: Apply Controls Before They Cost More
Risk controls applied at the start of development are far cheaper than those applied after deployment.
Inventory and classification. Establish a register for every AI model in use, covering its purpose, training data sources, performance benchmarks, and risk tier. The register needs to cover third-party and embedded models, not only those built in-house, and it needs to be maintained as models evolve.
Threat modelling. Standard threat modelling in software development focuses on system access and data exposure. AI adds two categories: model behaviour risks (such as susceptibility to prompt injection, where a malicious input manipulates the model’s output) and training data risks (such as poisoning, where corrupted data degrades model performance). Both require explicit scenario testing.
Third-party model assessment. Third-party AI model assessment belongs inside existing supplier risk processes, not alongside them. The criteria differ from standard software: vulnerability and bias assessments, training data provenance, and licensing terms all need to be part of the standard supplier questionnaire for any AI component. Supplier security attestations and independent bias testing are the baseline.
Data provenance documentation. The traceability requirements for AI training and fine-tuning datasets are analogous to chain-of-custody requirements elsewhere in risk and compliance. Collection methods, licensing terms, usage history, and any known quality issues should be documented before a model moves forward. This documentation is what makes a bias incident investigable after the fact.
Secure development practices. AI-assisted code generation introduces two specific exposures that standard development controls don’t catch well: insecure patterns that context-unaware static analysis tools miss, and credentials committed through AI-assisted workflows. Context-aware static application security testing and automated secrets detection in CI/CD pipelines address both. Human review of AI-generated code before it progresses is a control, not a formality.
Deployment: Test for AI-Specific Failure Modes
Standard pre-deployment testing checks whether a system does what it’s supposed to do. AI deployment testing must also check how a system fails and whether it can be manipulated.
Adversarial testing. Simulate attacks on model behaviour: malicious prompts, crafted adversarial inputs, stress conditions. The objective is to find the boundaries of the model’s guardrails before someone with harmful intent does.
Access control validation. AI agents operating with system permissions require particular scrutiny. Test whether agents can escalate privileges beyond their intended scope by simulating compromised credentials. Role-based access controls need to be verified against the model’s actual behaviour, not just its configuration.
Deployment gates. Real-time risk scoring integrated into your CI/CD pipeline allows deployments that exceed defined risk thresholds to be blocked automatically. Container and infrastructure scanning ensures the environment the model is deployed into is as secure as the model itself.
Production: Oversight That Matches the Pace of Change
A model that passes deployment testing is not a model that stays safe. Production is where risk management becomes an ongoing discipline.
Continuous monitoring. Production monitoring for AI should track performance, fairness metrics, and security indicators in real time, with alert thresholds defined for model drift, anomalous output patterns, and unusual usage. The monitoring framework is familiar. The metrics being tracked are what needs to be extended.
AI-specific incident response. Existing incident response plans cover system failures and security breaches. AI introduces failure modes that require their own response procedures: biased outputs propagating at scale, hallucinations in customer-facing applications, or agentic systems taking unintended actions. These scenarios need defined containment steps, clear escalation paths, and remediation procedures that are tested before they’re needed.
Post-incident learning. A single AI incident should trigger a review of the systems and controls that allowed it. The same incident happening twice means the review didn’t produce change. Build post-incident reviews into your AI governance cycle and track whether policy updates are actually implemented. One mistake is the message to improve the system. Two of the same mistakes means the system hasn’t changed.
Regulatory tracking. The regulatory landscape for AI is moving quickly. The EU AI Act, ISO 42001, and sector-specific guidance from financial and healthcare regulators are all active and developing. Monitoring this landscape belongs inside existing regulatory tracking processes, with the same ongoing attention applied to other evolving obligations.
Using the NIST AI RMF as a Foundation
The NIST AI Risk Management Framework provides a structured approach for managing AI risk across the lifecycle. Its core is four functions (Govern, Map, Measure, and Manage) designed to operate as a continuous cycle, not a sequential checklist.
Govern is the cross-cutting function at the centre of the framework. It addresses organisational risk culture, accountability structures, and AI-related policies. It answers the questions: who approves high-risk AI deployments, how are third-party models introduced, and how are resources allocated for safety testing.
Map focuses on understanding the AI system’s context, scope, and potential harms before decisions are made. After completing the Map function, organisations should have enough information to make an initial go or no-go decision about whether to proceed with a given AI system.
Measure covers assessing identified risks through a combination of quantitative and qualitative methods, evaluating performance, fairness, transparency, and security.
Manage addresses risk response: implementing controls, prioritising mitigations, and planning for incidents.
The framework is voluntary and deliberately flexible. Organisations customise it through profiles that reflect their specific context, regulatory environment, and risk appetite. If you already operate ISO 27001, SOC 2, or the NIST Cybersecurity Framework, treat the AI RMF as an overlay. Map your existing controls to the Govern and Manage functions first, then add AI-specific requirements where Map and Measure reveal gaps.
Start With Visibility
Before you can extend your existing controls, you need to know what you’re governing.
A practical starting point is the model inventory. Visibility across all AI in use, including models embedded in third-party software, is the prerequisite for everything else. Without it, you can’t apply risk tiers, assess suppliers, or set meaningful monitoring thresholds.
From that foundation, existing frameworks can be extended systematically: AI criteria into model risk policy, third-party assessment processes, incident response plans, and operational monitoring programs.
The risk controls that will protect your organisation are the ones embedded into processes people already follow, owned by people who already have accountability. That’s the existing risk function. The scope has changed. The structure doesn’t need to.
Your AI vendor showed you an accuracy rate above 90 percent. Your proof of concept worked. Your board approved the budget. So why is your production deployment making decisions you can’t explain, generating outputs you can’t trace, and occasionally causing damage you only discover after the fact?
The answer is mathematics.
The Probability Your Vendor Didn’t Show You
Enterprises are moving AI from single-step assistants into autonomous, multi-step agents. These agents don’t just respond to a prompt; they plan, execute, and hand off results to other agents. The workflows they operate in routinely span dozens of steps: retrieve data, reason about it, call a tool, validate the result, pass it on.
When you chain sequential decisions like this, the overall success rate is the product of each individual step’s success rate. This is Lusser’s Law, a principle well-established in reliability engineering.
An AI agent that performs at 95 percent accuracy per step sounds impressive. A 20-step workflow run at that accuracy succeeds only 36 percent of the time. At 90 percent accuracy, the same workflow succeeds 12 percent of the time. At 85 percent, it succeeds 4 percent of the time.
You didn’t approve a system that fails on four out of five attempts. But that may be what you deployed.
The implications are particularly severe in high-stakes environments. Healthcare workflows for prior authorisation or EHR data entry routinely span 50 to 200 discrete actions. A 60 percent error-free completion rate in that context is a compliance breach that delays patient care.
Oxford researcher Toby Ord quantified this degradation in a 2025 study covering 170 software engineering, machine learning, and reasoning tasks. Ord found that AI agent performance declines exponentially with task duration, and proposed that each agent can be characterised by its own “half-life”; a constant rate of failure for every minute a human would take to complete the same task. Claude 3.7 Sonnet, one of the models tested, had a half-life of approximately 59 minutes: a 50 percent success rate for a one-hour task, 25 percent for a two-hour task, and 6 percent for a four-hour task. The longer the task, the less you can rely on the output.
Why Errors Don’t Stay Where They Start
The mathematical degradation alone would be manageable if errors stayed isolated. They don’t.
The Open Worldwide Application Security Project classifies “Cascading Failures” as a top-tier risk in agentic AI deployments. Their definition is precise: a cascading failure occurs when a single fault, such as a hallucination, a corrupted tool output, a misread instruction, propagates across autonomous agents and compounds into system-wide harm.
The propagation happens because agentic systems communicate in natural language or loosely-typed data schemas. A semantic error, something that is wrong but grammatically coherent, passes validation checks and moves downstream as if it were correct. Subsequent agents receive it as verified fact. In multi-agent systems, by the time a problem surfaces, the original error is buried under several layers of decisions that all treated it as ground truth.
This is what OWASP describes as “memory poisoning”: one agent hallucinates a piece of information, stores it in shared memory, and every downstream agent inherits the contamination. Engineers can see the symptoms but cannot easily find the source. The failures are quiet, diffuse, and cumulative.
What Failure Looks Like in Production
Two incidents from 2025 illustrate what this mathematics looks like in the real world.
In July 2025, SaaStr founder Jason Lemkin used Replit’s AI coding agent to build a business contact database. After instructing the agent to freeze the code, the agent deleted the entire production database, erasing records for over 1,200 executives and nearly 1,200 companies. The agent then fabricated thousands of records to fill the void and incorrectly told Lemkin that a rollback was impossible. Lemkin was eventually able to recover the data manually. Replit’s CEO publicly apologised and committed to implementing automatic separation between development and production environments.
This was not a single catastrophic error. It was a sequence of small misalignments, a misread instruction, an unauthorised action, a cover-up, and a false status report, each compounding the last.
In February 2025, a user asked OpenAI’s Operator agent to compare grocery prices. The agent compared them, then completed a $31.43 Instacart delivery purchase without authorisation. OpenAI’s stated protocol required user confirmation before any purchase. The agent bypassed it. The compounding failure was subtle: the agent had developed a slightly incorrect model of its own permitted workflow, and no checkpoint caught the deviation before it became a real-world transaction.
Neither incident involved a model that was unreliable in testing. Both involved systems that failed because production conditions, for example ambiguous instructions, multi-step execution, real consequences, created the conditions where compounding errors thrive.
The Benchmark Problem
When vendors present accuracy figures, they are almost always presenting benchmark scores. The gap between benchmarks and production deserves your direct attention.
SWE-bench Verified, a widely cited software engineering benchmark, showed top agents achieving success rates above 70 percent. When Scale AI introduced SWE-bench Pro, designed to reflect realistic task complexity using diverse, multi-file codebases, those same top-tier agents achieved at most 23 percent on the public set and 17 percent on the private set at the time of publication. This is a structural difference: controlled benchmarks measure performance on curated tasks; production measures performance on real work.
The same gap exists in your observability tooling. When an agent fails due to an ambiguous input or a hallucinated memory entry, the underlying API calls may still register as successful. Standard monitoring infrastructure has no way to distinguish a technically-completed call from one that produced a wrong answer. The failure is invisible to your existing dashboards.
You are likely measuring availability, not accuracy. Those are not the same thing.
What Reliable Deployment Requires
The engineering response to compounding errors is not primarily about choosing better models. It is about building systems that account for the mathematics from the start.
Calculate the compound probability before you deploy. Map the longest realistic workflow in your system. Multiply the step-count against your measured per-step accuracy. If the resulting success rate falls below your acceptable threshold, you have a design problem, not a vendor problem, that no benchmark score will fix.
Separate reasoning from execution in high-stakes workflows. For environments where errors carry compliance, financial, or safety consequences, consider using AI to generate deterministic execution scripts at build time, rather than making probabilistic decisions at runtime. This removes the compounding dynamic from the most consequential parts of your workflow.
Build validation at every boundary. Multi-agent systems require guardrails at every boundary: input, output, and inter-agent handoffs. Each boundary is a point where errors can either be caught or amplified.
Implement circuit breakers. An orchestration layer that monitors agent performance and isolates agents after consecutive failures, routing tasks to alternatives or degrading gracefully to simpler processing that prevents a single malfunctioning agent from contaminating an entire workflow.
The Organisational Question
Most organisations deploying AI agents have invested heavily in the models and very little in the measurement. Observability infrastructure for agentic systems is not an optional upgrade but the mechanism by which you find out whether the system you deployed is the system that is actually running.
The incidents described above are not arguements against AI. They are practical demonstrations of what happens when systems encounter real conditions without the safeguards their architecture requires.
The mathematics do not change. But your response to them can.
AI has moved from experimental technology to material enterprise risk faster than most governance frameworks were built to handle. Across global boardrooms, directors now recognise AI as a force capable of reshaping business models, altering risk exposure, and redefining organisational culture. Recognition, however, is not the same as readiness.
The structural challenge is that boards are still wrestling with who owns AI oversight, how critical issues reach the table, and what level of technical understanding is genuinely required as AI embeds itself deeper across the enterprise.
The stakes are high. Directors face complex decisions about technology architecture, workforce transformation, data governance, and ethical boundaries. However, often there is no precedent to guide them. Waiting for certainty is not a viable strategy. Inaction carries as much risk as a wrong decision, and the window for learning from first-movers is closing faster than most boards appreciate.
Oversight Structures Are Being Rebuilt From the Ground Up
The rapid integration of AI requires a fundamental change in how boards structure their committees, schedule their engagement, and define their accountability. Historically, boards could rely on periodic management updates and established risk matrices. AI doesn’t fit that rhythm.
Investors and regulators are increasingly demanding formalised, board-level oversight of AI governance. Some organisations are establishing dedicated AI Oversight Committees. Others are integrating these responsibilities into existing Audit or Technology Committees. The structure chosen matters less than the outcome it must produce: a clear line of sight into how AI is deployed, managed, and monitored across the organisation.
Oversight Model
Primary Focus
Advantages
Challenges
Dedicated AI Committee
Comprehensive AI strategy, ethics, and risk
Deep focus; clear accountability; specialised expertise
Risk of siloing AI issues from broader business strategy
May lack the technical depth required for nuanced AI evaluation
Technology Committee Integration
Architecture, deployment, and cybersecurity
Aligns AI with broader IT strategy; strong technical oversight
Risk of overlooking ethical and workforce transformation impacts
Director Fluency Must Evolve Beyond General Tech Skills
General technological literacy is no longer sufficient at the board level. What is now required is AI fluency: a specific, working understanding of the capabilities, limitations, and strategic implications of AI technologies, including generative AI, autonomous agents, and operational automation.
The temptation is to solve this by recruiting a single technologist to the board. That solves the optics, not the problem. Every director needs a baseline appreciation of AI and data ethics, enough to challenge management assumptions, evaluate proposed architectures, and ask the questions that matter: How is our data being used to train these models? What biases are present? How does this deployment interact with our regulatory obligations? Does this decision align with where we are taking the business in five years?
Boards that delegate AI understanding to one person are one resignation away from a governance gap. Fluency must be built across the entire directorate.
Trust Is Built Through Constraints, Not Just Values
Trust in AI systems cannot be assumed. It must be earned through experience, through honest evaluation of where AI performs well and where it does not, and critically, through the governance structures that keep AI behaviour aligned with organisational values and regulatory obligations.
High-level principles are not enough. Boards must establish concrete, ethical guardrails: the operational boundaries within which AI systems must function, with clear accountability when those systems affect employees, customers, or the public.
Effective AI oversight requires four specific, tailored processes:
AI-Tailored Risk Management:Traditional risk frameworks need adaptation to address what makes AI genuinely different; algorithmic bias, model drift, and complex data privacy exposure that evolves as models are updated and retrained.
Transparent, Outcome-Based Reporting: Boards must demand metrics focused on actual outcomes and real-world impacts, not just technical performance indicators. A model performing well on benchmarks can still cause significant harm in deployment.
Human-Centric Supervision: In high-stakes decisions, human oversight is a design requirement. Maintaining meaningful human accountability is both an ethical obligation and an increasingly firm regulatory expectation.
Human Impact Measurement: AI changes how work gets done. Boards need regular visibility into the downstream effects on their workforce: shifting training requirements, changes in employee satisfaction and retention, and evolving hiring needs. These are governance questions, not solely HR ones.
The transition to an AI-driven enterprise is becoming less optional, and it will not wait for boards that want more time to deliberate. Organisations that build genuine director fluency, establish clear oversight, and implement governance frameworks grounded in real constraints will be the ones that manage AI risk without sacrificing its strategic potential.
Galdren’s artificial intelligence practice regularly advises companies and boards on the adoption of AI governance policies and assessing AI risk. Please contact us with any questions.
Every policy document begins with someone else’s bad day. This one is no different. These rules were written after AI systems behaved unexpectedly in production, after agents took actions that couldn’t be undone, after data went somewhere it shouldn’t have. They are not theoretical. They are the residue of consequences. Murphy’s Law has always applied to software. Applied to AI agents, it applies with unusual force. AI agents now read your documents, call your APIs, write and execute code, query your databases, and send communications on behalf of your users. That capability is the point. But it also means every security failure mode in traditional software now has a faster, harder-to-predict counterpart, and several entirely new ones. An agent that can write to a database can be manipulated into deleting one. An agent that can send emails can be convinced to send the wrong ones. An agent with access to your systems will eventually encounter an input designed to misuse that accessl by an attacker, by an edge case, or by its own unexpected behaviour. The attack surface for AI systems is language itself. You cannot enumerate every bad input. You cannot anticipate every manipulation. You cannot assume that because a system worked correctly a thousand times, the thousand-and-first will go the same way. What you can do is design systems that fail safely, fail loudly, and recover deliberately. That is what these rules are for.
What These Rules Are Trying to Achieve
These rules have three objectives.
Shrink the blast radius. When something goes wrong – and something will – the damage should be contained. Minimal privilege, rollback-first design, reversible actions by default, and human approval for high-stakes decisions mean that a failure is an incident you recover from, not a catastrophe you explain.
Make exploitation harder than legitimate use. Allowlists over blocklists, validated inputs, structured outputs, and authenticated actions ensure the path of least resistance runs through your controls, not around them. Attackers follow incentives. Design accordingly.
Create systems you can understand under pressure. Logging, monitoring, auditable logic, and documented decision-making mean that when an incident happens, you can diagnose it, contain it, and fix it. Rather than guessing at what the agent did and why.
Most of these rules apply lessons from decades of infrastructure and application security to a new class of system. The novelty is that AI agents can be manipulated through natural language, can behave unexpectedly at scale, and can take real-world actions faster than any human can supervise. That combination makes the familiar disciplines of least privilege, defence in depth, and fail-safe design more important, not less.
How to Use This Document
These rules are written for engineers building, deploying, or maintaining AI systems. They cover infrastructure, production operations, and security controls, not prompt engineering or model-specific optimisation. Treat them as a checklist for new systems and a diagnostic for existing ones. Integrate them into your own documentation, processes and workflow. Where a rule does not apply, document why. Where a rule creates tension with a business requirement, escalate. Do not simply bypass it. The pattern in these rules is consistent: teams that skip these steps encounter the consequences eventually. The goal is that you learn the pattern here.
Never inject untrusted input directly into a prompt. Structuring prompts with raw user input, unvalidated data, or external content is the AI equivalent of SQL injection. Use prompt templates with clearly delimited, sanitised inputs. Separate instructions from data in every prompt. This is the primary defence against prompt injection attacks.
Treat all inputs to your AI as untrusted. Validate every prompt, message, web page, email, document, and data source before passing it to a model (including other inputs you’ve created!). Reject inputs that fail validation. Never attempt to “fix” a malicious or malformed prompt. Always validate first, then either escape or sanitise hazardous content before it reaches the model. This rule includes models specifically designed to validate input.
On any error or unexpected AI behaviour, roll back and fail safely. Never allow an AI agent to continue a partially completed action. Never fail open. Roll back fully and start again from a known safe state. This is especially critical for agentic tasks with real-world consequences (sending emails, executing code, calling APIs).
Human-in-the-loop for high impact actions. To limit excessive autonomy, all high-impact, irreversible agent actions such as sending communications, modifying records, executing transactions, should require explicit human approval before proceeding. Expect the threshold for autonomy to shift over time as trust is established, but evaluate that threshold against the risk of failure, not the number of past successes.
Exercise extreme caution when AI agents make system calls, execute code, or call external APIs. Passing AI-generated output directly to a system call, shell command, or code interpreter is one of the highest-risk capabilities you can give an AI agent. If the agent doesn’t need access, don’t allow it. If it does, limit it to only what’s needed.
Protect sensitive data before it reaches an AI model. Mask, anonymise, or hash personally identifiable information (PII) and sensitive data before it is included in any prompt or context window. What you send to a model may be logged, retained, or exposed. Do not send data the AI does not need. Only send the minimum data necessary.
Sanitise and encode all AI output before downstream use. You have no control how your AI-generated output is going to be used. Before sending output via a user interface, a database, an API call, or a system command, treat it as you would any user input: validate, encode, and sanitise it to prevent attacks, misuse, and unintended execution.
Authorise every AI agent action individually. Do not assume that because a user or system has authenticated once, all subsequent AI agent actions on their behalf are permitted. Validate authorisation for every action an AI agent takes, especially for sensitive operations.
AI agents should operate using minimal access accounts. Apply the principle of least privilege strictly. Every access grant should be explicit, minimal, and reviewed. Individual and admin accounts grant excessive privilege and create accountability gaps. An agent that can do everything will eventually do something you did not intend.
Design AI systems to assume they will be manipulated. Plan for prompt injection, jailbreaks, model manipulation, data poisoning, and unexpected outputs. Each aspect of the system is a potential exploitation target, design accordingly.
Never trust AI output blindly. Validate AI-generated content before using it, especially when it will be used in code, database queries, system commands, or other sensitive contexts. AI output can be incorrect, manipulated, or adversarially crafted.
Use a secrets management tool. Use secrets scanning on every code commit to catch accidental exposure. This is especially critical for AI systems, where prompt content may be logged, cached, or leaked through model outputs.
Log, monitor, and alert on all AI system errors and unexpected behaviours. AI errors are signals so treat them accordingly. Do not allow AI systems to fail silently. Log every significant AI input, decision, output (redact PII and sensitive data), and telemetry. AI errors might be context drift, distortion, or misalignment so monitor for anomalies, unexpected behaviour, and policy violations that alert on threshold or trends as well as hard failures. This applies to AI APIs, agents, and pipelines, not just user-facing interfaces.
Use allowlists, not blocklists, to control what AI agents can do. Define explicitly what actions, tools, and data sources an AI agent is permitted to use. Blocklists are trivially bypassed. Allowlists are easier to maintain and far more reliable.
Prefer reversible actions. Design the system so if two paths accomplish the same goal, the agent chooses the reversible one by default. Irreversibility amplifies every other failure.
Secure the AI supply chain. This includes the models, datasets, tools, SDKs, vector databases, and embedding pipelines you use. Validate that every component you depend on is from a trusted source and is being used safely. Lock down your AI development environment, version control, CI/CD pipeline, and any system used to build or deploy AI. Validate this regularly.
Classify all data before sending it to an AI. Know what you are sending, how sensitive it is, and whether it is appropriate to send. Document sensitive data flows into and out of AI systems. Encrypt sensitive data in transit and at rest. Test these flows for security.
Use structured, typed inputs and outputs for AI systems. Avoid ambiguous, loosely formatted prompts and responses. Define expected input and output schemas. Use JSON schemas, structured outputs, or typed response formats where supported. Ambiguity in AI is a security and reliability risk.
Default all AI systems to the most restrictive settings. Require explicit configuration to expand permissions or capabilities. If you set restrictive defaults, users are more likely to leave them in place; which is exactly what you want.
Model the threats for all AI systems. Include AI-specific threats: prompt injection, data poisoning, training data extraction, adversarial inputs, model inversion, jailbreaking, denial-of-service, and agent misuse. Mitigate or eliminate all threats assessed as significant.
Secure the training and data pipeline end-to-end Vector stores, embedding pipelines, and retrieval logic are a distinct attack surface. Poisoned documents retrieved at query time can manipulate agent behaviour invisibly.
Verify the integrity of AI models and agents before deployment. Use model signing, checksums, or another integrity verification method to ensure models have not been tampered with. Immutable builds and verified deployments are the standard.
Apply appropriate API security controls to all AI endpoints. Such as rate limiting, authentication, authorisation, input validation, and monitoring to ensure protection against traditional attacks and failures.
Rate limit all AI agent actions. Nothing an AI agent does should be unlimited. Apply limits at every layer: API calls, tool invocations, file operations, external requests, and token consumption. Unlimited AI agents create unlimited risk.
Perform all critical AI validation and decision-making on the server side. Client-side safety controls can be intercepted or bypassed. Trust only what happens on systems you control.
Keep AI models, frameworks, SDKs, and dependencies up to date. Outdated components are a known risk, especially in fast-moving AI ecosystems. Where possible, automate updates and patching. Slow release and update processes are a serious organisational risk and should be treated as a priority for improvement. Temper this against not updating too soon as supply chain attacks exploit organisations that update uncritically.
Minimal agent footprint. At each step of execution, an agent should request only what it needs now, release access when done, and avoid accumulating permissions or retaining sensitive data across steps. This is least privilege applied dynamically, per action.
Enable strict safety and content settings in all AI frameworks and platforms. If a framework or API offers a strict mode, safety classifier, or content filter then turn it on. These are not enabled by default in all systems. Check, and enable them explicitly.
Implement anomaly detection for all AI agent interactions. Monitor for abuse patterns, eg unusually high request volumes, adversarial inputs, or attempts to probe the AI’s limits. Implement defences against prompt flooding, token exhaustion attacks, systematic jailbreak attempts, and bot-driven misuse. Log all interactions. Alert on thresholds that suggest an attack may be in progress.
Retest AI systems for safety regressions after every update or change. Safety testing is not a one-time activity. Build automated red-teaming, adversarial testing, output classifiers, and bias evaluation tools, and prompt injection tests into your CI/CD pipeline where possible.
Protect all AI infrastructure comprehensively. This includes the model endpoints, repositories, vector databases, embedding stores, training pipelines, and supporting systems. All must be hardened, monitored, logged, patched, and tested for security.
Apply security design principles to AI systems. Where possible, enforce least privilege (limit what the AI can access and do), zero trust (never assume an AI-to-AI call is safe), defence in depth (layer multiple controls), and attack surface reduction (limit the AI’s reach to only what is required). Do not rely on a single guardrail. Apply stricter principles for higher-risk AI systems.
Control what files and data AI agents can access, read, write, or delete. Use strict access controls. Treat all files produced by or passed through an AI agent as potentially untrusted. Ensure important data is backed up, stored encrypted, and protected with monitored access controls.
Prevent race conditions in AI agent workflows. When multiple agents or processes interact with shared state or external systems, use proper coordination, locking, and sequencing to prevent conflicts. Most modern AI orchestration frameworks provide tools for this.
Use established identity, authentication, and access control systems for all AI agent interactions. Do not build your own AI authorisation logic from scratch. Existing solutions are well-tested. Writing custom access control for AI agents introduces serious risk.
Apply encrypted, certificate-validated connections. Use HTTPS and validated certificates or similar for every AI API call. Follow your organisation’s cryptographic standards, or those of OWASP, NIST, or your relevant government body. Choose the strictest applicable standard and check that certificates are valid and from the expected host. Do not connect to unverified AI endpoints. This integrity check prevents man-in-the-middle attacks on AI API calls.
Follow a secure AI development lifecycle. Integrate safety and security activities at every stage: design, development, testing, deployment, and monitoring. If your organisation does not have a defined lifecylce, create one. Add safety activities to your existing SDLC where possible.
Select AI frameworks and platforms with strong, built-in safety features and guardrails. Do not write your own safety controls from scratch when established solutions exist. Always use a supported, up-to-date version of any AI framework or SDK. Then avoid bypassing content moderation, or undermining output constraints. If the defaults do not fit your use case, work with your security team, do not simply disable them.
Manage agent state explicitly and consistently. Establish a standard approach to manage context windows or agent memory. Inconsistent context management introduces subtle bugs and security risks that are difficult to detect and diagnose. Initialise AI agent state explicitly before use. Do not rely on implicit defaults or assume prior state is clean. Uninitialised or stale state in AI agents produces unpredictable and potentially unsafe behaviour.
Do not use real user data for AI development, testing, or fine-tuning without proper anonymisation and approval. Raw production data in non-production AI environments is a serious privacy and compliance risk. Use purpose-built anonymisation or synthetic data generation tools, not home-grown masking scripts.
Protect all AI API keys and administrative accounts with multi-factor authentication. Use long, unique, complex credentials for every account with elevated AI system access. Use a password manager. Reset credentials immediately if you suspect a breach. Rotate keys regularly.
Offer strong authentication to users of AI-powered systems. Provide MFA where possible. Implement defences against credential stuffing, supply chain, and malware attacks on their workspace. Carefully log all access and alert on suspicious patterns. Geoblock their access limiting exposure if access is breached.
Create response plans for AI failure scenarios. Assume a failure will happen and prepare in advance. Practice these scenarios where possible. Ensure your AI systems are included in business continuity and disaster recovery planning.
Audit AI system configurations and permissions at least annually. Review what models are in use, purpose, what access they have, what data they can reach, and whether their configurations remain appropriate.
Manage AI session tokens and credentials securely. Apply the same standards as secure cookie management: limit scope, set expiry, enforce secure transport, and never expose credentials unnecessarily.
Protect proprietary AI prompts and agent logic from unauthorised exposure. System instructions, reasoning chains, and agent architectures can contain competitive intelligence and safety-critical logic. Where appropriate, treat them with the same care as source code.
Protect AI interfaces from cross-origin and cross-site abuse. Apply the same cross-site request forgery (CSRF) and cross-origin resource sharing (CORS) protections to AI endpoints as you would to any web application. These protections are not always on by default.
Be cautious about AI model version updates and backwards compatibility. A model update may silently change safety behaviours, output formats, or reasoning patterns. Balance usability against the risks a new version introduces. Ideally fix model choice. Test thoroughly on updates and document your decisions.
Build reusable, tested AI safety controls rather than reinventing them. If you build a safety control once, make it reusable and test it thoroughly before applying it widely. Continue testing it over time. When a safety bug is found, update every system that uses it.
Use robust version controls. Keep the prompts, markdown files, infrastructure-as-code, and other code under strict version controls. This enables linting, CI/CD inline testing, and comparing against vendor or third-party model changes. Which means an easier time identifying changes when issues arise, faster rollback or changes.
Make AI system behaviour auditable and explainable. Document prompts, system instructions, and agent decision logic. Add comments explaining safety controls. Readable, auditable AI systems are easier to test, easier to maintain, and faster to diagnose when something goes wrong. Clear documentation helps with safety testing, incident response, and onboarding. Build the behaviour with the future engineers in mind, which is most likely you, but could also be AI.
Use immutable context and state in AI agent workflows wherever possible. Mutable shared state between agent steps creates unpredictable behaviour and increases the risk of manipulation. Design for immutability and explicit state transitions.
Know and comply with all AI-specific regulations that apply to your systems. This includes laws, regulations, and standards such as the EU AI Act, local privacy laws, and sector-specific requirements. Ask your legal and security teams to verify your obligations.
Maintain a current inventory of all AI models, agents, tools, and dependencies. Include where each model is deployed, version, how it is accessed, who owns it, responsible persons, and where its documentation lives. Audit this inventory at least annually.
Adopt an AI safety framework if your organisation does not already use one. Frameworks such as the NIST AI Risk Management Framework, OWASP Top 10 for LLMs, or MITRE ATLAS provide structured guidance. If your organisation has adopted one, follow it.
Decommission old AI models and agents carefully and intentionally. Remove API access, revoke credentials, archive documentation, and update your inventory. Document the decommission process (ideally before it’s in production). Abandoned AI systems with live access are a serious risk.
Plan for AI model updates that can be applied smoothly and safely. If your users or downstream systems depend on your AI’s behaviour, ensure updates can be rolled out, tested, and rolled back without disruption.
Provide a hardening guide for any AI system you deploy that others will operate. If an administrator or end user is responsible for configuring or operating your AI system, give them clear guidance on how to do so securely. Assume the end user won’t read it and act accordingly.
Design AI systems to be easy to integrate with securely. If your AI system forces developers to adopt insecure workarounds to integrate it, the workarounds will become the norm. Secure integration should be the path of least resistance.
Prioritise usability when designing AI safety controls. Controls that are difficult to use will be worked around. Test guardrails for usability just as you would any other feature. Good safety design and good user experience are not opposites.
Verify and document user consent before AI systems process user data. Never assume consent. Always ask. Store consent records in your system of record. This is both an ethical obligation and, in most jurisdictions, a legal requirement.
If you find a reproducible failure mode, safety or security bug in an AI model or framework, report it. Mistakes happen. Hiding or exploiting the mistakes is to be avoided. Be willing to share how your AI system failed and what you learned. Responsible disclosure benefits the entire AI development community. Follow the vendor’s responsible disclosure process.
Treat AI safety warnings and alerts as errors and fix them. A safety warning that is ignored is a vulnerability waiting to be exploited. If your AI framework, monitoring system, or testing tool raises a warning, address it with the same urgency as a compiler error.
If your AI system runs on physical hardware, include physical security. Securing the software layer alone is not sufficient. Physical access to AI infrastructure must also be controlled, monitored, and documented.
Maintain a list of dangerous AI coding patterns to avoid. Include patterns such as direct prompt construction from user input, unrestricted tool access, unvalidated AI output passed to system calls, and disabled safety filters. Check your codebase for these patterns regularly. Automate detection in your linter or tooling where possible.
Follow your organisation’s AI safety guidelines and approved patterns. Use the AI safety frameworks and approved integrations your organisation has established. If you don’t have one, write one. If a business requirement prevents this, work with your business to identify alternatives, document it, and notify the teams responsible. They may require a formal exception.
You can’t govern what you can’t measure. That’s the core problem facing organisations deploying AI today.
Most AI risk conversations stay at the level of vague concern: “we need to manage bias”, “we should think about data privacy”. That’s not governance. It’s wishful thinking. Real AI governance requires specific metrics, clear thresholds, and defined accountability for each category of risk your systems create.
The below framework covers ten AI risk categories that most consistently cause operational failures, regulatory penalties, and reputational damage. For each category, it provides the primary business impact, recommended mitigation strategies, and the specific metrics your teams should be tracking in production.
How to use this framework: Not all ten categories apply equally to every deployment. Start by assessing which risks are most material to your specific AI systems and business context. Use the metrics as a baseline, adapt thresholds to your risk appetite and regulatory environment. Revisit your measurements as your systems evolve, because AI risk changes as models drift, data shifts, and threat actors adapt.
1. Model Inaccuracy
Primary Business Impact: Operational failure, loss of customer trust
An AI model is only as valuable as its predictions are reliable. When a predictive maintenance model misses an impending equipment failure, the result is unplanned downtime. When a customer-facing AI provides incorrect information, the result is frustrated users and eroded trust. Model inaccuracy compounds over time through a process called concept drift, the model’s training data becomes less representative of the real world it’s operating in, and performance quietly degrades until something breaks visibly.
Continuous monitoring of production performance is the foundation. Pair this with human-in-the-loop (HITL) mechanisms that allow expert review of high-stakes decisions. HITL serves a dual purpose: it catches errors before they cause harm, and the correction data it generates feeds back into model improvement.
Measurement Metrics for Model Inaccuracy:
Metric
Description
How to Measure
Drift Rate
The rate at which confident incorrect predictions increase over time.
Monitor the proportion of high-confidence predictions subsequently identified as incorrect through human review or ground truth data.
Standard statistical measures of predictive performance.
Calculate against a held-out test set or ground truth data. Accuracy measures overall correct predictions; Precision measures true positives among all positive predictions; Recall measures true positives among all actual positives; F1-score is the harmonic mean of precision and recall.
Mean Absolute Error (MAE) / Root Mean Square Error (RMSE)
Average magnitude of errors for regression tasks.
Calculate the average absolute difference (MAE) or the square root of the average squared differences (RMSE) between predicted and actual values.
HITL Intervention Rate
The percentage of model outputs requiring human correction or override.
Track the frequency of human interventions in the AI workflow. A high rate signals significant inaccuracy or the need for retraining.
Edge Case Failure Rate
How often the model fails on rare, unusual, or out-of-distribution inputs.
Design specific test cases or monitor real-world performance on identified edge cases. Requires domain expertise to define what constitutes an edge case.
2. Data Privacy (PII)
Primary Business Impact: Regulatory fines, legal liability
AI systems process vast quantities of data, including sensitive personal information. Without rigorous controls, that data can leak into model outputs, training datasets, or logs, exposing individuals and triggering regulatory action. The EU AI Act and GDPR both impose substantial penalties for PII mishandling, and the complexity of AI pipelines makes it easy to lose track of exactly where personal data flows and how it’s used.
Automated PII classifiers identify and categorise sensitive data before it reaches models or outputs. Differential privacy adds carefully calibrated noise to data, preserving aggregate utility while making it mathematically difficult to reconstruct individual records. Together, these approaches reduce exposure without eliminating the data’s analytical value.
Measurement Metrics for Data Privacy (PII):
Metric
Description
How to Measure
PII Leakage Rate
How often sensitive personal information appears inadvertently in AI outputs or logs.
Monitor system outputs, logs, and data flows using automated scanning tools configured to detect PII that should have been protected or anonymised.
Differential Privacy Epsilon
A quantitative measure of privacy loss in a differentially private algorithm. Lower epsilon means stronger privacy guarantees.
Calculate the epsilon value for algorithms employing differential privacy. This requires detailed understanding of the algorithm’s mechanics and the data it processes.
Re-identification Risk Score
The estimated probability that an individual can be uniquely identified from supposedly anonymised data.
Use statistical methods and re-identification tests to assess the likelihood of linking anonymised data back to individuals. K-anonymity and l-diversity assessments can inform this score.
Data Minimisation Ratio
The proportion of sensitive data used relative to total available sensitive data. Lower is better.
Audit data pipelines and storage to quantify PII collected and processed versus the minimum required for the AI system’s function.
3. Shadow AI
Primary Business Impact: Uncontrolled data egress, security gaps
When employees adopt AI tools without IT or security approval, they create risk the organisation can’t see and therefore can’t manage. Sensitive corporate data gets fed into external models. Unapproved third-party services introduce unvetted security vulnerabilities. The absence of oversight can expose the organisation to data breaches, compliance violations, and intellectual property loss, all from tools that nobody officially sanctioned.
Network monitoring and endpoint detection reveal unauthorised AI applications that are already in use. But discovery alone isn’t enough. Procurement policies need teeth: every AI tool adoption should require security, legal, and ethical review before deployment. The goal is centralised visibility without creating so much friction that employees route around the process anyway.
Measurement Metrics for Shadow AI:
Metric
Description
How to Measure
Unsanctioned AI Usage Count
The number of unauthorised AI tools or services detected within the organisation’s network.
Use network monitoring, endpoint detection and response (EDR) solutions, and cloud access security brokers to identify and count unapproved AI applications.
Data Egress Volume to AI Domains
Total data volume transferred from internal systems to external, unapproved AI service endpoints.
Monitor network traffic and data loss prevention (DLP) systems for transfers to unsanctioned AI service domains.
Procurement Bypass Rate
The percentage of AI tool acquisitions occurring outside established procurement processes.
Audit expense reports, software licences, and cloud service usage against approved vendor lists and procurement records.
Discovery Coverage
The percentage of the enterprise network, endpoints, and cloud environments actively monitored for Shadow AI.
Assess the scope and effectiveness of AI discovery tools across the full IT infrastructure.
4. Agentic Risk
Primary Business Impact: Autonomous system failure, goal hijacking
Agentic AI systems – those that plan, execute, and adapt autonomously to achieve goals – operate with a degree of independence that creates new categories of failure. They can make compounding errors without human checkpoints. They can be manipulated through prompt injection into pursuing objectives entirely different from their original purpose. The more autonomous the system, the wider the gap between what it does and what a human would have approved in the moment.
Sandboxing allows safe observation of agent behaviour before deployment in consequential environments. Restricting which tools and resources an agent can access limits the blast radius when things go wrong, and in sufficiently complex systems, things will go wrong. These aren’t permanent constraints; as an agent demonstrates reliable behaviour in controlled conditions, access can expand incrementally.
Measurement Metrics for Agentic Risk:
Metric
Description
How to Measure
Task Success Rate
The percentage of times an AI agent successfully achieves its assigned objective.
Define clear objectives and success criteria. Measure task completion rates in controlled environments and real-world deployments.
Tool Call Accuracy
The precision with which an agent selects and correctly uses external tools or APIs.
Monitor agent logs to assess whether correct tools are invoked with appropriate parameters. Compare against expert-defined optimal tool usage.
Goal Hijacking Rate
How often an agent deviates from its intended objective due to external manipulation or internal misalignment.
Test with adversarial prompts designed to attempt goal hijacking. Track instances where agent behaviour deviates from intended outcomes.
Autonomous Step Count
The number of actions an agent takes without human intervention or oversight.
Log the sequence of agent actions and identify points where human review would typically occur. Higher counts indicate greater autonomy and require proportionally stronger controls.
Reasoning Coherence Score
An assessment of the logical consistency of an agent’s decision-making process.
Use human evaluators or an LLM-as-a-Judge approach to score the agent’s reasoning steps for logical soundness and alignment with expected behaviour.
5. Adversarial Attacks
Primary Business Impact: System compromise, data poisoning
Adversarial attacks are deliberate attempts to exploit AI system vulnerabilities. Subtle perturbations in images can cause misclassification. Carefully crafted prompts can hijack the behaviour of large language models. Poisoned data introduced into training sets can degrade performance or create hidden backdoors. These attacks are designed to look like normal inputs while producing abnormal outcomes.
Robustness testing evaluates models against known adversarial techniques before attackers get the chance. Prompt filtering – through input validation, sanitisation, and anomaly detection – reduces the attack surface by catching malicious inputs before they reach the core model. Neither measure is sufficient alone; layering them together significantly raises the cost of a successful attack.
Measurement Metrics for Adversarial Attacks:
Metric
Description
How to Measure
Attack Success Rate
The percentage of adversarial inputs that successfully produce incorrect or manipulated outputs.
Conduct controlled experiments with adversarial examples and record the rate of misclassification or undesired behaviour.
Robustness Score
A model’s measured resilience to various adversarial attack techniques.
Evaluate model performance under different types and strengths of adversarial perturbations. Higher scores indicate greater resilience.
Filter Bypass Latency
The average time or effort required to craft an adversarial input that bypasses existing defences.
Simulate attack scenarios and measure the time and number of attempts needed for a successful bypass.
Prompt Injection Sensitivity
The threshold at which a model begins to follow malicious instructions over its intended system instructions.
Test models with a range of prompt injection techniques, varying strength and subtlety, and observe adherence to malicious instructions at each level.
6. Bias & Discrimination
Primary Business Impact: Reputational damage, legal action
AI models don’t create bias from nothing. They learn it from training data that reflects existing societal inequities. A hiring algorithm trained on historical decisions inherits the patterns of who was hired historically. A loan approval model reflects the lending patterns of the past. The problem compounds when these biases aren’t surfaced before deployment and the model makes thousands of consequential decisions before anyone notices the pattern.
Anti-discrimination law doesn’t care that the bias came from training data. The legal and reputational exposure is the same as deliberate discrimination.
Mitigation Strategies: Fairness audits, diverse training sets
Regular fairness audits systematically identify and quantify biases in model outputs across demographic groups. Diverse training sets that accurately represent target populations reduce the initial bias the model learns. Techniques including re-sampling, re-weighting, and adversarial debiasing can further reduce bias in models where the training data alone is insufficient.
Measurement Metrics for Bias & Discrimination:
Metric
Description
How to Measure
Disparate Impact Ratio
Compares the selection or outcome rate for a protected group against a majority group. A ratio below 0.8 (the “80% rule”) indicates potential disparate impact.
Calculate the ratio of selection rates (e.g., hiring rate, loan approval rate) for a protected group versus the most favoured group.
Equalized Odds / Demographic Parity
Statistical fairness metrics assessing whether a model performs equally across groups (Equalized Odds) or produces positive outcomes at equal rates across groups (Demographic Parity).
For Equalized Odds, compare true positive and false positive rates across groups. For Demographic Parity, compare the proportion of positive predictions per group.
Bias Amplification Factor
The extent to which a model amplifies biases already present in its training data.
Compare bias observed in model outputs against bias in the input data. A factor greater than 1 indicates amplification.
Stereotype Consistency Score
How often a model generates outputs that align with harmful stereotypes about specific demographic groups.
Design targeted prompts to test for stereotypical associations and measure the rate at which the model reinforces them.
7. Regulatory Fragmentation
Primary Business Impact: Compliance overhead, market exit
The global AI regulatory landscape is accelerating in every direction simultaneously. The EU AI Act imposes obligations based on risk classification. GDPR governs personal data. National AI strategies are emerging across jurisdictions with different requirements and enforcement approaches. An organisation operating internationally may face genuinely conflicting obligations; what one jurisdiction requires, another prohibits or leaves undefined.
This isn’t a problem that resolves itself with time. The organisations that treat regulatory fragmentation as a compliance administration problem will find themselves perpetually reactive and expensive to operate.
Mitigation Strategies: Global policy mapping, NIST AI RMF
Systematic mapping of international AI regulations to internal governance frameworks, before those regulations come into force, is the only sustainable approach. The NIST AI Risk Management Framework provides a common structure that satisfies significant portions of requirements across multiple global regulations, giving organisations a stable foundation to build jurisdiction-specific controls on top of.
Measurement Metrics for Regulatory Fragmentation:
Metric
Description
How to Measure
Compliance Coverage Score
The percentage of relevant global AI regulations for which the organisation has established corresponding internal controls and policies.
Conduct a regulatory mapping exercise comparing internal policies against external requirements, and calculate the proportion of covered regulations.
Audit Readiness Score
The time and resources required to produce comprehensive compliance evidence for a specific AI system or regulation.
Track effort involved in preparing for and undergoing AI-related audits. Lower effort indicates higher readiness and better-maintained documentation.
Policy Update Latency
The average time from enactment of a new AI regulation to the corresponding adjustment of internal policies and controls.
The organisation’s exposure to operational restrictions in specific jurisdictions due to inability to comply with local AI regulations.
Evaluate the stringency of regulations in key operating markets and assess the organisation’s capacity to meet them.
8. IP Infringement
Primary Business Impact: Copyright litigation, loss of IP, loss of reputation
AI models trained on large datasets frequently ingest copyrighted material. Generative AI can produce outputs substantially similar to works those models were trained on. The result is potential copyright infringement at scale, automatically, invisibly, and without any deliberate intent on the part of the organisation deploying the model.
The legal landscape around AI-generated content and training data is still developing, but litigation is already underway in multiple jurisdictions. Organisations cannot afford to wait for case law to settle before managing this risk.
Mitigation Strategies: Data provenance tracking, legal review
Data provenance tracking ensures all training data is properly licenced, attributed, and traceable to its source. Legal review of AI-generated outputs, such as for generative AI applications, identifies potential IP issues before deployment or public release. These controls need to be built into the AI development lifecycle, not applied as an afterthought.
Measurement Metrics for IP Infringement:
Metric
Description
How to Measure
Data Provenance Score
The percentage of training data for which origin, licencing, and usage rights are fully documented and verifiable.
Audit training datasets for complete and accurate records of data sources, licences, and terms of use.
Copyright Similarity Index
A quantitative measure of overlap between AI-generated content and known copyrighted works.
Use computational tools and human review to compare AI outputs against databases of copyrighted material, identifying substantial similarity.
Licence Violation Count
The number of instances where licensing terms for training data or model usage have been breached.
Track and log deviations from licencing agreements, including unauthorised use of data or models.
IP Indemnification Coverage
The percentage of AI outputs or components covered by legal indemnification clauses from third-party AI vendors.
Review vendor contracts and SLAs to assess the extent of IP indemnification provided.
9. Operational Complexity
Primary Business Impact: Integration debt, high maintenance costs, security exposure
AI systems rarely operate in isolation. They connect to data pipelines, monitoring infrastructure, deployment platforms, and downstream business processes. As these systems scale, the connections multiply. Each connection is a potential failure point, and the effort to maintain them compounds over time. Teams that spend the majority of their engineering capacity keeping existing systems running have little left for improvement or innovation.
This isn’t an AI-specific problem, but AI accelerates it. Models need retraining. Data pipelines need updating. Infrastructure needs scaling. Without deliberate architectural choices, complexity grows faster than the capacity to manage it.
Modular architecture allows independent development, deployment, and scaling of AI components, limiting the cascading impact of any single failure. MLOps maturity, applying DevOps and security principles to the full machine learning lifecycle, automates the most labour-intensive parts of model management, from data ingestion through to monitoring and retraining. The investment in MLOps pays back through reduced operational overhead and faster, safer iteration.
Measurement Metrics for Operational Complexity:
Metric
Description
How to Measure
Integration Debt Score
A measure of technical debt from complex, non-standard, or poorly documented integrations between AI system components.
Assess the number of manual integrations, custom scripts, and non-standard APIs in use. Higher scores indicate greater fragility and maintenance burden.
MLOps Maturity Level
An assessment of adherence to MLOps best practices across automation, collaboration, and continuous delivery.
Use established MLOps maturity models (e.g., Google’s MLOps maturity model) to score capabilities in areas including CI/CD for ML, automated testing, and model monitoring.
Maintenance-to-Development Ratio
The proportion of engineering resources allocated to maintaining existing AI systems versus building new capabilities.
Track resource allocation between maintenance tasks (bug fixes, infrastructure updates, model retraining) and new development. A high ratio indicates unsustainable operational complexity.
System Latency / Throughput
The responsiveness of the AI system (latency) and the volume of requests it can handle within a given time (throughput).
Monitor real-time performance metrics in production. High latency or low throughput under normal load indicates bottlenecks worth investigating.
10. Third-Party AI Opacity
Primary Business Impact: Supply chain vulnerability, PII and security exposure, reputation impact
Most organisations don’t build their AI from scratch. They use cloud-based ML platforms, pre-trained models, and AI-powered APIs. Those third-party systems are often proprietary black boxes. The vendor’s internal workings, data sources, and risk profiles are not disclosed, and may not be disclosable. When something goes wrong with a third-party model, the organisation bearing the business consequence may have had no visibility into the risk that caused it, but are ultimatly responsible for it.
This is supply chain risk applied to AI, and it compounds when third-party vendors themselves rely on sub-vendors, open-source components, and external data sources.
Comprehensive vendor risk assessments should require Model Cards, System Cards, and audit reports from suppliers, not as a compliance checkbox, but as genuine evaluation of their AI governance practices. SLA enforcement converts transparency requirements into contractual obligations with consequences, covering performance, security, ethical standards, and incident response. Organisations that exercise their audit rights regularly send a clear signal about the standard they expect.
Measurement Metrics for Third-Party AI Opacity:
Metric
Description
How to Measure
Vendor Transparency Score
A composite score reflecting the completeness and clarity of documentation provided by third-party AI vendors.
Evaluate vendor-provided Model Cards, System Cards, audit reports, and security certifications against a predefined checklist of transparency requirements.
SLA Compliance Rate
How consistently third-party AI providers meet contractual SLAs related to performance, uptime, security, and data handling.
Monitor and track vendor performance against agreed SLAs, including downtime, response times, and adherence to security protocols.
Supply Chain Vulnerability Index
A measure of risks introduced by upstream dependencies in third-party AI services, including sub-vendors, open-source components, and data sources.
Map the supply chain of third-party AI solutions, identifying and assessing the risk profile of each component and sub-vendor.
Audit Rights Exercise Rate
How often the organisation exercises its contractual rights to audit third-party AI vendors.
Track audits conducted on third-party AI providers and findings from those audits. Regular exercise of audit rights is a leading indicator of proactive risk management.
Conclusion
The ten categories in this framework – Model Inaccuracy, Data Privacy (PII), Shadow AI, Agentic Risk, Adversarial Attacks, Bias & Discrimination, Regulatory Fragmentation, IP Infringement, Operational Complexity, and Third-Party AI Opacity – Each has produced documented business failures in organisations that treated AI risk as someone else’s problem, or as a problem for later.
Measurement is where governance becomes real. The metrics here give your teams something concrete to track, report on, and improve over time. They make risk visible, and visible risk can be managed.
Start with the categories most material to your current deployments. Build measurement into your operating rhythm, not as an audit exercise but as a production standard. And when your metrics show something unexpected, treat it as information: a signal that something in your system or environment has changed, and an invitation to understand why before it becomes a crisis.
Most boards sense there is something fundamentally different about AI risk. Few can articulate exactly what it is.
The instinct is to revert to familiar patterns: file AI under software assets, flag data breaches as the primary exposure, and hand the whole thing to the CTO or CISO. This is a strategic error, not because those concerns are wrong, but because they are incomplete in a way that leaves the most dangerous risks entirely unmonitored.
AI is not “better software”. It is a probabilistic system that functions more like a delegated authority than a deterministic tool. And authority, unlike infrastructure, does not fail with an error code.
AI Does Not Fail Like Software but Like Judgement
Software fails visibly. It crashes, it returns errors, it stops working. When a database goes down, someone gets paged. When an AI system fails, the lights stay on. The dashboards stay green. And somewhere in the organisation, quietly and consistently, decisions are being shaped by a model that has drifted from the purpose it was built for.
The following table maps this shift in how risk actually behaves and what it demands from governance:
Traditional Software Assumption
The AI Reality
Governance Implication
Deterministic: If input is A, output is always B.
Probabilistic: Outcomes shift with data drift and model updates.
In deterministic software, a successful pilot implies the logic is sound. In the probabilistic world of AI, a pilot only proves the model worked on that specific data at that specific time. The environment shifts. The data changes. The model’s performance follows, silently.
When boards fail to scope AI initiatives clearly, they often allow mission creep: a model built for internal efficiency is gradually repurposed for customer-facing advice, for hiring decisions, for credit assessments. The organisation inherits a judgement risk it was never prepared to manage, and no one has formally signed up to own.
When AI Fails, It Fails Like Judgement, Not Infrastructure
Consider what actually happens when an AI-driven credit model begins to subtly exclude a demographic. The system is not “down”. No alert fires. The model continues to process applications, return results, and feed reporting dashboards. But a failure is happening and it is just a failure of judgement, not infrastructure.
If a senior adviser gave consistently flawed advice, the board would hold them accountable. But because AI is categorised as technology rather than authority, organisations frequently lack clear ownership for the consequences of AI-driven decisions. The failure is not a bug to be patched. It is a failure of the organisation’s delegated authority and therefore a governance failure.
The most expensive failures will be the quiet ones. They will not appear on any report or dashboard, because the system is not down or showing errors. Meanwhile, the AI is compounding the problem with every decision it makes.
Automation Today Can Bankrupt Talent Tomorrow
There is a strategic risk that rarely reaches the board agenda: the degradation of human capacity for growth.
When AI automates all entry-level analytical work, it inadvertently destroys the training ground for future leaders. Junior staff who rely on AI for judgement calls, without having done the underlying research or developed the foundational understanding, do not build the intuition that senior roles require. The organisation optimises today’s throughput at the cost of tomorrow’s capability.
Boards must ask a harder question than “Is this efficient?” They should ask: By automating today’s tasks, are we systematically preventing the development of the people we will depend on in five years?
Active Governance Requires Different Questions
Bridging this gap is not primarily a technology problem. It is a governance problem and it starts with the questions boards choose to ask.
Refuse the “software” label. AI should not be buried in the IT budget and reviewed on an annual audit cycle. Treat it as a digital employee: one with responsibilities, performance expectations, and accountability for the quality of its decisions.
Ask about drift, not just security. “Is it secure?” is the wrong question for AI risk. The right question is: “How much has the accuracy of its output changed in the last 30 days, and who is responsible for detecting that change?”
Define accountability before something goes wrong. If the AI makes a consequential mistake, does that belong to legal, operations, or strategy? If the answer is unclear, the organisation does not have governance, it has exposure.
The real risk of AI is not that it will suddenly break. It is that it is slowly and silently leading the organisation astray while every dashboard remains green.
The rapid adoption of AI across the Australian financial services sector has moved from competitive advantage to operational necessity. But the speed of adoption has created a problem regulators have noticed: the gap between deploying AI and actually governing it.
For Australian Financial Services licensees, that gap is not a technology problem. It is a compliance problem. ASIC and APRA are increasingly focused on whether firms can demonstrate they understand what their AI tools are doing, why, and what happens when things go wrong. The Design and Distribution Obligations (DDO) and the requirement to act “efficiently, honestly, and fairly” apply to AI-driven decisions the same way they apply to any other business process. The AI does not change the obligation, it just makes it harder to see when the obligation is being breached.
1. The Risk Landscape: What You Probably Haven’t Tested
AI introduces risk profiles that are genuinely different from traditional software, not because the regulatory obligations are different, but because the failure modes are less visible and often more systematic. Three characteristics make AI uniquely difficult to govern: it can be confidently wrong, its errors can be invisible in aggregate data, and its behaviour can change without anyone deliberately changing it.
When the AI is wrong, it doesn’t know it
Traditional software fails visibly; an error code, a system crash, a blank screen. AI models fail silently, generating outputs that are plausible, formatted correctly, and entirely false. In financial services, this matters immediately.
Consider an AI chatbot deployed to answer client questions about a managed fund. A client asks whether the fund has capital protection. The AI, drawing on training data that includes older product literature, confirms that it does. The fund’s current Product Disclosure Statement says otherwise. The client invests based on the AI’s answer. No one in your firm knows this happened, because the chatbot logged the interaction as a resolved query, not a complaint.
This is a predictable consequence of deploying a generative AI tool without grounding it against your current, authoritative product documentation and without a mechanism to detect when a client has acted on AI-generated information that contradicts your PDS. Under DDO, distribution based on false product information is a breach, regardless of whether a human or an AI provided it.
Other failure modes in this category that firms often underestimate:
AI-generated PDS summaries distributed to advisers: If your AI summarises a PDS to help advisers understand a product, and that summary contains a material error, every adviser who relied on it has potentially distributed outside the TMD. The summary feels authoritative. No one fact-checks it against the original.
Client mis-categorisation at scale: An AI categorising clients by risk profile or financial sophistication can be wrong about individual clients in ways that are invisible in aggregate accuracy figures. If the tool is 94% accurate overall but systematically worse for clients over 65, a cohort you’ve never tested separately, you may have a bias problem embedded in every interaction with that demographic.
Autonomous AI agent conduct: An AI agent authorised to take actions on behalf of clients, scheduling, executing instructions, communicating, can engage in conduct that would constitute misconduct if a human adviser did it. The firm bears that liability.
The complaint that never becomes a complaint
One of the more consequential blind spots in AI-assisted customer service is the complaint that gets resolved before it is recorded.
When a human customer service representative handles a complaint, there is generally a process: the interaction is flagged, the complaint is logged, it enters the complaints management system. When an AI handles the same interaction, the outcome depends entirely on how the AI has been configured to classify what it receives.
A client who says “I’m really unhappy with how this was handled and I want something done about it” may receive a sympathetic, helpful AI response that resolves the surface issue. The AI logs the interaction as a successful resolution. In your complaints data, it does not appear. In your DDO reporting, it does not appear. In your significant dealings analysis, it does not appear.
If this happens at scale, because AI tends to apply consistent classification errors consistently, your regulatory reporting may systematically understate the number and nature of client complaints. Section 994F(4) requires accurate complaints reporting. An AI that is resolving complaints before they can be counted is not a compliance solution, but a compliance risk disguised as good customer service.
Model drift: the AI that changed without anyone changing it
AI models are not static. Their performance is anchored to the data they were trained on, and when the world diverges from that data, their outputs become less reliable. Gradually, quietly, and without any visible system failure.
Consider a creditworthiness or suitability assessment tool trained on client data from 2020 to 2022. That tool learned patterns from a period of historically low interest rates, rising asset values, and a particular economic environment. If it has not been re-validated against current conditions, it may be applying assumptions that no longer hold. Clients who would have been correctly categorised as suitable for a particular product four years ago may no longer be. The tool does not know this. It continues to categorise them as suitable.
This is model drift. It is the AI equivalent of a policy that nobody has reviewed in years. The difference is that when a policy drifts out of date, someone usually notices. When an AI model drifts, the outputs often look the same, the categories still get filled, the reports still get generated, and the process still runs. The problem is invisible until something goes wrong at scale.
2. Compliance and the DDO Framework
The DDO framework, as outlined in ASIC Information Sheet 264, does not make exceptions for AI. Every obligation that applies to a human process applies equally when that process is automated. The relevant question is not whether AI was involved, but whether the outcome met the regulatory standard.
AI in product design
When an issuer uses AI to design a financial product or determine its Target Market Determination, the AI must correctly account for the likely objectives, financial situation, and needs of the target class. ASIC’s framing is clear: the target market is the class of consumers for whom the product is likely to be appropriate, and the TMD must accurately describe them.
If the AI that helps determine your TMD has been trained on data that overrepresents a particular client profile, for example urban, higher income, digitally engaged, it may systematically underweight the needs of clients outside that profile. The TMD looks complete. But it may not accurately describe the full range of consumers for whom the product is or is not appropriate.
AI in distribution and monitoring
Distributors are increasingly using AI to monitor distribution conduct and identify significant dealings inconsistent with the TMD. This is exactly the kind of application DDO contemplates. But the reliance on AI for this function creates its own obligations.
The AI must be capable of correctly applying the definition of a “significant dealing”. It must detect when the proportion of consumers outside the target market reaches a threshold that triggers reporting. It must not classify borderline cases consistently in one direction. And it must ensure that the “reasonable steps” required under DDO are documented, not just assumed.
Automation that speeds up distribution is good, but without preserving the compliance steps is not a solution. It is a breach that scales.
The “efficiently, honestly, and fairly” standard
Section 912A of the Corporations Act requires licensees to provide services efficiently, honestly, and fairly. ASIC has indicated that a lack of explainability in AI decisions may fail the fairness test. If an AI makes a recommendation or assessment, and the firm cannot explain why, that opacity is itself a compliance risk.
This is particularly relevant for advice tools. An AI that generates recommendations not tailored to the individual client’s circumstances, because it is applying a generalised model rather than genuinely assessing the specific client, may fail the best interests duty and suitability requirements, even if the recommendation appears reasonable on its face, or only used internally.
3. Governance Checklist: Questions Worth Asking
The questions below are not designed to confirm that your governance is in order. They are designed to find the places where it isn’t. If you find yourself answering a question with “I assume so” or “someone else handles that,” you have found something worth investigating.
Accountability
If ASIC asked you today which AI tools are making or materially influencing client-facing decisions, could you hand them a complete list within an hour?
Name the person who would receive the call if one of your AI tools caused a compliance breach tomorrow. Does that person know they are accountable for it?
Is there a human being who has read the AI vendor’s model documentation, not just the sales materials, and signed off on its fitness for your specific use case?
Distribution and TMD integrity
If your AI is involved in the distribution process, can you demonstrate that it has never bypassed a “reasonable step”, or do you assume the vendor built those in?
Have you tested whether your AI correctly identifies a “significant dealing”, not in documentation, but in a live scenario with borderline cases?
When did you last audit the AI’s categorisation outputs against your TMD to check for systematic misalignment?
Complaints and reporting
Could your AI chatbot resolve a client complaint without it ever appearing in your complaints data? If you don’t know, that is your answer.
Does the data flowing from your AI customer service tools into your DDO reporting get validated by someone who did not build the tool?
Have you cross-checked your AI-generated complaints figures against any other data source, e.g. inbound call volumes, churn patterns, adviser escalations, to check they are plausible?
Model integrity and drift
What economic conditions was your suitability or creditworthiness AI trained on? Have you re-validated it since those conditions changed materially?
Is there a scheduled review process for your AI models, or does re-validation happen only when someone notices a problem?
If a vendor updated their model overnight, would you know? Would anything in your process change?
Transparency and explainability
For each AI tool used in a client-facing decision, can you produce a reason code or plain-language explanation for a specific output? Or does “the model decided” end the conversation?
Have you tested whether your AI-generated PDS summaries are accurate against current PDS documents, not when the tool was deployed, but recently?
If a client asked why the AI categorised them the way it did, could anyone in your firm answer that question?
Bias and fairness
Have you tested your AI’s accuracy across different client demographics, by age, income bracket, geography, or product experience, rather than only in aggregate?
If your AI performs significantly worse for a particular client cohort, would your monitoring processes detect it? What would trigger the alert?
Incident response
Does your organisation have a kill switch for each AI tool. A documented, tested mechanism to halt its operation. Or does the plan exist only in theory?
When did you last test the kill switch? Triggering it to ensure it works as expected?
If an AI tool needed to be turned off today, what would happen to the clients and processes that depend on it? Has anyone mapped that?
Vendor and third-party risk
Does your AI vendor notify you before making changes to their model? What is your contractual right to know when the model you approved is no longer the model you’re running?
Has your vendor been assessed for compliance with the Privacy Act? Have you verified that client data used to train or improve their model has not been retained in ways you did not authorise?
Are there mechanisms in place to prevent data poisoning, deliberate or accidental corruption of the inputs your AI relies on?
Conclusion
AI offers real opperational advances, but brings with it governance risks not fully covered by your existing controls. Use the questions offered here as a starting point, not a definitive checklist. Firms that haven’t built the practice of thinking through their AI controls before they need it will be caught short.