The unit of safety is the working system, not the model
A recurring mistake in AI product programs is validating the model while deploying a system. The latest model can perform well on a benchmark or within a pilot and still be unsafe, ineffective, or operationally unacceptable when embedded in a real workflow. The deployed object is not just a language model. It is a socio-technical system composed of a model, prompts, memory, retrieval, tools, permissions, a user interface, human operators, business rules, escalation paths, and the surrounding organisation.
This distinction becomes more important when the system is an agent rather than a text generator. An agent does not only propose an answer. It may interpret a goal, choose a sequence of steps, call tools, update records, send messages, change state, and create consequences outside the model boundary. The useful question to ask: under realistic operating conditions, does the complete system reliably support the intended human and business outcome, and does it tend to fail in bounded and recoverable ways?
A useful parallel comes from airplane cockpit and fly-by-wire design. Aviation does not treat the flight-control computer as an isolated artefact. The aircraft is designed around the interaction between automation, pilots, displays, procedures, training, authority boundaries, and abnormal situations. The Federal Aviation Administration describes human factors as a multidisciplinary effort that applies knowledge of human capabilities and limitations to systems, procedures, training, staffing, and environments, not just to a component in isolation. A recurring design observation from fly-by-wire experience is that an automation mode that works from an engineer’s perspective can obstruct pilots when it does not match their task model, expectations, or ways of working.
Drawing on decades of critical design choices from aviation, end-use validation benefits from evaluating the human-machine operating system, including what happens when assumptions break. For AI, this suggests moving away from a one-time benchmark mindset toward a continuing assurance discipline, though what that discipline looks like in practice is still being worked out.
What end-use validation might need to cover
End-use validation is evidence that a system is fit for a defined operational context. It is broader than model evaluation and narrower than a vague aspiration to “trustworthy AI”. The core question is whether the system meets requirements for capability, reliability, safety, security, usability, and recoverability in the workflow where it will be used.
One starting point is defining the unit of evaluation. A useful specification tends to cover at least five elements:
| Element | Validation question |
|---|---|
| User and role | Who relies on the system, with what expertise, incentives, workload, and authority? |
| Task and outcome | What real-world result is the system meant to produce, not merely what text should it generate? |
| Operating envelope | For which inputs, data conditions, tools, time pressures, and failure states is it approved? |
| Authority boundary | What may the agent read, recommend, prepare, execute, or never do? |
| Failure policy | How should the system abstain, escalate, roll back, or enter a safe state? |
This framing helps avoid a common category error: treating a response-quality score as a proxy for business safety. A customer-support agent might produce fluent summaries while exposing private information. A procurement agent might select plausible suppliers while exceeding budget authority. A software agent might write valid code while deleting the wrong production resource. Impressive-looking text can coexist with an unacceptable end use.
NIST’s AI Risk Management Framework treats trustworthiness as something to incorporate from design through evaluation, organising work around governing, mapping, measuring, and managing risk . The generative-AI profile further notes that risks can arise at any stage of the lifecycle, and that the relevant risk may exist at the model, application, use-case, or ecosystem level 4. For agents, those stage and system-level distinctions are worth examining during validation planning, though how organisations apply them will vary considerably depending on context.
Automation tends to work better when designed around the operator
A cockpit is not a screen attached to an aircraft. It is an interface for maintaining a pilot’s situational awareness, selecting modes, understanding system state, detecting deviations, and intervening under pressure. Its design addresses questions such as: What is the aircraft doing now? What will it do next? Why is it doing that? What authority does the automation have? How can the pilot change or override it? What happens if a sensor, subsystem, or assumption fails?
An AI agent may benefit from an equivalent operational interface. Depending on the use case, that interface might be a case-management screen, an approval queue, a developer console, an operations dashboard, or an API contract. The aim is to make the agent’s state and authority legible, rather than presenting only a polished final answer.
The cockpit comparison suggests several design considerations, though the right combination will depend on the specific context:
- Mode awareness. Users generally benefit from knowing whether the agent is drafting, planning, recommending, simulating, awaiting approval, or executing. Hidden transitions between these modes tend to create problems.
- Action visibility. Showing the planned action, the tool invoked, the data accessed, the parameters used, and the resulting state change gives users a basis for meaningful oversight.
- Predictable authority. Permissions that are explicit, bounded, and consistent with the role and task tend to be easier to govern than permissions inferred from the model’s confidence.
- Effective override. A human override needs to be timely, technically real, and understandable. A button that appears to cancel an action after an irreversible tool call has completed is not a meaningful override.
- Abnormal-procedure support. An interface that helps users handle uncertainty, partial failure, conflicting instructions, and degraded service is more likely to produce good outcomes than one that only supports the happy path.
- Training for system behaviour. Users who understand not only the happy path but also the agent’s failure modes, escalation cues, and recovery procedures tend to be more effective supervisors.
That last point extends beyond the software interface. Aviation human-factors work considers reliance, over-reliance, skill degradation, situational awareness, failure recovery, and the definition of roles and responsibilities. If users become passive approvers, if the agent’s explanations are too verbose to inspect, or if escalations arrive without enough context, the human remains nominally responsible but may no longer be operationally effective. The field does not yet have consensus on how to prevent this; it is an active area of both research and practice.
Evaluating a non-deterministic system before release
There is no single score or benchmark that can guarantee success from a non-deterministic system. A more tractable approach is constructing a layered evidence case in which nondeterminism is measured, bounded, and connected to consequences.
A task-based test set drawn from real work tends to be more informative than synthetic or generic benchmarks. Such a set usually covers ordinary cases, edge cases, ambiguous requests, adversarial inputs, incomplete data, conflicting records, tool failures, and cases where the correct behaviour is to ask for clarification or decline. Each test benefits from specifying both acceptable and prohibited outcomes. For an agent, the test may need to cover not only the final response but also the plan, tool selection, arguments, sequence of actions, permissions used, and resulting side effects.
Repeated trials capture variation. The objective is not to pretend the system is deterministic but to estimate the distribution of behaviour. Recording the rate of successful completion, unsafe action, unnecessary escalation, unsupported claim, policy violation, and recovery failure gives a more complete picture than a pass rate alone. In high-consequence workflows, a low average error rate can conceal unacceptable tail risk, so release criteria that include tolerances for severe failures, not only average quality, are more protective.
Automated checks, deterministic business rules, expert review, and end-to-end simulation each contribute different things. Automated graders are useful at scale, but they are not a sufficient authority for consequential tasks. Expert review helps judge whether an action is materially safe and useful in context. Simulation helps surface how users, tools, timing, and partial failures can change behaviour in ways that isolated prompt testing will not reveal.
A practical set of release tests might look like the below, although the specific conditions will vary by organisation and risk tolerance:
| Evidence layer | What it tests | Example release condition |
|---|---|---|
| Model and prompt tests | Basic capability, refusal behaviour, instruction following | No critical policy regression; known failure classes documented |
| Workflow tests | End-to-end completion using realistic data and tools | Required task success rate and bounded escalation rate |
| Adversarial tests | Prompt injection, data exfiltration, privilege misuse, malicious content | No unmitigated critical path to unauthorised action |
| Human-factors tests | Comprehension, workload, trust calibration, override success | Users can detect, explain, and stop unsafe behaviour within the required time |
| Change-impact tests | Effect of model, prompt, tool, data, or policy changes | No material regression against protected scenarios |
| Operational rehearsal | Monitoring, rollback, incident and crisis response, degraded modes | Team can detect, contain, investigate, communicate, and recover from a simulated incident |
The underlying principle is to validate the decision and action path, not only the outputs. For a read-only assistant, this may mean evaluating factuality, relevance, privacy, and escalation. For an action-taking agent, it extends to authorisation, transaction correctness, idempotency, reversibility, and impact if things go wrong.
Model changes tend to behave like changes to a safety-relevant subsystem
A model update is probably better treated like a change to a safety-relevant subsystem than like a routine content refresh. Versioned configuration and reproducible evaluation matter here. At minimum, retaining the model identifier, system prompt, tool definitions, retrieval configuration, policy version, data snapshot or references, decoding settings, and software dependencies for every evaluated and deployed release gives teams something to reason from when things change.
Dividing the evaluation suite into two parts works well for most teams. The first is a stable regression suite covering high-value, high-risk, and previously failed scenarios. The second is a living suite built from production incidents, expert input, user corrections, near misses, newly discovered attacks, and shifts in the operating environment. The stable suite protects known requirements; the living suite keeps the system connected to what is actually happening.
What to test when a change occurs depends on what changed. A model update may affect refusal behaviour, tool selection, verbosity, latency, or instruction hierarchy. A retrieval change may affect data provenance and privacy. A tool change may alter the action surface even if the model remains fixed. Dependency-aware testing tends to catch more than model-centric testing alone.
Staged exposure reduces risk during release. A shadow mode allows the new version to observe and compare live-like traffic without taking action. A canary release limits exposure to a controlled population or lower-risk workflow to measure how it performs in production. Progressive rollout can expand when outcome, safety, and human-override metrics remain within acceptable bounds. Automatic rollback becomes more valuable when a severe invariant is violated, such as an unauthorised action or a surge in failed transactions.
The underlying shift in thinking is to not expect identical wording but define properties that the organisation has decided must remain true. For example, the agent must not exceed a role’s authority, must cite or expose required source information, must request approval for defined actions, must preserve required records, and must fail safely when a critical dependency is unavailable. What those properties are is a judgement call, and the judgements will evolve as the field learns more.
Drift detection is easier to name than to operationalise
Aviation monitors flight parameters continuously, not just at certification, because they appreciate drift is not one phenomenon. Input drift changes what the system receives. Data drift changes the meaning or quality of retrieved information. Concept drift changes the relationship between signals and correct decisions. Behaviour drift changes how the model or agent responds. Workflow drift changes the surrounding process, policies, or user behaviour.
Useful monitoring tends to combine statistical signals with outcome and control signals. Statistical monitoring can identify changes in input distributions, missingness, language, document types, tool errors, latency, or refusal rates. Outcome monitoring asks whether recommendations, classifications, or actions remain correct and useful. Control monitoring asks whether the agent continues to follow authorisation, approval, logging, privacy, and escalation requirements.
The most informative drift indicators are often leading indicators tied to the workflow. An increase in human edits to generated decisions, repeated user re-prompts, rising override frequency, growing disagreement between the agent and qualified reviewers, more frequent fallback paths, or a change in the distribution of tools invoked can all indicate that something is shifting. In an action-taking system, monitoring attempted and blocked actions, not only completed actions, may reveal emerging pressure against the control boundary before a harmful outcome occurs.
Defining thresholds, owners, and responses in advance is more useful than responding to each signal ad hoc:
| Signal | Possible interpretation | Initial response |
|---|---|---|
| Input distribution shift | New customer segment, policy, attack, or upstream data issue | Sample and review affected cases; check data pipeline |
| Increase in abstention or escalation | Capability gap, ambiguity, dependency failure, or policy change | Triage by workflow and compare with recent changes |
| Increase in human overrides | Quality degradation or poor trust calibration | Review override reasons and high-severity cases |
| Tool-call pattern change | Prompt/model behaviour change or workflow shift | Freeze rollout; inspect plans and permissions |
| Unauthorised or near-unauthorised attempt | Security or instruction-hierarchy failure | Contain access, preserve evidence, initiate incident response |
| Outcome deterioration | Concept drift, bad data, or process change | Reduce autonomy; revert or revalidate before expansion |
A monitoring dashboard that shows drift but leaves the agent fully empowered is not a control. Drift detection needs to connect to a response such as autonomy reduction, tool disabling, routing to human review, rollback to a known-good version, or temporary suspension of the affected workflow.
Post-deployment monitoring for systems that take actions
Cockpits record flight data precisely so incidents can be reconstructed and Post-deployment monitoring tends to be more useful when it observes across four layers: intent, execution, consequence, and recovery.
Intent monitoring records what the agent believed it was trying to do: the interpreted request, plan, selected policy, uncertainty signals where meaningful, and proposed tool calls. Execution monitoring records what actually happened: tool use, arguments, authorisation context, response, retries, timing, and state changes. Consequence monitoring examines the business result: whether the transaction was correct, whether a customer or employee was affected, whether downstream records remain consistent, and whether the action was reversible. Recovery monitoring verifies that approvals, cancellations, compensating actions, and incident procedures work when they are needed.
This requires an audit trail detailed enough for investigation without indiscriminately storing sensitive data. Logs benefit from access control, tamper-evidence where appropriate, retention limits, and correlation across the agent, tools, identity system, and business application. A team that can answer the following questions after an incident is in a much better position than one that cannot: What request initiated this action? Which version acted? What data did it use? Which policies and approvals applied? Which tools were called? What changed? Who or what stopped it?
For high-impact actions, independent controls outside the model may provide more reliable protection than relying on the agent to correctly describe its own behaviour. Transaction limits, allowlists, dual approval, separation of duties, rate limits, duplicate detection, schema validation, and post-action reconciliation are options that different organisations have used to varying effect, depending on their specific context and risk profile.
Monitoring also benefits from an incident taxonomy. A hallucinated sentence and an unauthorised money transfer are not the same class of event. Severity that reflects the affected asset, reversibility, exposure, recurrence potential, and whether a control failed tends to produce more useful records than a flat log. Near misses deserve attention because they reveal how the system behaves at the edge of its operating envelope, before a harmful outcome occurs.
Allocating human authority by risk, not by presence
“Human in the loop” is too imprecise to function as a design requirement. A more practical framing is to allocate human authority by risk, reversibility, uncertainty, and contestability.
Actions that are high impact, difficult to reverse, legally or ethically sensitive, outside established policy, or based on ambiguous identity or intent are candidates for human approval. Human intervention also tends to be valuable when the agent encounters a novel situation, conflicting evidence, degraded tools, insufficient data, or a security signal. Conversely, requiring a human to approve every low-risk, reversible action can create approval fatigue and encourage rubber-stamping, which may be worse than no oversight at all.
The intervention point needs to be placed before the irreversible boundary. Showing a person a summary after an email has been sent or a payment has been initiated is observation, not meaningful oversight. A reviewer needs enough context to make a real decision: the request, relevant evidence, the proposed action, risk flags, affected parties, alternatives, and the consequences of approval. An interface that makes approval an active decision rather than a default continuation is more likely to produce genuine oversight.
Human oversight also has a staffing dimension. The organisation needs to define who is accountable, what expertise is required, how quickly intervention must occur, what happens outside business hours, and how the reviewer’s decision is recorded. Training that includes realistic failures and degraded modes tends to produce better outcomes than training that covers only normal operation. Discussion of highly automated flight systems makes a related point that operators benefit from understanding how the automation behaves in contingencies so they can anticipate and intervene when necessary.
A useful way to think about autonomy levels is as a hierarchy that can be dialed down as well as up:
| Level | Autonomy | Appropriate use |
|---|---|---|
| Orchestrate | Sequence multiple actions and adapt | Only with mature monitoring, rollback, and incident response |
| Execute within bounds | Act automatically under strict limits | Low-risk, reversible, well-understood tasks |
| Prepare | Draft transactions or changes for approval | Repetitive, reviewable work |
| Recommend | Propose a decision or plan | Human remains decision-maker |
| Observe | Read and summarise; no external action | Early validation and sensitive workflows |
The appropriate level is not a permanent property of the technology. It can move down when drift, incidents, staffing changes, or external conditions increase risk, and may move up as evidence accumulates and the organisation gains operating experience.
Security controls for agents with access to tools, data, and internal systems
Agent security needs to cover more than the model endpoint. The agent’s identity, instructions, memory, retrieval sources, tools, credentials, data flows, and resulting actions all represent parts of the attack surface. OWASP’s Agentic AI guidance treats agentic systems as a distinct security concern because the combination of generative models and autonomy expands both capability and risk.
Least privilege by design is a frequently recommended starting point: giving the agent only the tools and data required for the task, with permissions scoped to the user, case, tenant, environment, and transaction. Short-lived, narrowly scoped credentials and mediated tool calls are generally preferable to unrestricted access. Separating read, draft, approve, and execute permissions reduces the consequence of any single failure.
Instruction and data separation addresses a different risk. Retrieved documents, emails, web pages, and tool responses are untrusted data, not authoritative instructions. This is the typical prompt injection attack path. A system that can’t distinguish between system policy, developer constraints, user intent, and external content is exposed. The systems should sanitise or isolate content where appropriate, to make it harder to manipulate through prompt injection.
Action mediation provides an additional layer. Every consequential tool call passing through a policy enforcement point that validates identity, purpose, parameters, target, limits, approval state, and current context gives the organisation a place to catch problems before they take effect. A mediator that rejects malformed, out-of-scope, duplicate, or high-risk actions adds a control that does not depend on the model’s judgement. Where feasible, idempotent, reversible, rate-limited, and transaction-reconciled actions limit the damage from any single failure.
Containment and recovery planning reflects the assumption that the agent will sometimes be manipulated, confused, or wrong. Sandboxed execution, network egress controls, environment separation, secret isolation, quotas, circuit breakers, kill switches, and rollback or compensating-action mechanisms can all reduce what one failure is able to affect. The security architecture benefits from asking: if the agent makes a serious mistake, how much of the system is exposed?
Collecting evidence matters for development, containment, learning, and recovery. Recording the identity and version of the agent, the relevant inputs and retrieved sources, the tool calls, authorisation decisions, approvals, and outcomes makes incidents reconstructible. Many failures only become visible in hindsight, so an incident that cannot be reconstructed is much harder to correct or audit reliably.
A release-to-operations loop, not a gate
Organisations that treat validation as a continuing loop rather than a one-time gate tend to maintain better awareness as their systems evolve. The loop begins with a use-case definition and risk assessment, continues through scenario design and simulation, and extends into deployment monitoring, incident learning, and revalidation.
One way to describe that operating model:
Specify → Simulate → Constrain → Release gradually → Observe → Intervene → Learn → Revalidate.
Each transition provides evidence. Specification produces the operating envelope and behavioural contract. Simulation produces test results across normal, edge, adversarial, and failure scenarios. Constraint design produces the authority map and independent controls. Gradual release produces comparative evidence against a baseline. Observation produces telemetry about behaviour and outcomes. Intervention produces incident and near-miss records. Learning updates the test suite, controls, training, and workflow. Revalidation asks whether the next version remains fit for use.
This framing changes what “launch” means. Rather than the point at which responsibility transfers from engineering to operations, launch can be understood as a progression through which a monitored system enters a controlled operating regime with known limits, named owners, and an explicit plan for what happens when evidence changes.
Design for competent intervention over imagined perfection
The central lesson from airplane cockpit design is not that humans need to manually supervise every automated step. It is that automation tends to work better when designed around real tasks, real operators, explicit authority, visible system state, and credible recovery. The aim is not to eliminate automation but to make automation dependable within a defined scope and manageable outside it.
For AI agents, end-use validation is a form of systems engineering. It needs to cover the model, and also the workflow, tools, permissions, data, interface, human role, organisational response, and failure modes. It benefits from accounting for nondeterminism without surrendering to it, treating model changes as behaviour changes, detecting drift through both statistical and operational evidence, monitoring consequences rather than merely outputs, placing human intervention before irreversible boundaries, and enforcing security independently of the model’s own judgement.
The field is still developing the methods and experience to do all of this well. What is clearer is the direction: an agent is better positioned for release when the organisation can say more than “we have impressive examples”. It is better positioned when it can say: here is what the system is allowed to do; here is how we tested it; here is how we know when conditions change; here is where a human takes control; here is how we contain failure; and here is the evidence that the complete system remains fit for its intended use. Whether any specific version of that answer is the right one is a question the industry will keep revising as it accumulates experience.