A chatbot answer is designed to be certain. An explanation or disclaimer after the event will not help. This article argues that explainability should work like a gauge on a dashboard, which shows the system’s condition while you use it. A live signal tells you when an answer deserves trust, when it does not, and what you can do next.
Explanations that arrive after a decision cannot protect the person who relied on it
Most discussion of AI explainability starts after the system has decided. People ask why it rejected an applicant, why it suggested a diagnosis, or why it flagged a transaction. These questions matter for audit, review, and accountability but come too late for a person who has already accepted the answer, acted on it, or been harmed by it.
A loan officer who sees a clear explanation after an automated denial cannot change how the officer used the system at the moment of the decision. A clinician who reads a convincing rationale for an AI recommendation may still be misled if the system does not say that a key test result was missing.
The central risk is false certainty. Many AI tools present every answer with the same polish, whether it rests on strong evidence or on weak, incomplete or unfamiliar input. When a system hides its uncertainty, it trains users to mistake fluency for reliability. A system may be able to detect some of its limits, but that ability protects nobody if the system does not show it.
| Retrospective explainability | Operational explainability |
|---|---|
| Examines an output after the system produces it. | Shows the system’s operating state while people use the output. |
| Supports audit, review, and appeal. | Supports calibrated trust, intervention and escalation. |
| Focuses on the factors that influenced a result. | Includes uncertainty, evidence coverage, degraded modes, and confidence limits. |
| Is often technical and written for analysts. | Must be clear and actionable for the user at the moment of use. |
| Can become a persuasive story about a completed decision. | Adds friction before users put too much weight on a decision. |
Performance measured in development may not hold where the system is used
A model can perform well where its developer tested it and poorly somewhere else. Users rarely see that difference unless the system tells them.
The Epic Sepsis Model is a widely used tool that warns hospital staff about patients who may develop sepsis. In 2021, researchers at the University of Michigan tested it on 38,455 hospital stays. The model missed about two-thirds of the patients who developed sepsis. It also sent alerts for 18% of all hospitalised patients, and its accuracy was substantially lower than its developer had reported. Epic disputed the study and said that hospitals need to tune the model before use.
Whatever the cause, a clinician looking at an alert had no live signal that its reliability in that hospital might differ from the published figures. For the individual clinician, that gap may mean misplaced trust in one alert. For the organisation, it may mean thousands of alerts that staff learn to ignore, and a tool that costs attention without returning safety.
Fluent answers can hide changes in a system’s condition
AI systems do not run in one stable condition. Data sources fail, underlying models change, tools become unavailable, context runs out, and inputs arrive that are unclear, inconsistent or unlike anything the system was designed for. The system may then choose between competing instructions, apply a safety constraint, or give a partial answer because it cannot check a claim.
These are normal conditions for a deployed system. The danger is that users often cannot see when they occur.
Consider a hypothetical customer support assistant that loses its connection to the order management system. If it continues to speak with the same confidence, customers cannot tell a verified order status from a general guess. A research assistant that could not open a source, or received contradictory documents, presents the same risk. A polished summary that omits those limits is not made transparent by listing its sources later.
A real case shows what this costs. In 2025, Deloitte Australia delivered a report worth about A$440,000 to the Department of Employment and Workplace Relations that included references to sources that did not exist and a quote attributed to a court judgement that the court had not made. Deloitte revised the report, disclosed that a generative AI model had helped draft it, and refunded the final payment. The report still reached the client looking finished and authoritative, with no sign of which claims a person had checked.
A system earns trust when it stays readable under pressure. It should tell you when it is unsure, when its evidence is thin, and when it has changed how it operates. It should also tell you when it has declined or narrowed part of a request to follow a higher-priority rule. A system that does this is not a failure, because an account of its own condition forms part of the response itself.
Users need four kinds of live signal
A live explainability layer does not need to expose the model’s private reasoning or flood users with technical detail. Raw internal traces can confuse users, create security risks, and give false comfort. The goal is to give a reasonable user what they need to judge whether an output fits its intended use.
At a minimum, users should be able to see four kinds of status.
| Operational state | What the system should communicate | Useful user action |
|---|---|---|
| Uncertainty about the conclusion | The answer is tentative, more than one interpretation is plausible, or the system cannot reliably separate the alternatives. | Treat the response as a starting point, and ask for clarification, a second source or human review. |
| Incomplete evidence or context | Important records, sources or details were missing, out of date, contradictory or unchecked. | Supply the missing material, or avoid a consequential decision based on the response alone. |
| Reduced operating capacity | A required tool, data source, calculation or quality check was unavailable or did not finish. | Delay the task, run it again when the capability returns, or use another process. |
| Instruction or policy constraint | The system could not follow part of the request and has offered a limited alternative, so the answer may be less direct or less complete. | Understand the limit, and choose a safe permitted alternative or a human process. |
Word choice matters because a label such as “low confidence” is often too vague. It does not say whether the system lacks evidence, found conflicting information, met an unclear question, or is working outside its usual tasks. A useful signal names the kind of limit, its likely effect, and the next sensible action.
For example, a legal information assistant should do more than show a small uncertainty icon. It could say: “This response uses general information and does not account for your state or your full agreement. Check local law and get professional advice before you file or sign anything.” A diagnostic support tool could show a probability estimate and also say whether a key test result is missing or the image quality was too low. It could also say whether the patient differs from the population used to test the model.
The design principle is to make uncertainty specific enough to change what the user does next, but without overloading or creating alert fatigue for them to learn to ignore it.
A single confidence score hides the reason for doubt
A numerical confidence score is attractive because it looks precise, comparable, and easy to display. A single number often combines several different questions into one unclear signal. A model can be statistically confident while it relies on unsuitable input, misses key context, or works in a setting where its calibration no longer holds. In the other direction, a system may be unsure between two harmless phrasings and still be fully able to complete the task.
While operational health uses confidence for internal evaluation, it needs a richer vocabulary than “confidence: 87%” presented to the user. A system could report confidence separately for each question that matters: Is the answer sound? Is the input familiar? Is the evidence complete and current? Did the required checks succeed? The confidence score itself also needs validation to ensure it’s not miscalibrated.
The amount of disclosure should rise with the cost of being wrong
Live explainability does not have to mean clutter. A low-stakes writing assistant might show a short note that it guessed some missing context. A system that advises on medical care, welfare eligibility, safety, credit or employment should show prominent, plain-language notices. It should also require an explicit review step when uncertainty crosses a defined threshold.
Friction should increase with the potential cost of an error. This proportion turns explainability from an optional note into a safety control.
A system can disclose the effect of a constraint without exposing how the constraint works
Some people will object that full disclosure of internal state could expose confidential instructions, private data, proprietary methods or security controls. The objection is valid, but it does not alter the argument, because operational transparency does not require full internal disclosure.
A system can describe the effect of a constraint and keep the mechanism private. It can say, “I cannot complete that action as requested, so I have offered a safer alternative”, without quoting restricted policy text. It can say, “I could not verify this claim against the source you asked for”, without revealing credentials, hidden prompts or security design. It can say, “This conclusion is tentative because the records do not include the latest laboratory result”, without exposing unrelated private information.
The test is to disclose anything that materially changes the user’s ability to judge and safely use the output. Security and privacy can justify keeping details private, but only when the user is not left with a false impression of certainty, completeness, or capability.
Live explainability is designed, tested and governed, not added at the end
Live explainability can not be attached to a finished product. It depends on how the system is built, evaluated, monitored, and governed. Teams need to define when the system is operating normally, when it is degraded, and when it is outside its confidence limits. They also need to test whether alerts are calibrated, whether users understand them, and whether the system’s tone changes when its evidence weakens.
This work needs more than a library of disclaimers. It needs instrumentation that detects missing inputs, retrieval failures, drift, contradictory sources, unfinished workflows, and blocked steps. It also needs evaluation of disclosure quality alongside answer quality, using four questions:
- Did the system admit the relevant limit?
- Did it describe the limit accurately?
- Did the message help the user choose the right next step?
- Did it prevent unjustified reliance?
Organisations should keep auditable records of material state changes and disclosures, especially in high-impact settings. If a system later causes harm, investigators should be able to see what it said and what it knew about the reliability of that answer at the time.
Regulators and standards bodies are moving in the same direction. The NIST AI Risk Management Framework and the OECD AI Principles both treat transparency, human oversight and the communication of limits as parts of trustworthy AI. The EU AI Act goes further for high-risk systems. Under Article 14(4)(b), the people who oversee a system must be able to recognise automation bias. The Act defines this as the tendency of “automatically relying or over-relying on the output” of an AI system. Live signals are one of the most direct ways to make that awareness possible in daily use.
Trust depends on what the system tells you before you act
The purpose of explainability is not to make an AI system sound reasonable after it has acted. It is also not only a technical record that satisfies an audit or monitors the system. Its purpose includes helping people decide when to rely on a system, when to ask better questions, when to seek a second source, and when to stop.
AI systems will always work under uncertainty. The responsible response is to show the limits of their confidence at the moment people decide. Fluent language or confident phrasing should not hide those limits, and users should not discover them afterwards.
A trustworthy AI system should be able to say, in effect: “Here is the answer, here is what supports it, and here is what’s not known. Here is what changed in the operating conditions, and here is what you should do before you rely on it.”
That is a live signal of operational health rather than a post-mortem, and people should expect it from any system they are asked to trust.