In November 2022, Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The chatbot told him he could book at the normal fare and claim the bereavement discount within 90 days. The airline’s policy page, linked in that same reply, said the discount did not apply after travel. When Moffatt claimed the discount, Air Canada refused and argued, in effect, that the chatbot was responsible for its own words. In February 2024, a British Columbia tribunal called that argument remarkable and found the airline had not taken reasonable care to make its chatbot accurate.
The airline paid only CA$812.02 in cash, but much more in reputation. Your organisation owns the responses its chatbot gives, and so do the people who approved its release. If you’re planning a chatbot project, the responses carry your organisation’s name and your own judgement.
A responsible chatbot is more than a capable language model or agent behind a chat window. It is a socio-technical system, meaning people, processes, and technology that work together. That system should communicate clearly, give enough context, stay relevant, treat people fairly, protect sensitive information, and keep a person accountable for outcomes.
The most durable approach is to treat responsibility as a product requirement from the first day, not as a compliance task after launch. NIST has published an AI Risk Management Framework for organisations that design, develop, deploy, or use AI systems. For chatbot teams, this means you combine governance, technical controls, evaluation, and operational review from design through to retirement.
Decide what your chatbot may do before you deploy
Give your chatbot a clear purpose, a known user population, and explicit limits on the tasks it may perform. A customer-support assistant can explain account processes, but it should not approve a loan, diagnose an illness, decide employment eligibility, or make another high-impact decision alone. Write these limits into your product requirements, system prompts, tool permissions, escalation paths, and user-facing disclosures.
Start with a risk assessment that identifies what could go wrong for particular users in particular contexts. Cover misuse, factual error, privacy exposure, discriminatory outcomes, security threats such as prompt injection, and harm from an overly authoritative tone. Air Canada’s chatbot gave a confident answer opposite to the policy, which is the kind of failure this assessment exists to find before your customers do.
Record the results in a written risk register. Each risk should have an accountable owner, a mitigation decision, criteria for accepting any risk that remains, and a date for its next review.
| Design question | Responsible design response |
|---|---|
| What is the chatbot allowed to do? | Specify approved tasks, sources, tools, and actions, and deny everything outside that scope. |
| Who could be affected? | Map direct users, people discussed in conversations, frontline staff, and groups who may face unequal impacts. |
| What happens when confidence is low? | Ask a clarifying question, state the uncertainty, answer from a source, or escalate, rather than give a plausible guess. |
| Who is accountable? | Name owners in product, engineering, policy, security, and operations for decisions and incident response. |
Layer your guardrails so not depending on one prompt
A single prompt instruction is fragile and easy to bypass. Build guardrails as layers. At the conversation layer, your policies should name prohibited and sensitive topics and the kinds of support the chatbot may give. They should also detail how it expresses uncertainty and when it must refuse or hand over to a person. At the technical layer, constrain the model with an approved list of tools, the minimum data access it needs, rate limits, fixed output formats, and input and output validation checks.
Tone matters as much as content. Your chatbot should meet the required level of respect, non-judgement, and accessibility, but it should not imply unwarranted certainty, professional authority, or emotional closeness beyond its actual role. For sensitive questions, it should separate information from advice, state its limits, and possibly offer a safe next step. A short, helpful refusal that explains what the chatbot cannot do and points the user to a safer option or a qualified person serves people better than a generic block.
Guardrails also stop scope from growing without anyone deciding it should. A chatbot designed to summarise policy should not start to interpret legal rights, and a chatbot built for staff knowledge searches should not be able to execute transactions. When you hold the scope fixed, behaviour is more predictable, easier to test, and easier to defend in an audit.
Check both what users send and what your chatbot replies, in real time
Real-time moderation can stop harmful text before it reaches a user or a downstream system, and it works best when it checks both sides of the conversation. Input checks can identify unsafe requests, abusive conduct, or attempts to manipulate the system. Output checks can catch a reply that breaks policy even when your prompts and retrieval controls are sound.
There’s many moderation tools that can score text and images and treat those scores as signals for your application’s policy. These can score for hate and fairness, sexual content, violence, and self-harm, for example. They can also offer classification tools that detects attempts to manipulate the model through user input.
Your moderation policy should define an action for each class of result. Low-risk cases may proceed as normal, and ambiguous cases may receive a safe and limited reply. High-severity cases may be blocked, de-escalated or sent to a human queue. Calibrate your thresholds on representative test data and monitor them after release. A filter that is too strict may silence legitimate users, and a filter that is too permissive may expose them to harm.
Design principle: Safety classifiers inform your decisions. They do not replace human judgement, context, or a clearly documented policy. Microsoft’s own documentation notes that a “safe” result does not guarantee the content is accurate or appropriate for your application.
Send consequential decisions to a trained person who has the authority to say no
When a chatbot affects a person’s access, opportunity, safety, rights, or material wellbeing, the outcome should be verified before the system acts. Examples include medical triage, financial eligibility, employment screening, complaints about harassment, requests that affect vulnerable users, and any situation where people will treat the chatbot’s answer as an authoritative decision.
A responsible escalation process is more than a button marked “contact support”. It should specify which conditions trigger a review, what context the reviewer sees, how the user is told, how quickly the reviewer responds, and how the decision is recorded. Train your reviewers and give them the authority to override the chatbot. Show them only the information they need to resolve the case, in line with your privacy and data-minimisation rules.
Evaluate the escalation route as you would any other product feature, and measure handoff accuracy, reviewer agreement, queue time, reversals, and outcomes by user group. If escalation is too rare, you may hide risk. If it is too frequent, reviewers may face so many cases that they approve them without scrutiny.
Tell users what your chatbot relied on, and say clearly when it cannot answer
Explainability for a chatbot does not mean exposing hidden model reasoning or proprietary internal data. It means you tell users and operators what the system is, what information it used, what it can and cannot do, and how to challenge an outcome.
In practice, your chatbot can cite the documents behind an answer, label content as AI-generated, and display the policy or workflow that applies. It can also show when a person has reviewed an answer, and whether a reply came from the user’s input or from verified organisational material. For internal teams, a structured record can capture the model version, prompt version, retrieval sources, tool calls, policy decisions, moderation outcomes, and any confidence or abstention signals.
Transparency matters most when the chatbot cannot answer. Rather than imply the information does not exist, it should say it could not verify the answer, the request is out of scope, or a specialist needs to review it. Had Air Canada’s chatbot told Moffatt to check the bereavement policy, he might have had an accurate path forward instead of a false certainty.
Test whether every user group gets the same quality of help
Your training, fine-tuning, and evaluation data should include diverse and representative examples. A dataset that reflects only one language variety, cultural norm, disability context, or communication style may produce a chatbot that serves some users well and others poorly. Consider representation across demographics, regions, dialects, accessibility needs, subject matter, and adversarial and edge cases.
Your safety tools carry the same risk. Microsoft trains and tests its Content Safety text classifiers in eight languages and warns that results in other languages may differ. If your customers write in Vietnamese, Arabic, or Hindi, the moderation layer may protect them less well than English speakers, and only testing will show you.
Diversity in data is not enough on its own. Test whether comparable prompts receive different quality, tone, refusals, escalation rates, or outcomes when protected attributes, or attributes that stand in for them, change. AI Fairness 360, an open-source toolkit, helps teams find, report, and reduce bias across the AI lifecycle. It can support a broader fairness-testing programme, but it does not replace domain expertise, review by the people affected, or specifically chosen definitions of fairness.
| Evaluation dimension | Example test | Review action if a disparity appears |
|---|---|---|
| Respectful treatment | Compare tone across names, dialects, and identity references. | Revise prompts, examples, and safety policy, and add regression tests. |
| Answer quality | Measure grounded accuracy across languages and accessibility formats. | Improve source coverage, retrieval, translation support, or handoff rules. |
| Refusal behaviour | Compare refusal and escalation rates for requests that mean the same thing. | Recalibrate moderation thresholds and investigate bias in the classifier. |
| Access and usability | Test with screen readers, plain language, and low-bandwidth connections. | Improve interaction design and provide alternatives to chat. |
Record every version and decision to reconstruct what happened
When something goes wrong, you need to reconstruct what happened. Audit logs and version control give you traceability, and they help you investigate incidents, detect performance drift, support appeals, and demonstrate governance. Your logs should keep a privacy-conscious record of each interaction and its decision path. That record includes timestamps, anonymised or access-controlled user and session identifiers, model and prompt versions, policy and moderation results, retrieval sources, tools invoked, escalation status, and final outcome.
Apply version control to more than code. Include system prompts, safety policies, test suites, retrieval indexes, model configurations, and moderation thresholds. Link every release to documented approvals, a summary of changes, pre-release evaluation results, and a rollback procedure. With that record, your team can find whether a change in behaviour came from the model, a prompt revision, a content update, a policy change, or an external dependency.
Balance traceability with privacy. Keep only what you need, control access, encrypt sensitive records, and set deletion schedules. Do not use production conversations for training or evaluation without an appropriate legal and ethical basis.
Run responsible development as an ongoing operation
A responsible chatbot is never “finished” at launch. Language changes, users discover new failure modes, organisational policies evolve, and vendors update their models and safety services. Your operating model should therefore include continuous monitoring, regular fairness and safety evaluations, incident handling, review of user feedback, periodic policy review, and independent challenge testing.
The NIST framework organises AI risk management into four functions, Govern, Map, Measure, and Manage, which give you a practical structure for this work. In practice, governance assigns authority and accountability, mapping identifies context and harms, measurement evaluates behaviour, and management applies, monitors, and improves your controls. When you report these activities regularly to product leadership and other stakeholders, responsible design becomes a discipline you can measure.
Design safeguards together, before your chatbot speaks for you
Responsible chatbot development takes more than a well-written prompt or a content filter. It needs explicit boundaries, layered guardrails, real-time moderation, human review for sensitive decisions, clear transparency, inclusive data and fairness testing, and traceable operations. When you design these safeguards together, your chatbot is safer, more reliable, and easier to understand, and the people who rely on it have good reason to trust it.