The rapid adoption of AI across the Australian financial services sector has moved from competitive advantage to operational necessity. But the speed of adoption has created a problem regulators have noticed: the gap between deploying AI and actually governing it.
For Australian Financial Services licensees, that gap is not a technology problem. It is a compliance problem. ASIC and APRA are increasingly focused on whether firms can demonstrate they understand what their AI tools are doing, why, and what happens when things go wrong. The Design and Distribution Obligations (DDO) and the requirement to act “efficiently, honestly, and fairly” apply to AI-driven decisions the same way they apply to any other business process. The AI does not change the obligation, it just makes it harder to see when the obligation is being breached.
1. The Risk Landscape: What You Probably Haven’t Tested
AI introduces risk profiles that are genuinely different from traditional software, not because the regulatory obligations are different, but because the failure modes are less visible and often more systematic. Three characteristics make AI uniquely difficult to govern: it can be confidently wrong, its errors can be invisible in aggregate data, and its behaviour can change without anyone deliberately changing it.
When the AI is wrong, it doesn’t know it
Traditional software fails visibly; an error code, a system crash, a blank screen. AI models fail silently, generating outputs that are plausible, formatted correctly, and entirely false. In financial services, this matters immediately.
Consider an AI chatbot deployed to answer client questions about a managed fund. A client asks whether the fund has capital protection. The AI, drawing on training data that includes older product literature, confirms that it does. The fund’s current Product Disclosure Statement says otherwise. The client invests based on the AI’s answer. No one in your firm knows this happened, because the chatbot logged the interaction as a resolved query, not a complaint.
This is a predictable consequence of deploying a generative AI tool without grounding it against your current, authoritative product documentation and without a mechanism to detect when a client has acted on AI-generated information that contradicts your PDS. Under DDO, distribution based on false product information is a breach, regardless of whether a human or an AI provided it.
Other failure modes in this category that firms often underestimate:
- AI-generated PDS summaries distributed to advisers: If your AI summarises a PDS to help advisers understand a product, and that summary contains a material error, every adviser who relied on it has potentially distributed outside the TMD. The summary feels authoritative. No one fact-checks it against the original.
- Client mis-categorisation at scale: An AI categorising clients by risk profile or financial sophistication can be wrong about individual clients in ways that are invisible in aggregate accuracy figures. If the tool is 94% accurate overall but systematically worse for clients over 65, a cohort you’ve never tested separately, you may have a bias problem embedded in every interaction with that demographic.
- Autonomous AI agent conduct: An AI agent authorised to take actions on behalf of clients, scheduling, executing instructions, communicating, can engage in conduct that would constitute misconduct if a human adviser did it. The firm bears that liability.
The complaint that never becomes a complaint
One of the more consequential blind spots in AI-assisted customer service is the complaint that gets resolved before it is recorded.
When a human customer service representative handles a complaint, there is generally a process: the interaction is flagged, the complaint is logged, it enters the complaints management system. When an AI handles the same interaction, the outcome depends entirely on how the AI has been configured to classify what it receives.
A client who says “I’m really unhappy with how this was handled and I want something done about it” may receive a sympathetic, helpful AI response that resolves the surface issue. The AI logs the interaction as a successful resolution. In your complaints data, it does not appear. In your DDO reporting, it does not appear. In your significant dealings analysis, it does not appear.
If this happens at scale, because AI tends to apply consistent classification errors consistently, your regulatory reporting may systematically understate the number and nature of client complaints. Section 994F(4) requires accurate complaints reporting. An AI that is resolving complaints before they can be counted is not a compliance solution, but a compliance risk disguised as good customer service.
Model drift: the AI that changed without anyone changing it
AI models are not static. Their performance is anchored to the data they were trained on, and when the world diverges from that data, their outputs become less reliable. Gradually, quietly, and without any visible system failure.
Consider a creditworthiness or suitability assessment tool trained on client data from 2020 to 2022. That tool learned patterns from a period of historically low interest rates, rising asset values, and a particular economic environment. If it has not been re-validated against current conditions, it may be applying assumptions that no longer hold. Clients who would have been correctly categorised as suitable for a particular product four years ago may no longer be. The tool does not know this. It continues to categorise them as suitable.
This is model drift. It is the AI equivalent of a policy that nobody has reviewed in years. The difference is that when a policy drifts out of date, someone usually notices. When an AI model drifts, the outputs often look the same, the categories still get filled, the reports still get generated, and the process still runs. The problem is invisible until something goes wrong at scale.
2. Compliance and the DDO Framework
The DDO framework, as outlined in ASIC Information Sheet 264, does not make exceptions for AI. Every obligation that applies to a human process applies equally when that process is automated. The relevant question is not whether AI was involved, but whether the outcome met the regulatory standard.
AI in product design
When an issuer uses AI to design a financial product or determine its Target Market Determination, the AI must correctly account for the likely objectives, financial situation, and needs of the target class. ASIC’s framing is clear: the target market is the class of consumers for whom the product is likely to be appropriate, and the TMD must accurately describe them.
If the AI that helps determine your TMD has been trained on data that overrepresents a particular client profile, for example urban, higher income, digitally engaged, it may systematically underweight the needs of clients outside that profile. The TMD looks complete. But it may not accurately describe the full range of consumers for whom the product is or is not appropriate.
AI in distribution and monitoring
Distributors are increasingly using AI to monitor distribution conduct and identify significant dealings inconsistent with the TMD. This is exactly the kind of application DDO contemplates. But the reliance on AI for this function creates its own obligations.
The AI must be capable of correctly applying the definition of a “significant dealing”. It must detect when the proportion of consumers outside the target market reaches a threshold that triggers reporting. It must not classify borderline cases consistently in one direction. And it must ensure that the “reasonable steps” required under DDO are documented, not just assumed.
Automation that speeds up distribution is good, but without preserving the compliance steps is not a solution. It is a breach that scales.
The “efficiently, honestly, and fairly” standard
Section 912A of the Corporations Act requires licensees to provide services efficiently, honestly, and fairly. ASIC has indicated that a lack of explainability in AI decisions may fail the fairness test. If an AI makes a recommendation or assessment, and the firm cannot explain why, that opacity is itself a compliance risk.
This is particularly relevant for advice tools. An AI that generates recommendations not tailored to the individual client’s circumstances, because it is applying a generalised model rather than genuinely assessing the specific client, may fail the best interests duty and suitability requirements, even if the recommendation appears reasonable on its face, or only used internally.
3. Governance Checklist: Questions Worth Asking
The questions below are not designed to confirm that your governance is in order. They are designed to find the places where it isn’t. If you find yourself answering a question with “I assume so” or “someone else handles that,” you have found something worth investigating.
Accountability
- If ASIC asked you today which AI tools are making or materially influencing client-facing decisions, could you hand them a complete list within an hour?
- Name the person who would receive the call if one of your AI tools caused a compliance breach tomorrow. Does that person know they are accountable for it?
- Is there a human being who has read the AI vendor’s model documentation, not just the sales materials, and signed off on its fitness for your specific use case?
Distribution and TMD integrity
- If your AI is involved in the distribution process, can you demonstrate that it has never bypassed a “reasonable step”, or do you assume the vendor built those in?
- Have you tested whether your AI correctly identifies a “significant dealing”, not in documentation, but in a live scenario with borderline cases?
- When did you last audit the AI’s categorisation outputs against your TMD to check for systematic misalignment?
Complaints and reporting
- Could your AI chatbot resolve a client complaint without it ever appearing in your complaints data? If you don’t know, that is your answer.
- Does the data flowing from your AI customer service tools into your DDO reporting get validated by someone who did not build the tool?
- Have you cross-checked your AI-generated complaints figures against any other data source, e.g. inbound call volumes, churn patterns, adviser escalations, to check they are plausible?
Model integrity and drift
- What economic conditions was your suitability or creditworthiness AI trained on? Have you re-validated it since those conditions changed materially?
- Is there a scheduled review process for your AI models, or does re-validation happen only when someone notices a problem?
- If a vendor updated their model overnight, would you know? Would anything in your process change?
Transparency and explainability
- For each AI tool used in a client-facing decision, can you produce a reason code or plain-language explanation for a specific output? Or does “the model decided” end the conversation?
- Have you tested whether your AI-generated PDS summaries are accurate against current PDS documents, not when the tool was deployed, but recently?
- If a client asked why the AI categorised them the way it did, could anyone in your firm answer that question?
Bias and fairness
- Have you tested your AI’s accuracy across different client demographics, by age, income bracket, geography, or product experience, rather than only in aggregate?
- If your AI performs significantly worse for a particular client cohort, would your monitoring processes detect it? What would trigger the alert?
Incident response
- Does your organisation have a kill switch for each AI tool. A documented, tested mechanism to halt its operation. Or does the plan exist only in theory?
- When did you last test the kill switch? Triggering it to ensure it works as expected?
- If an AI tool needed to be turned off today, what would happen to the clients and processes that depend on it? Has anyone mapped that?
Vendor and third-party risk
- Does your AI vendor notify you before making changes to their model? What is your contractual right to know when the model you approved is no longer the model you’re running?
- Has your vendor been assessed for compliance with the Privacy Act? Have you verified that client data used to train or improve their model has not been retained in ways you did not authorise?
- Are there mechanisms in place to prevent data poisoning, deliberate or accidental corruption of the inputs your AI relies on?
Conclusion
AI offers real opperational advances, but brings with it governance risks not fully covered by your existing controls. Use the questions offered here as a starting point, not a definitive checklist. Firms that haven’t built the practice of thinking through their AI controls before they need it will be caught short.