How to Evaluate an AI-Powered AML Platform

Last updated: October 2026

An AI AML platform has to pass every test you would set any AML system, covering coverage, case management, data and cost. It also has to pass four tests that a rules-based evaluation tends to underweight: the evidence behind every output, human confirmation by design, change control over models and agents, and outputs an examiner can read. Run all four on your own data before you sign.

TL;DR

• An AI-powered AML platform needs four tests on top of a general AML evaluation: the evidence behind every output, human confirmation by design, change control over models and agents, and outputs an examiner can read.

• Ask for the data, the source and the reasoning behind any single recommendation, in writing. A fluent summary with no path back to the transactions fails the first test.

• Find every point where a person decides to close, escalate or file, and ask who set those limits and on what evidence.

• The April 2026 interagency model risk guidance (SR 26-2) covers machine-learning models but places generative and agentic AI outside its scope. Examiners and buyers still expect testing and validation.

• Run the tests on your own historical alerts, then in shadow alongside your analysts, before the platform touches a live decision.

What should you test in an AI-powered AML platform?

Test four things that a rules-based AML evaluation underweights: whether the platform shows the evidence behind every output, whether a person confirms each recommendation by design, whether changes to models and agents are approved and recorded, and whether an examiner can read what it produces. Then run all four on your own data.

1. Evidence behind every output. Each recommendation carries the data used, its source and the reasoning, and an analyst can open the underlying transactions.

2. Human confirmation by design. The platform shows where people decide, and nothing closes, escalates or files without a decision the institution configured and can defend.

3. Change control over models and agents. Every change to a model, prompt or configuration is versioned, tested and approved before it reaches live alerts.

4. Outputs an examiner can read. Logic and case records make sense to someone outside the AML team.

The question underneath all four is the one buyers ask first. A chief compliance officer at a payments company, building a program from scratch, put it this way: "How am I going to know that the systems are operating as intended? How do I know AI systems are working?"

A rules engine answers part of that by construction: the same input fires the same rule, and the rule can be read. A language model does not.

A compliance operations lead for onboarding at a bank said language models seem to have good days and bad days, so their team had stripped as much subjectivity out of the process as it could. An IT delivery manager at an asset manager described an earlier pilot that dispositioned financial-crime alerts with a general-purpose language model prompt. Nothing in it was auditable or explainable, the results were wrong, and it was switched off after a short trial.

These four tests add to a general evaluation. For coverage, case management, data and cost, start with the general AML software buyer's guide and an evaluation framework for transaction monitoring platforms. For the difference between a prompt and an agent that works within set limits, see what an autonomous AI agent is in risk work.

Test 1: the evidence behind every output

The question to ask: for any single recommendation, show me the data the system used, where each piece came from, and the reasoning that connects them, in writing.

An examiner will ask how the institution reached a conclusion, and pointing at the system is not an answer. A director of compliance and BSA at a community bank described the problem directly: "if I don't have it in writing, I won't be able to defend it." An AML quality assurance lead at a fintech asked for the same thing from the review side: documentation they could show an examiner of where the system got each piece of information.

The analyst also has to be able to reach the evidence. The same director said that, however much they like AI, they do not fully trust it, and they need to be able to go back and research an alert themselves. A platform that shows a summary but not the transactions behind it leaves the analyst unable to check the work.

Probe two failure modes on purpose:

• Fabricated facts in drafted text. A head of product platform strategy at a payments company said that if a drafting tool hallucinates, sending FinCEN a report with even a few wrong words in it would be a very big problem.

• Evidence from the wrong place. A payments underwriting, risk and compliance VP at a bank asked how a vendor would stop an investigation summary from citing activity that sits in a different portfolio.

A weak answer here is a polished narrative with no link back to the records, or a reasoning trace that cites data the analyst cannot open. Auditors already ask for the proof behind documented findings: a compliance manager at a consumer lender said that was the question after their most recent BSA/AML audit. To test it in a demo, pick a closed case you know well, ask the platform to show its evidence, and check every cited fact against your own systems.

Test 2: human confirmation by design

The question to ask: show me every point where a person confirms, escalates or files, and show me what, if anything, can happen without one.

Regulators expect the institution to demonstrate oversight. An MLRO at a payments company said that, presenting an AI system to a regulator, they would need to explain where individual people make the decisions on escalation and SAR filing, and be able to demonstrate those decision points. The platform should make that map easy to produce.

Then ask what can close without review, who configured it and on what evidence. A compliance AI lead at a bank described the order they would follow before allowing any closure without review: build up alert history, run parallel testing to show the agent works as expected, and only then move, in step with the bank's internal model review lifecycle. That decision belongs to the institution, and the platform should record it as one.

Ask, too, whether behavior changes on its own. A BSA team member at a credit union said they had assumed the system would learn from an alert their team closed over and over and start closing it by itself. Behavior should change only through an approved, recorded change, which is Test 3.

Guardrails around the model matter as much as the model. The IT delivery manager at the asset manager put it plainly: "you need deterministic guardrails on the front. You need the LLM in the middle and you need deterministic guardrails on the back." Ask which steps are fixed rules, which are model output, and where a person signs off.

A weak answer describes human review as a setting the customer can switch off, or cannot say who set the thresholds. For where human sign-off sits across a whole program, see what AI changes in an AML program.

Test 3: change control over models and agents

The question to ask: show me the history of every model, prompt and configuration change, who approved it, and what the results looked like before and after.

An AI-powered AML platform changes in more places than a rules engine: models get retrained, prompts get edited, agent instructions get revised. Each change can alter which alerts get attention and what the case file says. The compliance AI lead at the bank said the end-to-end process must be defendable under internal model risk audit, which means being able to show "the history of the prompts or the history of the results, what's changed over time".

BSA teams already work this way for rules. A BSA officer at a bank contrasted the two sides of the house: fraud teams change rules first and send them to model risk for review later, while BSA rule changes go through review before they change. A BSA program owner at a credit union described the same duty from the owner's seat: explaining how the system alerts and, where it changed during the period, how they ensured the change was sound.

Three things to ask for:

• Approval before live. Every change to a model, prompt or configuration that touches AML alerts is approved by the institution before it reaches production.

• Audit trails for test runs. During a proof of concept at a bank, the evaluation team asked whether audit trails exist for test runs, not only for production decisions. They should.

• Model documentation that names its inputs. A compliance and risk leader at a bank asked what the model documentation would be, for example a record that the agent was configured from the bank's own procedures and the rules carried over from its previous system.

Where model risk guidance stands

On 17 April 2026 the Federal Reserve (SR 26-2), the OCC (Bulletin 2026-13) and the FDIC (FIL-15-2026) issued revised interagency model risk management guidance. It supersedes SR 11-7 and the 2021 interagency statement on model risk management for BSA/AML systems. Machine-learning models used in AML fall within it. The OCC bulletin is explicit about where it stops: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."

The agencies have said they plan to issue a request for information on model risk management, considering banks' use of AI in particular. As of October 2026 it has not been issued.

Examiners still expect testing. A senior sanctions compliance officer at a bank said every system goes through validation under their model risk policy, and that AI, being newer, needs extensive testing to show the regulator they are comfortable with it. The BSA program owner at the credit union said independent model validation is guidance rather than law, but that it now feels like a requirement examiners will raise if it is missing.

A weak answer is a vendor release schedule with release notes as the only record, where nobody at the institution approves a change that alters its alerts.

Test 4: outputs an examiner can read

The question to ask: export your logic and one complete case record, and hand them to someone outside the AML team. Can they follow both?

The chief compliance officer at that same fintech described turning the rules into a policy document that lists each rule and what it does, so an examiner with a general business background and no AML specialism could follow it. That is the standard to hold an AI AML platform to: logic and case records written for a reader who does not use the product.

Readable outputs also protect the institution's own understanding. The director of compliance and BSA at the community bank said a system the team cannot understand would be very hard to defend, because "regulators like knowing that you understand how a system works."

Completeness is part of readability. The MLRO at the payments company said the things a regulator calls out are missing information and numbers that do not foot: if an alert covers a set of transactions, the write-up should cover every one of them. Check that the platform's case record reconciles to the alert.

Finally, ask the vendor what examiners have raised at other institutions using the platform. The compliance and risk leader at the bank asked exactly that, including whether validation documentation had come up. A weak answer is logic readable only inside the product, or explanations written in model terms an examiner cannot check against the case.

The four tests at a glance

Use the table as a scorecard during the proof of concept, one row per test.

How to run the tests on your own data

A demo on the vendor's data shows the vendor's best case. The director of compliance and BSA at the community bank said that because the workflows are customized and tailored to the institution, a proof of concept on its own data was a requirement before buying.

1. Agree the acceptance criteria first. Decide what a pass looks like for each of the four tests before any data moves.

2. Backtest on historical alerts. Run the platform against closed alerts with known outcomes and compare what it recommended with what your analysts decided.

3. Run in shadow. Put it alongside analysts on live alerts, with no effect on decisions, and track agreement over time. Read each disagreement as well as the overall rate.

4. Ask who tunes and who tests. The chief compliance officer at the payments company called monitoring and testing from a regulatory compliance perspective mind-numbing but critical, and wanted to know whether the institution or the vendor tunes the rules.

5. Ask how bias, fabrication and drift are tested. A senior compliance manager at the same consumer lender asked how a vendor prevents bias, hallucination and drift so the lender does not end up with a UDAP problem. The compliance and risk leader at the bank asked how the vendor validates its AI governance, including bias. Ask for the test results themselves.

6. Bring your regulator along. A risk management leader at a sponsor bank described running AI in parallel with existing processes as the way to get regulators comfortable over time.

Common mistakes in an AI AML evaluation:

• Testing on the vendor's sample data instead of your own alerts.

• Treating a summary as evidence.

• Accepting a claim that the system learns without asking who approves each change.

• Setting acceptance criteria after the proof of concept has started.

If you oversee fintech programs

The same four tests apply across every program you oversee, with one more question: can you see each partner's alerts, evidence and changes in one place, and show an examiner which program's logic changed and when? A chief digital and risk officer at a sponsor bank said an examiner had pressed hard on data validation, AI validation and model validation at a previous institution, which is why those questions now come first.

A head of BSA/AML technology and program assurance at a sponsor bank described adding an independent testing layer between operations and audit, so the bank can show examiners the targeted and thematic testing it has already done. For how a sponsor bank structures that oversight, see sponsor-bank AML oversight across fintech programs.

Frequently asked questions

Is an AI AML platform different from AML software that uses machine learning?

An AI AML platform usually means more than a machine-learning model scoring transactions. AI transaction monitoring, where a model scores or prioritizes alerts, is one part. The rest is AI applied to the investigation itself: gathering evidence, summarizing activity and drafting recommendations or narratives for an analyst to review. That second part is where evidence, human confirmation and change control need their own tests.

What audit trail should an AI AML platform keep for examiners?

For each decision, the audit trail should show the data used and its source, the reasoning, the recommendation, the person who confirmed or overrode it, and the version of the model, prompt or configuration that produced it. It should also record test runs and every change with its approval. An examiner should be able to rebuild any decision from the record alone.

Does model risk management guidance apply to AI AML platforms?

In part. The revised interagency guidance issued on 17 April 2026 (SR 26-2, OCC Bulletin 2026-13, FIL-15-2026) covers machine-learning models and supersedes SR 11-7 and the 2021 BSA/AML model risk statement. The OCC states that generative and agentic AI models are not within its scope, and the request for information the agencies plan on model risk management, with AI in particular, had not been issued as of October 2026. Institutions still validate and test those systems because examiners expect it.

How can an AI AML platform reduce manual review while keeping human sign-off?

The platform collects the evidence across systems and drafts a recommendation, and the analyst reviews it and decides. Any change to what needs review is the institution's call, made after backtesting and shadow running and recorded as an approved change. The time saved comes from the collection work, and the sign-off stays with people.

Where Oscilar fits

Oscilar's agents can run as a standalone layer on top of infrastructure you already have, or as part of the unified platform. They are built so a person confirms each recommendation: the agent gathers the evidence and recommends, and the analyst decides. Oscilar keeps a full audit trail per decision with stored reasoning, and is human-in-the-loop by design.

The platform is designed to fit the interagency model risk management guidance. That guidance places generative and agentic AI outside its scope, so no claim of fit extends to agents: hold the agent layer to the four tests above, on your own data.

To see how this applies to a bank's AML program, read about Oscilar's AML solution for banks.