AI Agents for Compliance: Is Your Pilot Real, or Just a Really Good Demo?

At the start of our recent “build vs. buy” webinar, we asked the room a simple question. Where is your organization with AI agents? 

Most people (62%) said they’re piloting one or two.

Jonathan Awad, co-founder and CEO of Baselayer, didn't let that slide idly by. "Piloting one to two agents is a bit of a cop-out answer," he said, live, to a room full of people who had just chosen it. 

He had a point, and the rest of the panel spent the hour backing it up. Jason Chorlins (Financial Services Advisory, Kaufman Rossin) and our own Saurabh Bajaj (Chief Product Officer) joined him, with Phil Goldfeder (American Fintech Council) moderating.

The short version: most agent pilots clear the demo bar but fail the operating bar. Below are five myths compliance teams tell themselves about their pilots, and what the panel said back. Consider this blog your October dose of agentic AI mythbusters. 

TL;DR: What separates an AI agent pilot that works from one that stalls?

A working agent changes how your team spends its day, moves the numbers your CCO tracks, and keeps up when your procedures change. A stalled pilot produces output that looks good in a demo and changes nothing on a normal Tuesday. The five myths, in brief:

• "We're piloting" doesn't tell you much. The real test is whether your team would notice if the agent disappeared.

• Good-looking output isn't proof. Have an experienced investigator redo a sample of the agent's decisions.

• Your analysts usually aren't the bottleneck. It’s your data and workflows.

• Building it yourself is cheaper only if you build it perfectly. For most teams, buying first and building later is the safer path.

• The first working agent isn't the finish line. Rules, lists, and products change, and the agent has to keep up.

What's the difference between the demo bar and the operating bar?

The demo bar is whether an agent looks impressive when someone shows it to you. The operating bar is whether it holds up in your queue, at your volume, every day. Saurabh put a number on how often teams confuse the two.

"Nine out of ten agent projects clear the demo bar, but they fail the operating bar." Saurabh Bajaj, Chief Product Officer, Oscilar

How can you tell which side of the line your pilot is on?

Each of the five myths below is one version of mistaking the demo bar for the operating bar, and each ends with a quick check you can run on your own pilot.

Myth 1: "We're piloting a couple of agents."

Jonathan's pushback was about honesty. Piloting can mean an agent doing real work you trust, or it can mean "very simple, quick things that don't really matter." Most of the time, he said, it's the second.

He described agentic AI in banking as a barbell. Most financial institutions aren't touching agents at all. A small number, mostly with big dedicated teams, use them for nearly everything. His example was JPMorgan, which recently moved to block outside agents from interacting with its accounts while building agents of its own.

Saurabh sees the middle of that barbell filling in. A year ago, most prospects he talked to didn't know where to start. Now the loose proof-of-concept conversations are turning into one or two agents that teams are sure about.

So what does "working" look like? Jonathan's answer was simple. The agent becomes part of the routine, helping your team clear its queue and aid in training new hires. It works when, according to him, "you can't imagine your life without it anymore."

He also named the trust gap. Very few teams let an agent clear anything on its own, not even obvious false positives. The goal he described is a morning queue sorted by how confident the agent is, so analysts know exactly where to put their heavier focus first.

"Piloting one to two agents is a bit of a cop-out answer." Jonathan Awad, Co-Founder and CEO, Baselayer

Check this instead: If you switched your agent off on Friday, would anyone notice on Monday?

Myth 2: "The outputs look great."

This is the demo bar in its purest form. Saurabh's warning: "A lot of people are shipping what I call the LLM dropped on one of your surfaces, and they call it an agent." It might write a tidy summary, clear the demo, but it doesn't actually change how the team works.

He also pointed out that good-looking AI output is now cheap. Anyone can generate pages of research and recommendations, and much of it ends up as documents nobody reads (but probably asks their own AI to summarize in turn). That raises the bar for what counts as value.

To put it another way, a CCO doesn't judge an agent by its prose. Saurabh listed what they actually track:

• Alert handling time

• Decision accuracy

• Time to adapt to new strategies and typologies

• Lift to the team from using the agent

He made it concrete. If an analyst spends about 60 minutes drafting a SAR, or about 50 minutes on a first-level alert review, the question is whether an agent can bring that down to a fifth. Jason added the test he trusts most. Take a sample of the agent's real outputs, hand them to an experienced investigator, and have them reproduce the decisions. Where they disagree, feed that back into the agent and run it again.

"This is a much better test of readiness than asking whether the summaries look good to you." Jason Chorlins, Kaufman Rossin

Kaufman Rossin holds its own agents to the same standard. Jason said his team runs three or more agents on first-level review work, and before trusting them, their team wrote down exactly what format each output should take and the minimum standards every output has to meet. 

Oscilar's AML alert review agent is built on the same idea: it suggests a disposition with its reasoning, and the analyst makes the call.

Check this instead: Give 25 recent outputs to your most senior investigator. How many decisions would they change?

Myth 3: "Our analysts are the bottleneck."

Overheard in the chat: One attendee put it bluntly. When an analyst has to check 15 things across messy systems, slow work gets blamed on the person instead of the system. Most teams, they added, don't realize how complicated their own workflows can get.

Saurabh had been making the same point on air. A typical SOP for an AML transaction monitoring or sanctions review says something like "check these 15 things." Checking them means going into different systems and figuring out whether you can even pull the data. That's a data orchestration problem, and banks' data can be messy. An agent dropped on top of it inherits the mess.

Jason described what changes when that layer works. He calls it flipping the ratio. Analysts used to spend about 80% of their time preparing an alert for review and 20% actually investigating it. With agents doing the gathering, he's seeing that flip, so the analyst's time goes to deciding whether the activity makes sense for that customer.

He also listed what struggling pilots tend to have in common:

• Incomplete customer data

• Unclear investigation standards

• Narratives analysts can't trace back to the underlying records

None of those are analyst problems. Neither is the gap another attendee raised in the chat: their vendors are all adding AI agents, but the agents don't share context. Fraud, onboarding, AML and customer service affect each other, so separate agents give each team an incomplete view.

"Adding 100 people to the job will not make sense. You'll need to figure out how the current 100 people do the job of a thousand. What does it mean to give them Iron Man suits?" Saurabh Bajaj, Oscilar

Check this instead: Follow 10 alerts from open to close. How much of the analyst's time went to gathering information, and how much to making the call?

Myth 4: "We'll just build it ourselves."

Phil asked Jonathan whether "build vs. buy" was even the right title for our panel. Jonathan's answer: the first decision is how, where, and when to use agents at all. “Build vs. buy” comes after that, and it depends on two things. Do you have the culture for it, and how many engineers do you have?

If you're still working on buy-in, he said, building is risky. Getting an organization excited about agents and then shipping something rushed or wrong "would really set folks back pretty significantly." If you have a strong engineering team and the culture to match, building some things can make sense.

Saurabh gave a simple way to sort it. Plot the problem on two axes: how fast it changes, and how many signals it pulls from. Small, self-contained tools that don't change much are good candidates to build in-house. Anything fast-moving or multi-signal is a job for a partner.

Only one quadrant is a build. Everything that moves fast or pulls from many signals goes to a partner.

Cost came up in the Q&A, too. An attendee asked about the main risks of building, from data security to the stability of the outputs. Jonathan said a vendor absorbs much of that risk, since this is the vendor's whole business and not a side project. Is building cheaper? "If you do it perfectly, for sure. If you mess it up, definitely not. It ends up costing way more."

Jason added that many smaller community banks don't have that expertise in-house, which makes picking the right partner even more important.

Not sure where your next agent should come from? Our 2-minute build vs. buy readiness check scores your team on the questions the panel raised.

Myth 5: "Once it's live, we're done."

Jason has watched programs succeed and stall, and he sees the same split. The ones that succeed know their own workflows in detail: what counts as a routine alert versus an escalation, which data is reliable, and who has the authority to decide. They also have the capacity to maintain integrations, test changes and monitor performance after launch.

The ones that stall treat the first working agent as the finish line. Then an AML rule changes, a sanctions list updates, or a new payment product brings new patterns, and the agent falls behind. "You never want to treat it as a finish line," Jason said.

Saurabh explained why keeping up is harder than it sounds. SOPs in AML and fraud change constantly. If every change means asking an engineer or filing a vendor ticket, you're back to depending on someone else for every update. And the agent's output needs set rules for what happens next, like when to pull more data or escalate. It can't improvise its way through a compliance workflow.

"The programs that I see stall are the ones that treat the first working agent as almost the finish line." Jason Chorlins, Kaufman Rossin

Two more things change once an agent is live. You need to know it's still making the right calls across hundreds of alerts, not one at a time. And when an examiner asks how a decision was made, you need to replay it with the evidence behind it. Saurabh noted that doing this by hand can take a team one to two months.

Check this instead: Pick your last procedure change. How long did it take to reach the agent, and who made the change?

How do you know if an AI agent for compliance is working?

Ask the five questions below. If your honest answers sit in the middle column, your pilot is still at the demo bar.

Pushing your agentic AI pilot past the demo

None of the panelists argued for slowing down, but emphasized how to move fast without fooling yourself. In short, pick one high-volume workflow you understand well. Measure what your CCO measures. Have someone you trust check the agent's work. Then you'll know whether you have a pilot or a demo.

Want the full conversation? Watch the on-demand recording.

FAQs

How do you know if an AI agent for compliance is working?

Check whether it changes daily work, not whether its output looks good. A working agent is part of the team's routine, moves measurable numbers like alert handling time and decision accuracy, and holds up when an experienced investigator reproduces a sample of its decisions.

Should compliance teams build or buy AI agents?

It depends on your culture, your engineering capacity and the problem. Small, self-contained tools that don't change often, like a policy Q&A tool or a summarizer, can make sense to build. Problems that move fast or pull from many signals usually favor a partner. Many teams start by buying and build later.

What metrics should you use to evaluate a compliance AI agent?

Start with what a chief compliance officer tracks: alert handling time, decision accuracy, time to adapt to new strategies or typologies, and the lift to the team. Pair those with a sample review, where an experienced investigator redoes the agent's decisions.

Why do AI agent pilots stall?

Usually because of what sits under the agent. Struggling pilots tend to have incomplete customer data, unclear investigation standards and narratives analysts can't trace back to records. Pilots also stall when teams treat the first working agent as the finish line and it falls behind as rules and products change.

Can AI agents clear compliance alerts on their own?

Few teams allow it today, even for obvious sanctions and false positives. A common approach is for the agent to gather evidence, recommend a disposition with its reasoning, and rank alerts by confidence, while an analyst makes the final decision. The BSA team stays responsible for the investigation and the SAR decision.