How to Evaluate an AI Risk Agent Before Production

Last updated: September 2026

The demo was good. That is usually the honest starting point, and it is worth saying plainly rather than treating the demo as a trick. Someone showed you an agent working an alert, it reasoned sensibly, it reached the conclusion your team would have reached, and it did it in a fraction of the time.

Now you have to decide whether it belongs in production, where you are accountable for what it decides. And the uncomfortable part is that a demo answers a question you did not really need answered. It shows the agent can be right once, on an alert someone chose. Production asks whether it stays right across your typologies, whether you can explain any single decision nine months later, and who answers when it is wrong.

There is a useful distinction underneath all of this: a model is monitored, an agent is evaluated. A model produces a score you can check against a label. An agent takes a sequence of steps — gathering evidence, calling tools, weighing what it found, reaching a disposition — and there is usually no label for any of it. What you have instead is what your analysts would have done. That turns out to be enough, and it is also what makes this evaluable by a risk team without a data science function.

TL;DR

• A demo proves one output. Production asks about sustained performance, reconstruction, governance, and operational fit.

• Evaluate an agent against the decisions your own analysts make, not against an abstract accuracy score. There is rarely a ground-truth label; there is a competent human judgment.

• Four dimensions: sustained performance, decision reconstruction, governance and sign-off, operational fit.

• The method is parallel running. Agent and analyst work the same queue; the comparison is the evaluation.

• You do not need an ML team. You need labelled human decisions, which are your closed cases.

• Know what to measure, then set your own thresholds.

• Evaluation does not end at go-live. Agents degrade for reasons models do not.

What AI agent evaluation is

AI agent evaluation is the practice of establishing whether an agent's decisions hold up over time against the judgments your own team would make, and whether each decision can be explained and reconstructed afterwards.

It differs from model validation in a specific way. Validating a model means checking its outputs against known-correct answers on a held-out set. An agent has no single output to check — it has a path. It decided which evidence to gather, decided that what it gathered was sufficient, and reached a disposition. A run can arrive at a defensible conclusion through reasoning you would not accept, and it can arrive at the wrong conclusion through reasoning that was sound until one input was missing.

The practical consequence: evaluating an agent you did not build is a different job from instrumenting one you did. Most published guidance addresses engineers building agents and choosing test harnesses. If you are a risk owner deciding whether to accept a vendor's agent into a regulated process, you need a different set of questions.

What a demo cannot show you

None of this means the demo was dishonest. It means a demo and a production decision are different objects.

The right-hand column is the evaluation. Each row is a dimension, and none of them can be answered in a meeting.

The four things to evaluate

Treat these as the questions to put to a vendor, and to your own team about your readiness to judge the answers.

1. Sustained performance. Does the agent agree with your analysts, segmented by typology, over a period rather than a session? Aggregate agreement hides the thing you care about: a single typology where the agent is systematically wrong. Ask how agreement is measured, on what volume, and whether you can see the disagreements individually.

2. Decision reconstruction. Can you retrieve, months later, what the agent saw and why it concluded what it did? Not a summary generated after the fact — the evidence as it stood, stored with the decision. It is worth seeing what an alert queue and an audit trail actually look like before you evaluate anyone's, so you know what you are asking for.

3. Governance and sign-off. Is there a defined operating boundary for the agent, a human approval point, and a record of both? Whatever your model risk function asks for when it validates a model, it will ask for here, and the artifacts are the ones above.

This is the dimension where the regulatory ground has just moved, and it moved in a direction that puts more weight on your evaluation rather than less. On April 17, 2026 the OCC, Federal Reserve and FDIC replaced the 2011 model risk letter — SR 11-7 — with revised guidance issued as SR 26-2, OCC Bulletin 2026-13 and FDIC FIL-15-2026. The new guidance places generative and agentic AI explicitly outside its scope, on the grounds that such models are "novel and rapidly evolving," and directs institutions to rely on their broader risk management and governance practices instead. The agencies also said they plan a joint request for information covering banks' use of AI. That RFI has not been issued. The guidance is aimed principally at banks above $30 billion in assets, but the expectations travel downward in practice.

Read that carefully, because it is easy to read as relief and it is not. Agentic AI being out of scope does not mean out of view. Examiners will still ask how you tested the agent, how you monitor it, and where human oversight sits. What has changed is that there is no published framework to answer them with — which means your evaluation record is the answer. An institution that ran a structured evaluation has something to show. One that relied on the vendor's demo has a conversation to have.

4. Operational fit. What does the analyst queue look like the week after this goes live? Who reviews the output, how does an analyst disagree with the agent, and where does that disagreement go? An agent that is accurate and operationally unworkable does not survive its first quarter.

Why the benchmark is your analysts, not an accuracy score

This is the reframing that makes the whole thing tractable, and it deserves its own section because it is the part people get stuck on.

For most risk decisions there is no ground truth. Whether a given alert should have been escalated is not a fact sitting in a database — it is a judgment, made by a trained person, against a policy. So the measurable question is not "was the agent correct?" It is "did the agent reach the same disposition as a competent analyst, and where it did not, why not?"

That has two consequences worth sitting with. The first is practical: you already have the evaluation data. Every closed case is a labelled human decision. You do not need to build a test set; you need to route the agent at the one you have.

The second is that you do not need an ML team to run this. Agreement rates and disagreement review are risk-operations work, not data science work. The team that already does quality control on analyst decisions can do quality control on agent decisions.

One honest caveat: your analysts are not infallible. Disagreement is a prompt to look, not proof the agent is wrong. Some of the most useful findings in these exercises are cases where the agent was right and the original disposition was not.

How to run the evaluation

Evaluation is not an event that happens before deployment. It is a period of parallel running.

Start against your history. Run the agent over a set of closed alerts where you already know what your team decided. This is cheap, it touches nothing in production, and it tells you whether a live trial is worth anyone's time. Oscilar structures this as the opening phase of a design partnership: data access and workflow scoping first, then evaluation against a labelled set of your own closed cases.

Then shadow mode. The agent works the live queue alongside your analysts without deciding anything. Your analyst makes their call as normal, then spends a couple of extra minutes seeing what the agent concluded and whether they agree. That overhead is real, and it is the evaluation. It runs for weeks, not months.

Then quality control on top. A QC process applied to the agent's decisions is itself a human-in-the-loop evaluation, and it is how confidence is built rather than asserted. Sampling agent decisions the way you sample analyst decisions gives you a running measure rather than a one-time verdict. Applying that QC procedure to every agent decision rather than a sample is an active direction of development rather than something available today; the appeal is that if every decision passes a consistent check, the human sample can shrink, because it is sampling the check rather than the raw output.

Then a readout, with a human decision at the end. Someone looks at the agreement rates, the disagreement patterns, and the reconstruction spot-checks, and decides what — if anything — routes to the agent for first-pass review. That decision belongs to a named person. Note what it is not: the agent does not post a disposition. The analyst's call stands and the maker-checker structure is unchanged. What changes is how much of the gathering and analysis arrives pre-assembled.

For a sense of what the far end looks like: a regional bank in deployment is seeing 92% agent agreement with its analyst team on alert disposition, with full human sign-off, and a 75% reduction in analyst review time. Agreement with analysts is exactly the metric this article argues for, which is why that number is the relevant one rather than a benchmark score.

What to measure

Categories, not targets. Anyone who hands you their pass mark is selling.

Set your own thresholds against your own risk appetite. The acceptance criteria for a proof of concept — what constitutes a pass — is a separate exercise and should be written down before the trial starts, not negotiated after the results come in.

A related trap, and a real one: a metric with no agreed definition underneath it is worse than no metric. If "agreement" is not defined precisely before the trial, the number you get at the end will not mean anything, and everyone will read it differently.

Common mistakes

Evaluating the final answer only. The disposition can be right while the path was wrong, which means it will be wrong on the next alert that differs slightly.

Evaluating once, before go-live, and never again. This is the most consequential one. An agent can degrade because your policies changed, your typologies moved, or its own instructions were edited — none of which touch the underlying model. Drift in agentic systems deserves its own treatment, and evaluation that stops at go-live will not catch it.

Measuring against a labelled set that is not your portfolio. A vendor's benchmark tells you about the vendor's data.

Accepting a metric with no definition. See above.

Treating a good demo as evidence of operational fit. They are unrelated properties. The demo says nothing about what happens to your queue.

Where the platform matters

One structural point, because it determines whether two of the four dimensions are even answerable.

Reconstruction and sustained measurement are properties of the layer underneath the agent, not of the agent itself. If the evidence an agent used was assembled from several systems at the moment of decision and never stored as a unit, you cannot reconstruct it later — you can only re-run it, which is not the same thing and will not satisfy anyone reviewing it.

This is why the decisioning layer matters to an evaluation that is nominally about agents. On a unified platform, the decision, its evidence, and its reasoning are stored together because there is one place for them to be stored. When you are evaluating any vendor, ask where the decision record lives, and whether it is one record.

If you want to see the agents this applies to, the Agent Hub is the place to start.

Frequently asked questions

What is AI agent evaluation?

AI agent evaluation is the practice of establishing whether an agent's decisions hold up over time against the judgments your own team would make, and whether each decision can be explained and reconstructed afterwards. It differs from model validation because an agent produces a path of reasoning and actions rather than a single scoreable output.

How do you evaluate agentic AI vendors for banking?

Across four dimensions: sustained agreement with your analysts segmented by typology, the ability to reconstruct any decision months later with the evidence as it stood, a defined operating boundary with human sign-off recorded, and operational fit with your analyst queue. Run it first against a labelled set of your own closed cases, then in shadow mode on the live queue, and set your own acceptance thresholds before the trial begins.

Why are traditional evaluation methods insufficient for agentic AI systems?

Traditional model validation compares an output to a known-correct label. An agent takes a sequence of steps with no single label — it chooses what evidence to gather, judges whether it is sufficient, and reaches a disposition. A defensible conclusion can be reached through unacceptable reasoning, so evaluating only the final answer misses most of what happened.

Can you evaluate an AI risk agent without a data science team?

Yes. Because the benchmark is agreement with your analysts' decisions rather than an abstract accuracy score, the evaluation data is your closed cases and the work is risk operations rather than data science. The team that already performs quality control on analyst decisions can perform it on agent decisions.

How long should an AI agent evaluation take?

Plan in months, not weeks. The run against your closed cases is quick. Shadow mode needs long enough to cover a representative spread of typologies and to show whether agreement is stable rather than a first impression, and a full design partnership from kickoff to live typically runs around twelve weeks. It does not end at go-live; agents need ongoing measurement because they degrade for reasons that never touch the model.

How are human review, auditability, and agent performance governed?

Through a defined operating boundary for the agent, a human approval point that forms part of the decision record, an audit trail storing evidence and reasoning with each decision, and continuing comparison against human decisions. Since April 2026 the agencies' revised model risk guidance excludes generative and agentic AI from its scope, so this sits under your broader risk management and governance framework rather than under model validation — which makes the evaluation record you build the primary evidence of oversight.