Why Fraud Rules Stop Working (and How to Tell When Yours Have)

Based on real attack patterns our analysts investigated, with details changed.

A few weeks ago I was going through a lender's fraud rules, and one of them looked perfectly healthy. No alerts. No complaints from the review team. It had been written to stop a wave of automated loan applications, and from the dashboard, it looked like it had done its job.

It wasn't catching anything at all.

So why do fraud rules stop working? Most of the time, the attacker changed one of the signals the rule depends on. A rule that needs a fixed set of conditions to show up together can't fire if one of them goes missing. 

That's why a quiet rule can mean two very different things. Either the attack stopped, or the rule did.

TL;DR

• A lender's fraud rule stopped catching an automated application attack because the attacker dropped one of the three signals the rule needed. Across roughly 290,000 evaluations, it matched zero.

• Nobody noticed, because dashboards show what rules catch, not what they miss.

• Loosening the rule wasn't the answer. It would have pulled a lot of real customers into review or step-up.

• What caught the attack was the raw device and behavior data, not the standard risk signals.

• To protect yourself, alert on rules that go quiet, check each condition of a rule on its own, and come at every attack with more than one rule.

What happened

Earlier in the summer, this lender was hit with automated loan applications. The team wrote a rule that needed three signals at the same time: a specific browser setup, an automation flag, and a network error. It was a reasonable rule, and it fit the attack it was written for.

Then the same operation came back, and a few things had changed:

• They used a modified browser running on real Windows computers.

• Every application came through its own residential proxy connection.

• The identity details were generated and pasted into the form, not typed.

Over about four weeks, they sent roughly 1,650 applications. This was bot fraud, but not the clumsy kind.

The key change was that the attacker had stopped triggering one of the three signals. The other two signals still showed up on roughly 97% or more of the attack traffic. The third showed up on about 1%. Because the rule needed all three, it never completed. Across roughly 290,000 evaluations, it matched zero.

One thing to be clear on: none of these applications made it through to the final approval phase. This isn't a story about money walking out the door, but rather one layer of defense going quiet while everyone assumed it was still working.

Why did the rule stop firing?

This is the basic weakness of rule-based fraud detection. Think of the rule as "A and B and C." All three have to show up. If C disappears, A and B can be on every single application and the rule still won't fire.

The obvious fix is to loosen it: drop the missing condition, or change it to "A or B." That usually makes things worse, because some signals on their own are just too common. Plenty of legitimate customers trip an automation flag too. For this lender's customer base, an "A or B" rule would have pushed a big chunk of real applicants into review. Loosen the rule and you trade a fraud problem for a friction problem.

So you end up stuck between two bad options. Tight rules go stale. Loose rules flood the queue.

Why didn't anyone notice?

Nobody missed this because they weren't paying attention. A few things made it hard to see:

• No alerts looks like good news. Most fraud rules engines report what fires, and dashboards show what rules catch, not what they miss.

• Every device was brand new on the day it applied, so there was no history to build on.

• Every application came from its own connection, so velocity checks had nothing to count.

• The identity details looked like real people.

What should you do when a fraud rule stops firing?

Rules always react to an attack pattern. So when a rule stops firing, it's usually one of two things. Either the pattern it was built for isn't around anymore (kind of like banks still having security guards even though old-school robberies barely happen), or the attacker changed something.

To figure out which, I break the rule down. I check what's still flagging and what isn't, and I compare that against the attack signature. In this case, two of the three conditions were still lighting up on almost all of the attack traffic. That told me the attack hadn't gone anywhere. It had just stepped around one condition.

That's also why I always suggest more than one rule. Come after the fraud signature from multiple angles, so if the fraudster changes one thing, the others still catch them.

What caught it

The standard risk signals weren't much help here. Only one of them still fired at scale, and it fires on plenty of normal traffic too, so it couldn't separate this attack on its own.

What told the story was Oscilar's raw device and behavior data underneath. When we compared the attack traffic to normal applications, a few things stood out that a real browser on a real computer just doesn't produce:

• A component that every standard desktop Chrome install reports was missing.

• The browser window was the same size on almost every application, even though the screens behind them varied a lot in size.

• The cursor moved in evenly spaced, machine-like steps, not the way a person moves a mouse.

• In just one sample we pulled, the exact same sequence of clicks and keystrokes showed up on 285 separate applications. No two people fill out a form the same way. A script does it identically every time.

The new rule keys on one condition a normal Chrome install never produces. We tested it against about 75,000+ applications over roughly 3 months, and it matched nothing outside the attack. It also runs at the very start of the application, before any identity data is even submitted.

How do you know if a fraud rule is still working?

You don't need a big project to check. A few questions go a long way:

• When did each of your rules last fire? A rule that suddenly went quiet deserves a look.

• Do you get alerted when a rule's fire rate drops? Most teams set alerts for rules that fire too much. Almost nobody alerts when a rule goes quiet.

• For rules that need several conditions at once, how often does each condition fire on its own? If one has dropped to near zero, the rule can't fire.

• Are any quiet rules sitting next to rising approval volume in one segment? That combination is worth digging into.

• Have you re-tested older rules against recent attacks, not just the ones they were written for?

How do you build fraud rules that don't go stale?

No rule lasts forever. Attackers adapt; that's their whole job. But a few habits help rules hold up longer:

• Use more than one rule against the same attack, from different angles, so one change doesn't take out your whole defense.

• Combine device, network, and behavior signals, and monitor each piece, not just the combined rule.

• Key on behavior rather than addresses or networks. Those rotate constantly.

• Measure the false-positive cost before a rule ships.

• Watch new rules for a while before you let them block anything.

When the standard signals stop working, raw device and behavior data tells the story. 

That's what caught this attack, and it's the layer Oscilar's device intelligence is built around.

We also treat monitoring as a whole-system job, not a rule-by-rule one. That means watching for rules that go quiet, not just rules that get noisy. It also means testing new rules against historical traffic before they go live, so you know what they'll catch and what they'll cost in friction before a single real customer sees them.

Want to see what your own traffic looks like at that level? Try our fraud scanner.