Automation vs. Manual Review for Holiday Fraud Prevention: Preparation and Outsourcing Tips | CNP Feature

Automation vs. Manual Review for Holiday Fraud Prevention: Preparation and Outsourcing Tips | CNP Feature

October 31, 2018

By Courtney Fahrer, Founder, OpsTales

A beautiful bundle of fraud products have emerged over the recent years, with a seemingly simple goal in mind: automate fraud prevention.

Automated systems track many data points to generate a score denoting the riskiness of each user or action. Merchants set thresholds so that users with a score higher than “x” are considered bad, and those with a score lower than “x” are good. With systems spitting out scores based on real user data, fully automating fraud prevention should be somewhat simple, right?

Not yet. Ultimately, the best fraud prevention systems are a combination of automation and manual review.

Drawbacks of Pure Automation in Fraud Prevention

While the current state of fraud prevention automation is advanced, there are three factors that a purely automated system cannot yet account for:

  1. Control of your decline rate
  2. Temporary and dramatic changes in user behavior
  3. Grey-Area Users  

Control of your decline rate is an important piece of the equation. Giving an automated system the power to reject potentially good orders can be scary. While the Machine Learning (ML) models used for automation are a superb development, with seasonality there are temporary and dramatic shifts in user behavior that can result in them declining good orders. Let’s dig into an example:

During Christmas, your average order value might increase significantly, shipping locations can change, distances between location and delivery sometimes grow, new credit cards are added to accounts, and IP addresses can change.

A user that normally ranks low in riskiness will immediately be considered risky given these common holiday behavioral changes. An entirely automated ML model can take time to “learn” and account for these seasonal changes, causing the system to fluctuate, lose short-term trustworthiness, and ultimately, to reject perfectly good orders.

Additionally, there is a category that a purely automated system cannot protect against: Grey-Area Users. These are users that are close to the decline mark, but not quite there. As we improve our methods, fraudsters continuously do the same. This often means that fraudsters figure out pieces of the riskiness calculus and are able to strategically avoid those triggers. In a purely automated system, this allows fraudulent orders to go undetected.

So, while it is safe to say that there is a line of definitely good—with the exception of friendly fraud—the line for obviously bad is a bit blurry.

Let’s think about a scale of 100.

  1. Users with a score of >89 = fraud without a doubt
  2. Users with a score of <20 = definitely good
  3. Users 75 89 = Grey-Area User with some suspicious characteristics

We cannot flag this last category as obviously good or bad. In many automated systems, it would pass. In a complex fraud prevention system, an order from such a user would get flagged for additional verification through manual review.

Pure automation can reduce your malicious fraud, but not to the fullest extent. Depending on your business, these grey-area users can be a significant driver of your overall chargeback rate. To optimize for a minimum chargeback rate, your system must have a manual review component.

Why Manual Review?

Manual review is the process during which extra verification takes place and often involves a person reviewing the case. It’s about learning how to distinguish key characteristics associated with good and bad users, and applying that experience to specific cases in deciding whether to allow a purchase or action on your platform. It’s about training your eye, and your gut, to assess gray-area situations in ways a computer cannot.

This all boils down to the fact that manual review is here to stay for at least the short-term future. As we approach the holidays, the next step is to focus on what you can do to optimize your manual review process during this upcoming holiday season.

Manual Review during the Busy Season

Seasonality and fraud aren’t the best of friends. Seasonality blurs the lines between good and bad customers, all while playing host to a massive spike in fraud. This triggers an increase in Grey-Area Customers and a need for more human eyes assessing these cases. So what do you do? This is the time when manual review becomes critical, yet also grows in volume significantly.

Whether you’re just getting started or already have a process in place, here are some things to consider for the upcoming busy season:

  1. Start prepping for the holidays early
    1. Review historical data
    2. Assess how your process performed in recent years
    3. Isolate lessons learned & key characteristics of good and bad behavior
  2. Optimize your manual review process
    1. Compile a master list of good and bad signals
    2. Set up a way in which orders can be put into a “manual review queue”
    3. Make it quick and easy to find the data to review each user
    4. Recognize that your chargebacks will increase and that’s okay
    5. Build a training regime to reduce time spent on boarding new analysts
    6. Pro tips
      1. Timebox the review to around 5-10 minutes per user
      2. Communicating with customer via phone or email helps
  3. Simplify your process/strategy further into streamlined, repetitive pieces
  4. Hire an outsourced holiday staff

Once you’re confident in your optimized manual review process, the next important piece is to hire holiday staff—an expensive proposition. Holidays are the big boost to EOY sales and you don’t want to offset this by increasing costs. I get it.

A newer option in the fraud space is to outsource. You can hire a holiday team at <$10/hr that executes on the streamlined, repetitive tasks while your internal team focuses on the highly skilled, non-repetitive tasks. As long as you can simplify the process and build a strong training regimen, it’s possible to outsource this work.

How to Outsource Successfully

Outsourcing is an incredible resource that, when done correctly, can really help your business. It’s extremely low-cost, frees up time for your internal team to focus on key value-added work, and forces you to evaluate your current processes with optimization in mind.

There are some basic requirements, but the key to successful outsourcing is a clearly defined, simplified process broken out into tasks. The outsourced work should be repetitive and you should have a system in place to track quality.

Incorporating outsourcing in your fraud prevention strategy can make it easier to find holiday hires for a far lower price compared to onsite employees. It also forces you to break down the system into simplified data points and rules. The more complex they are, the more confusing it gets when faced with abnormal user behavior, and the longer it takes to review each case. This leads to a myriad of issues, like delayed shipments (yikes), higher chargeback rates, and unhappy customers. To outsource manual review, there are a few things you should consider implementing:

  1. Estimate the total manual review volume you expect, then estimate how many agents you need to add to your team
  2. Find an in-house person that can manage the outsourced team or go to an agency
  3. Follow the tips above for building a strong and simplified manual review process
  4. Make a checklist of everything the analyst should review. This reduces the training time for new hires because they have guides to refer to instead of memorizing the process
  5. Set a time limit for each user review
  6. Establish an escalation process for when an outsourced agent still isnt sure
  7. Actively monitor performance metrics like time spent, number reviewed, daily decline rate per agent

A successful manual review process and a successful outsourced team share a key factor: simplicity. Complexity begets confusion, so break things down wherever possible.

As you approach the holidays, get your fraud ducks in order so you can minimize loss and maximize profit as you close 2018.

Courtney Fahrer is the founder of OpsTales, an outsourcing agency based in San Francisco that enables companies to optimize staffing needs in chargeback management and fraud prevention, customer support and more. She has also served in several retail roles, including head of marketplace operations for online marketplace Wanelo.


  • Share this Article:
Courtney Fahrer