Bad Bots - An Evolving Threat
In 2019, every website operator should worry about bots. If you hold users’ identities, sell products, offer coupons or if the content on your site is your main asset, bots can abuse your website, target your commerce process and earn money from it.
Bot operators are improving their techniques constantly to fly under the radar of modern detection systems. Five years back, the common bot technique involved primitive engines hitting API endpoints at scale. Today we see bots that run browser engines, render content and mimic human behavior and interaction. Because of this sophistication, detecting bots has become a bigger challenge than ever before.
Bot Attacks Cost Billions
More businesses are going digital, driving online sales to new highs. Automated bot attacks are also cashing in, continuously innovating and avoiding detection. In 2018 account takeover (ATO) attacks alone resulted in $4 billion in losses.
This is about more than the bots that launch DDoS attacks. Today, there is a new generation of sophisticated bots that specialize in business logic abuse attacks like ATO, fake account creation, carding, gift card fraud, online ticket scalping and web scraping.
Not only do bot attacks hurt online revenue, competitive edge and brand reputation, they also drive up infrastructure costs and degrade site performance. Bot traffic skews web analytics throwing a wrench into business intelligence reporting as well.
Homegrown Bot Management
It Served its Purpose
Financially-motivated bot attacks started in 2014, and in the beginning, the problem was not clear to website owners. Web application firewalls (WAF) with coverage for OWASP top ten threats like DDOS and SQL injection were missing the attacks. The obvious answer was to build a bot detection solution in-house to take on the bot attacks. Several other factors also contributed to the decision to go with a homegrown solution.
First, as the website operator, the in-house team knows their website traffic inside-out. They know the legitimate users, the sources, the typical characteristics, user-interaction metrics and flow patterns. Second, the in-house team knows the business better than any external vendor. They define the most important KPIs to monitor, the most relevant threats and the use-cases to prioritize. Defining and tuning the appropriate rules and policies for bot detection and mitigation is hard, but manageable. Third, being part of the company affords them access to resources within the company to help address their challenges.
In addition, bots were likely costing the company millions of dollars in losses, justifying the investment in homegrown bot defenses.
Homegrown bot management solutions scored some early victories but they have since become an expensive and fruitless endeavor battling and losing to the ever-evolving bot threat.
Grow the Business or Fight Bot Attacks?
Cyberwar has always been an asymmetric war. Hackers require far fewer resources to launch successful attacks while effective defenses, especially homegrown ones, require exponentially more resources to stop them. Bots tip the scales in the hackers' favor.
Launching bot attacks is getting easier and more accessible. A developer of any skill level can access thousands of DIY guides online, build their own bot and launch it at high scale. And anyone can rent or buy off-the-shelf botnets. The billions of dollars in online transactions give plenty of motivation for bad actors. They generate profits that sustain a development community that provides new tools to stay relevant.
Signature-based Detection: Losing Efficacy
Homegrown solutions started with building signatures and pre-configured rules and policies, such as volumetric-based and geo-based detection. The efficacy of signature-based detection has declined rapidly over time.
The recent Hotelscombined sale day in Japan provides a good example. One of its local marketing teams in Japan planned a TV promotion and forgot to keep the security team in the loop about the promotion. The resulting traffic spike that came from real users within Japan could have accidentally been blocked by a simple volume-and geo-based homegrown bot detection solution.
Hyper-distributed Attacks: The New Normal
Signature-based systems have a hard time dealing with hyper-distributed bot attacks. These sophisticated bots impersonate many different devices on many different IPs, making them harder to detect than ever before. Hyper-distributed attacks are the new normal.
Hyper-distributed attacks first started as successful volumetric DDoS attacks. Attackers are learning from each other and doubling down on what works.
Behavior-based Bot Detection: The Need for Massively Parallel Computing
A behavior-based approach to bot detection can block hyper-distributed attacks. The behavioral approach relies on collecting multiple data and activity signals to analyze users' behaviors and identify anomalies.
Not all behavior-based methods are created equal. The number of signals collected and the size of the known dataset of good and bad bots determines the efficacy of behavior-based methods.
Effective behavioral-based detection requires a breadth of sensing capabilities, real-time tracking of numerous traffic data parameters, and a finely tuned massively parallel computing infrastructure. This infrastructure needs to be optimized specifically for real-time evaluation of large amounts of sensor data. Remember, bot detection should not slow down user sessions or page load times. Without such large-scale, real-time decision-making capabilities, keeping pace with the speed and evolution of today's attacks is nearly impossible.
These datasets have to be stored for historical reporting and forensic analysis. This data store can quickly bloat into many petabytes depending on the number of signals collected per user session, and since the costs of the data storage are usually overlooked, the total cost of ownership for a homegrown bot management solution is skewed, appearing lower than it actually is.
False-positives: Preserving User Experience
Effective mitigation of bad bots is critical, but no website operator can sacrifice user experience and potentially hurt conversion and revenues.
Blocking legitimate users would spell disaster.
Homegrown solutions take the easy route to tackle high false-positive rates. Limited detection capabilities require them to lower the detection threshold and resort to presenting CAPTCHAs or blocking real users. This low bot detection threshold approach results in degraded user experience, lost users and leaves the door open for more bad bots.
Top Apparel Brand noted, “With our homegrown solution, we are experiencing significant blocking and disruptions for legitimate users.”
Predicting Evolving Threats: Need for a Dedicated Team
As attackers evolve their approaches to bypass bot defenses, the security team needs to update their defenses and stay ahead.
To make behavior-based detection efficient, relying on the data is not enough. The security team should also proactively stay abreast of new methods, recent threats and trends. Effective security research must both pore over massive traffic data, but also should incorporate advanced research from multiple external resources, which are likely out of reach to the team. Leveraging security experts that understand how attackers think and predicting their next moves is critical.
For a homegrown solution, what happens if your lead bot management engineer that developed the solution leaves the company? Would you be able to find a replacement quickly? How long would it take for the new hire to fully understand the custom-built logic? How far behind would you be keeping up with new bot techniques as this process unfolds?
Homegrown Bot Management: Time to Reconsider
The minimum requirements to keep pace with evolving bot attacks include: 1) a dedicated bot security team, 2) extensive data collection, 3) behavior-based detection 4) real-time threat analysis, 5) highly optimized massively parallel compute and 6) crowd-sourced intelligence.
It is clear as day: a homegrown bot management solution is cost-prohibitive. If you are not one of the top five internet giants that can afford to invest millions in a home-grown solution, you have to make a decision. Fight bot attacks on your own or go with a behavior-based solution powered by machine learning and security experts.
Find out about the key criteria you should consider when choosing a bot management solution in our next blog.