Tips for Managing Bot Traffic

In the past year, Tenzing has seen a large growth in bot traffic to many of our customer environments, sometimes causing performance issues. To help our customers, our teams have developed a number of tactics to manage bot traffic and mitigate its impact.

But first;

What are bots?

A bot (short for “robot”) is a program that operates as an agent for a user or another program, or simulates a human activity. There are lots of different types of bots, some malicious and some not.

Good Bots are from well known companies like Google, Pinterest, Yahoo, and Bing. These bots collect information from your website to power their service. They are crucial to your online presence and search rankings.

Bad Bots, on the other hand, include comment spammers, SQL Injection worms, vulnerability scanners and more.

But why are bots a problem?

Bot traffic—including scrapers, hackers, spammers, impersonators—has been estimated to be as high as 61 percent of all internet traffic. We have seen levels from 30% to 90% in some of our client environments.

‘Bad’ bots can steal data or even take a site down. In fact, more than 95% of all website attacks are carried out by malicious bots. Even benign bots can cause problems by using precious system resources. If bot traffic is not taken into consideration when projecting traffic patterns the environment may be under scoped.

So what can be done about them?

There are several ways to manage bot traffic and its potential impact:

  • Monitor bot activity – if you don’t know it is happening you can’t mitigate against it.  Our team has designed a monitoring system to track bot activity over time.  We are currently looking at extending this functionality to include alerting when bot traffic exceeds a specific threshold.
  • Leverage a WAF (Web Application Firewall) to block bot traffic.  If the client has a CDN, most suppliers offer this functionality.
  • Serve different content to bots; you can display a less resource intensive site and protect your assets.
  • Deploy a separate server that just handles bot traffic, separating it from user traffic. This means if bot traffic is negatively impacting the site, your users are not impacted because they are on another server.
  • Lower the web server session limits for bots.  For example, if the time out value for a session is 30 minutes, set the session limit to 5 minutes for a bot, terminating the sessions faster.
  • Scale back Google’s crawl rate.

If your site is experiencing bot traffic, hopefully the above tips will help. If your managed service provider that doesn’t help you manage bot traffic contact us today. We specialize in supporting ecommerce retailers and work with them to ensure performance, security and scalability.

Aisling McCaffrey

Demand Marketing Specialist at Thinkwrap
Aisling is our Demand Marketing Specialist at Thinkwrap, and loves working with both technology and humans. She studied International Business (concentrating in Marketing) and has spent several years living and working in China, mostly in Shanghai, where she became passionate about global innovation and how the use of social media changes in different cultures. Aisling likes to keep up on internet trends - from business to memes - and is always looking for new ways to learn or entertain herself.