Referrers, Bots, and Search Engine Ranking, Oh My.
Pinterest isn’t the only referrer that our ecommerce clients are talking about, referral spam is growing and we are working with many retailers to protect their sites and rankings. Barry Reddy, one of our Senior Linux Administrators is here today to explain what referral spam is, what it does and what to do about it.
First things first, what is a referrer?
A web server typically logs a set of information for each browsing session, storing the address the user is browsing from, the date and time, a record of the page requested, information about the browser and operating system as well as the site that the browsing user arrived from when requesting that page. That last one, that’s what we call the referrer – by checking the referrer, the new webpage can see where the request originated.
A referrer is a useful part of website analytics, since it helps with understanding where your website traffic is originating, which in turn becomes valuable marketing data. Knowing where your visitors are coming from might give your marketing team additional insight into their customers and may even determine where they spend marketing budget.
What is referral spam?
Referral spam is something that occurs when malicious webmasters try to promote websites by creating a web crawler, which behaves like google, or bing’s search bots. In the case of google, bing, and other search engines, these search engines use bots that comb through websites, and update search indexes.
In the case of referral spam, these bots send fake referral information to any websites they visit and replace the referring site information, with bogus site urls, linking to sites that often contain explicit or unrelated website content. The purpose of this referral spam may be to drive or leech traffic from busy successful web sites, and possibly to benefit from any positive search engine ranking that highly trafficked website may have garnered.
Why should I care?
Over time, referral spam can negatively affect your search engine rankings. It can also negatively impact your site traffic analytics and indirectly promote suspicious or nefarious sites. The resulting bot traffic also negatively skews log file statistics, which can be the basis of business and advertising spend not to mention the additional performance load and infrastructure costs from the bots.
So what can I do?
There are a few techniques for protecting your web sites against referral spam.
- Configure your referrer log data so that it is not indexable by search engines.
- Use access rules, like .htaccess to block requests that don’t match expected traffic.
- Use of web application firewall technology to detect and filter suspicious traffic.
It may be worthwhile to periodically examine your website log analytics for patterns suggesting referral spam. Being aware of the problem can help protect your site and infrastructure from search engine ranking penalties and unwanted performance problems associated with the load caused by referral spam traffic.