Let’s conclude this series on the spiders and bots of web analytics 2.0 with a framework for ranking the spider and and bot capability maturity of  a web analytics practice! 

Before I do so, let me mention a few things:

• How/if you detect spiders and bots depends on your data collection methods.  It is thought that page tags exclude spiders and bots because they don’t execute javascript or request images. Until they do. Log file data must be examined and filtered (either programmatically, manually, or both), and those filters need to be maintained by a web analytics professional.
• Bots are used for a many purposes.  From inflating metrics, to indexing text for search, to chatting, to spamming, to data mining, to site performance monitoring, to click fraud, and more.
• Bots are constantly evolving. Just like web analysis.
• Not all bots are bad news.  You want bots crawling your site.  Some bots are good and helpful for your online business. Imagine if you blocked Googlebot.  Segment the robotic traffic into separate reporting (in the best case) and make sure its filtered from your externally reported numbers.   

Now without further adieu, here’s a web analytics 2.0 framework for ranking your spider and and bot capability maturity:   

• Red.  You are in the red if the following applies:
  • I don’t understand the impact of spider and bot traffic nor do I measure, filter, or segment it at all. 
  • If I run an in-house solution, I haven’t updated my list of filters and exclusions since I deployed it. 
  • If I run a hosted solution, I rely solely on my vendor to control all aspects of bot filtering and data purification.
  • I don’t know when or how the Googlebot hits my site, but it sure does! 
  • I’ve never thought of reporting spider traffic for SEO.
  • I think the words “yahoo” and “slurp” refer to delicious soup.
  • Spiders scare me.
  • All bots are bad (no, they aren’t!)
• Yellow. You are moving out of the red zone, into the yellow if:
  • I occasionally look for suspicious traffic. 
  • If I run an in-house solution, I have occasionally updated my list of filters and exclusions since I deployed it. 
  • If I run a hosted solution, I rely solely on my vendor to control all aspects of bot filtering and data purification, but I ask for verifications of compliance with industry standards.
  • I’ve heard of the Googlebot and maybe I use or I am thinking about using Site Maps (do it!). 
  • I know about Yahoo! Slurp.
  • It would be cool and useful to report on spider and bot traffic and learn which bots are good and bad. 
• Blue.You are in the blue if at least every month or on an ad hoc basis:
  • I work with my vendor or in-house team to recognize and remove spider and bot traffic, conforming to industry standards. 
  • I do a monthly update of my filter and exclusions list, or I know my vendor does.
  • I capture bot traffic in my server log files, but I may or may not report on it.
  • I know all about the Googlebot, and I use Site Maps. 
  • I realize the Googlebot and Yahoo! Slurp behave differently on my site, but I don’t know exactly how.
  • I know which bots are good (like search bots) and bad (like content scrapers).
• Green.  You are in the green if:
  • I have established a process for regularly removing spider and bot traffic and for keeping my lists of exclusions and filters up-to-date, or I am *absolutely certain* my vendor has such processes.
  • I am in compliance with industry standards.
  • I measure spider and bot traffic and segment it into distinct reporting separate from my human traffic reporting.
  • My SEOers love me, the web analyst, and I am involved in educating people in my company about bot traffic from IT to managers to consumers of reporting and analysis.
  • I know that there’s more than one Googlebot, and I know how its crawl differs from Yahoo! Slurp (hint: Slurp visits more).
  • I know detailed metrics like the “total time online” of all my bots.

What I’ve covered in this four-part series only covers a little bit of what I know and what there is to know about spiders, bots, and crawlers and how they affect web analytics.   I’d enjoy hearing other experiences or opinions, so please share your comments if you feel like it.  Until next time, fine readers.  Thanks for visiting!

Add to the Conversation

Name:

Email:

Your email (required) will not be published.

Website:

Your Thoughts:

Please note that contributions are moderated and may take a little while to appear.