*cough*. Sorry. Couldn’t resist really. Probably should have. Too many flashbacks to Wizard of Oz.

Actually wasn’t awffull, was one of the others: “visitors”. The problem I had was logs that had Apache mod_usertrack cookies embedded to betterer identify uniques. But no tools existed to actually make use of that - well that I had access to. I’d written a pretty nasty perl script. Slow. Overly complicated.
So I took that base and re-wrote in C and that became “visitors”.

Most of the code is wrapper. The core logic is quite small.
Basically check to see if we get more than one bite at a given cookie. See it twice or more, then it’s probably good.
I can take you through the detail if you’re really bored. Or: Read the Source.
I use comments, it’s not … too scary.

But part of making “visitors” work, was *really* understanding how people and browsers use a/the site. It’s not perfect, but I am seeing a remarkable similarity between it’s numbers and GA’s (we switched on a few weeks ago). Which did wonders for my ego.

I haven’t, yet, pulled this logic or code into awffull. Intend to. Looking more like being sooner than later, but too much crufty old code that needs trimming and cleaning first.

Cheers!

Peter: I agree this is a huge problem for accuracy and analysis. Thank you for sharing your opinion, insights, and learnings. Your tip is right on regarding images, and that resource is very useful. Keep up the excellent work on OWA! As a major proponent of open source, I think what you’re doing is fantastic and gaining gravity.

Steve: I hear you about the “vibe,” but I think that would depend on the type of user needs that the site satisfies. For example, content sites that are aggregated and scraped because they are juicy with good content can see bot traffic in excess of 70% in personal observations - that affects the pattern, if you will… but only if you don’t filter…

Good advice on the headers too. It echoes Peter’s recommendation from above too… Hard earned learning from your work on AWFFULL (http://www.stedee.id.au/awffull).

The “bot’s intent” a counterpoint to the “searcher’s intent” for the modern web, ay?

Mat: Good points. One of the reason why I use the tools I do for web analytics is that SEOers need to know how relevant searchbots crawl sites. It’s amazingly helpful information.

I also agree that value may be generated simply from excluding the traffic and not reporting on it! Exclusionary rules are powerful. Indeed.

Thanks to all of you for commenting and reading my blog!

It has been my experience that bot traffic is fairly useful stuff when it comes to search-bid analysis, identifying which companies to spend your SEO dollars with, etc… so I would want to separate out ACTUAL bot traffic to know who is crawling my site the most (and ergo has the best, latest, deepest knowledge of my content) where I’d rather just forget about MSIE altogether and use an exclusionary rule to keep a few records from being written to my database.

Several thoughts:
* I find robots inflate my numbers but rarely change the … pattern or vibe if you will. Do get unusual spikes, but generally they just increase numbers.
* Bots that hit with regularity are usually “is the Internet Alive?” monitoring type tools. If they didn’t ask upfront, I block them at the firewall.
* I have a funky AWK script/regular expression, that does a fair job. Been meaning to wrap it in a proper perl script with a “bot per line” config/include file. Time… need more time…

Jacques: We have 4500 people that come from the same IP address at work. I’ve worked in places where that number gets into the high 10’s of thousands. All from a single IP address. Be careful of your assumptions. I usually cross check off cookies stored in the logs - and if the cookies match on images and pages.

Which incidentally Judah - is the biggest giveaway I’ve found. Bots don’t request images/CSS etc as much as people do. Some tricky HTTP headers to force browsers to reload, say certain images or CSS, can go a long way towards helping identify bots - who won’t 304 the image or css.

Fortunately it’s minor, but “Intent” of the ‘bot can matter. If I use MSIE to spider and archive a copy of your site, is that a bot? If I use wget to achieve the same thing? Tricky…

Cheers!

Also, Gary Keith’s Browscap project (http://browsers.garykeith.com/) is a must have component of any web analytics / spider detection effort.