AVG has obfuscated their user agent.  One of the current agents for customers of their free and paid tool now cloaks itself as IE6:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

In addition to the easily detectable user agents:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

User Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)  

User Agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

This news is not good.  If you filter SV1 agent, you risk filtering legitimate traffic from the IE6 browser.  A few folks have commented to me that one should filter the user agent anyway, because 1) IE6 is in decline and 2) most IE6 users have .NET installed, which will show in the user agent.  Still filtering it makes me a little uneasy.

Is this the death toll for log file analysis and services provided by ABCe (since they can’t filter this user agent either)?  Maybe it is.  AVG is touting that agent lacks HTTP Accept-Encoding, which is just dandy, but that information isn’t normally captured in logs.

So the current situation is this:

1. AVG has two user agents.  Both are filterable, but the SV1 agent is problematic to filter because you risk filtering legitimate traffic.
2. Both agents in the current version request gifs in noscript tags, inflating counts in page tag implementations with noscript configurations.  AVG claims they will fix this issue.
3. The bot uses”mad” bandwidth.  I’ve heard stories of bandwidth increasing 100x normal levels.  Some webmasters are serving dummy files to the recognizable user agents, some aren’t serving content to IE 6 browsers (crazy), and some are redirecting the bot back to AVG (thus inflating AVG’s bandwidth, LOL!).
4. Evidence points to this bot NOT inflating clicks from paid search (i.e. PPC) and thus NOT committing click fraud.   But it doesn’t remain out of the realm of possibility that the scanner may be accessing an ad vendor click redirector and causing a click.  Not trying to spread FUD here, just making a point. 
5. AVG is looking at option of checking either an external db (hosted by AVG) or a local cache to verify sites in SERP’s have been “scanned by AVG,” instead of repeatedly scanning sites every time they are listed in SERP, to reduce the bandwidth issue and minimize fraudulent entries in log files.
6. AVG is thinking about enabling white listing of sites, so they are skipped by the scanner.
7. AVG is thinking about exposing a meta-tag that instructs the scanner to ignore the site.

Good luck with this nasty bot!  Interestingly, here’s how you smurf a site with the AVG LinkScanner. 

Here’s the deal.  AVG LinkScanner doesn’t execute javascript nor take cookies.  I had that confirmed by the Chief Research Officer at AVG, Roger Thompson. 

So why is the AVG user agent showing up in that data collected from certain page tag configurations?  The AVG LinkScanner currently requests gifs in noscript tags!

A best practice in web analytic’s page tag configuration is to use the noscript tag to serve the gif to non-javascript executing browsers.  Here’s some commonly seen (obscured) code for doing that:

<noscript>
<div><img alt=”foo” id=”bar” width=”1″ height=”1″ src=”http://
foo.bar.com/xyzab57yw10000s1s8g0boozt_9t1x/foo.gif?baruri=/
nojavascript&xy.js=No&xy.tv=1.2.3″ mce_src=”http://
foo.bar.com/xyzab57yw10000s1s8g0boozt_9t1x/foo.gif?baruri=/
nojavascript&xy.js=No&xy.tv=1.2.3″div>
</noscript>

<NOSCRIPT>
<IMG
src=”//foo.bar.com/xyz.gif?Log=1&URL=/javascript_disabled” mce_src=”//foo.bar.com/xyz.gif?Log=1&URL=/javascript_disabled”
BORDER=”0″ WIDTH=”1″ HEIGHT=”1″ />
</NOSCRIPT>

<noscript>
<img src=http://pt.foobar.com/images/xyz.gif?js=0” height=”1″
width=”1″
border=”0″ hspace=”0″ vspace=”0″ alt=”"> 

Thus, if you are using noscript tags in your page tag *and* someone with the AVG Linkscanner views a SERP (search engine results page)  from Google/Yahoo/MSN that lists your site, the traffic from the LinkScanner will be counted. 

Of course the simple solution to fix this problem is to exclude the user agent: 

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

If don’t have full control over your page tag based web analytics implementation (i.e. hosted), you need to verify that your vendor has excluded this agent.   And you should have them audit your data going back to April, and refund/credit you any money.  Good luck with that though!

How big is the problem?  Well, it depends!

The amount of AVG traffic will vary dramatically by site.  Your site must show up in the SERP’s on computers of visitors that have AVG LinkScanner installed, and you must be using noscript tags to serve the gif.

I’ve made AVG aware of this issue.  And frankly, they’ve been a fantastic company to work with, so I’m sticking with them (for now ;).  First they allowed me to join a private Google group to discuss my findings, both the Head of Global Communications and Chief Research Officer quickly responded to all my emails (good social media response), and their engineers are looking into this issue so that they can fix it…  That’s pretty impressive and quick response.  So cheers to them!

It’s worth mentioning that the LinkScanner isn’t _supposed_ to request images, so I do think this issue will get fixed.

Only time will tell whether or not AVG obfuscates the user agent so it looks just like a “normal” browser.  Let’s hope not! 

What I do find interesting is that I’m already hearing that an agent exists with the string (Mozillia/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813). Note the “ia” mispelling of Mozilla as incorrectly documented here.  And it accepts cookies.  So AVG’s agent is already being spoofed.  Not good, not good.

The  well-researched answer is “no.”  The AVG LinkScanner Bot appears to prefetch the js and the gif (and pretty much everything else on the page), which for certain tools and their tag configurations generates false page views and visits (and the derivatives thereof), just like it’s “legitimate” traffic. 

If your tag configuration is set up with noscript tags, AVG will fetch the content in the tags, including the gif, which means that:

• The bot may be infesting the data of customers of web analytics vendor who configure page tag-based data collection in this way. 
• The bot may be inflating the data in such products/services offered by various web analytics companies.
• Customers may be paying for server calls generated by this bot.

Vendors, of course, could easily filter the user agent to protect their customers:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813) 

But I haven’t heard a peep from any SaaS vendors about excluding the user agent, filtering already collected data, or refunding customers the cost of robotically generated server calls (regardless of AVG). Have you?

Think about this: many SaaS page tag vendors don’t provide detailed visitor-level data and user agent reporting.  That means that their customers have no ability to investigate this bot or detect it by filtering their reported data by the the true user agent.

I’ve been talking about JS executing bots screwing with web data for about a year now.  SEOMoz and the folks at SlickSurface confirmed it quite recently (quoting me no less in their fantastic analysis).  So they do exist…

Now let me tell you a little story.  Once upon a time I was at a conference called eMetrics when the CEO of a company came up to me and said “hey I read your blog about bot detection, and I looked in my web metrics tool for traffic with high page view to visit ratios.”  Then he narrated a story to me about how he found a bunch of traffic that had page view to visit ratios of 5,000 to 1.”  I said “do you use page tags” He said “that’s all my vendor provides, so yeah.”  And I said “you’ve found a javascript executing bot in your data.”  “I know” he said. “Well did you call your vendor and let them know?”  I said.  Now for the punch line:  he told me that the vendor (who shall remain nameless) told him “well, the traffic executed server calls”  And they wouldn’t give him a refund!

It’s worth mentioning that this bot definitely affects log file tools and packet sniffer tools.  Both must be configured to filter the AVG LinkScanner user agent.

Now here’s the rub for me.  I use AVG!!!  But I now find it increasingly difficult to support the company or continue using their products.  Why?  Because they are wearing a “bad hat” here:

• First, they are fully aware of the affect of this bot on web analytics systems. They just don’t seem to care (yet).  UPDATE:  They have set up a Google Group to discuss this issue.  They must understand how companies of all types in all sectors use web analytics data to optimize their sites, set their marketing budgets, determine expected server load, and much more.  What do their Internet Marketers think? 
• Second, the Link Scanner tool may have a short shelf life and may offer limited protection.  Malware creators will easily adjust. Check out what my friend Steve McInerney, a very smart security expert, said on the Web Analytics Association’s Yahoo Forum:

What strikes me about this particular solution by AVG is how
incredibly … stupid it is on several fronts.

1. Noticeably impacting a users bandwidth is, technically, a security
breach in the first place, aka Denial of Service Attack.

2. Some of us live in countries that have rather severe bandwidth
charges/limits and the like, whom shall I send my excess bandwidth
bill to?

…(this) method is fundamentally
flawed. ie malware ignores any first request and only infects on a
second request - alternate cloaking. Whatever. This type of “solution”
only provides weak protection for a strictly limited period of time.

…not just “no security” but bad
security. Because folk feel they are being protected when they are
not, and hence will take greater risks and hence inflict greater harm
on themselves.  

Ignoring the balance of positive to harm that this problem inflicts on
the users who use this product.

• Third, AVG just doesn’t seem to “get it” yet.  They are potentially messing with the ability to drive commerce via data driven decision making, e-commerce analytics, site optimization, and online media measurement!  To quote The Register “chief of research Roger Thompson - who designed the AVG LinkScanner - indicated he may do away with that unique user agent. His chief concern is security, and he doesn’t want webmasters or malware writers gaming his scanner. “In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being,” he argues.

WebMasterWorld has some good stuff about to say here.  Read the Register’s first article here.  And check out the dude’s blog who broke the news first and responses from AVG here and here.

Interesting stuff. So what do you all think? Have you seen evidence of this bot in user agent data from your page tag solutions that use the noscript tag for the image? 

Mobile analytics for Internet-enabled wireless devices is a fairly hot topic for companies seeking to acquire customers, extend their brand, or expose content in “innovative” ways.  Obviously, the iPhone and Blackberry are pushing development in this area forward, but there really aren’t a lot of players in this space. 

Nedstat, CoreMetrics, and Omniture offer capabilities mixed into their current offerings.  Nedstat even carves out some mobile specific reporting.  You can gain some insight into mobile activity from companies that enable log file processing, like Unica and WebTrends, but be prepared to configure a bunch of filters to isolate the data.

Lesser known companies pushing mobile offerings include: Amethon, Mobilytics, Bango, TigTags, Xiti, and AdMob.  Some of these mobile players are even offering capabilities where they cross-sell analytics as an integrated part of their ad networks, content delivery  and transactional processing systems, marketing and barcoding services, and even as infrastructure or network appliances.

On the audience measurement side, we’ve seen comScore acquire M:Metrics, which was no surprise to me.

On the multivariate testing side, we see my friends at SiteSpect offering mobile MVT testing capabilities. 

And I’ll bet we see Google get into this space within the next 6 months…  I’d even wager an announcement at eMetrics DC…

From what I can gather, when we’re talking about “mobile analytics” we’re talking about “mobile browser” activity across a variety of handsets, not everything that happens on the device. 

Measurement issues in this area include:

• Data Collection.  As many of you know, not all mobile browsers will execute javascript.  They cached the imagesThus, vendors offer us choices.  Folks like Mobilytics and Bango use an image-based data collection method, while Amethon offers a packet sniffer (they call it wireline detection), and we even have Omniture and Coremetrics talking about “no tag” implementations - what my good friend Phil Kemelor mentioned on his CMS Watch blog (”To compensate, you need to stuff the image tag with query strings that will collect the data you require for reporting.”)  Then we have Unica and WebTrends with log files.  Interestingly, packet sniffing has some advantages here because some devices pass unique id’s (such as the phone number) in the HTTP header or other unique id’s.
• Unique visitor identification due to lack of cookie support and IP addresses changing.  IP addresses change, I’m told, as they switch from tower to tower.   In addition many mobile devices will take the IP address of the gateway, making all the devices look the same “person.”  I’ve certainly seen evidence of the host changing pretty quickly during a mobile session. Compounding the difficulty in assessing “uniqueness” is that not all mobile devices support cookies.  In web analytics, cookies are used to define uniqueness.  The fallback method when you can’t use a cookie is IP address/user agent.  If you can’t set cookies and the IP address and user agents are the same, how do you identify uniqueness?   However, when you can detect a unique value in the header, you can easily detect uniqueness.
• Handset capability detection.  Does the device support WAP pushing, streaming video, ringtones, downloading video clips, and so on?
• Phone and Manufacturer identification.  Database from WURFL and DeviceAtlas can be used to identify phone and manufacturer device attributes.  Larger vendors are further behind on integrating this data into their current offerings, whereas the smaller niche players are making use of it. 
• Screen resolution detection.  The Mobile Marketing Association’s (MMA) standards for the four “standard” screen sizes may carry enough weight to push this disdained piece of metrics trivia available from javascript based tagging in web analytics into a brighter spotlight.
• Traffic source detection.  Capabilities for traffic sources seem rudimentary.  I don’t just want to know about search and direct entry.  But I want detection of sources from my marketing and advertising campaigns, rss feeds, and email newsletters, if mobile visitors are coming in from those channels.   Interestingly, Bango solves the campaign tracking issue by pushing you to a Bango-specific URL.
• Geographic identification.  Where are the visitors viewing your site coming from?  And what does the mobile audience environment “look like” in each country.  From this information you can extrapolate country-specifics for site optimization.  But not all devices enable geographic detection because the gateway’s IP address is used or the IP address from the network is used, not a GPS signal.
• No standards.  There are few, if any, commonly supported mobile standards and no web data standards, so the problem is no standards for the devices and no standards for the tools.  There are no standards.  Did I mention that there are no standards. 

So I was thinking, what would I want to see in a mobile analytics solution?  Allow me to riff here.

• Dashboards for KPI and specific-metric reporting.  Views, visits, visitors, referrers, popular pages, traffic sources, resolutions, geography, time-based reporting and custom defined KPI’s….
• Support for multiple data collection methods.  Logs, no-js image tags , and packet sniffers.  Let me pick what I need for whatever application fits my goals.
• Support for mobile-specific constructs not present in historic web analytics data.  Manufacturers, operators, handsets, and device capabilities.
• Advertising-based reports.  CTR, CPM, eCPM, that stuff…
• Tracking for mobile downloads, installed applications, SMS, and MMS.  Seems like a no-brainer.
• API’s.  Closed systems are dead ends for integrated marketing, so give me an API or enable pre-built integrations with other systems, like CRM.
• Segmentation.  By country, by device, by network, by manufacturer, and so on.  It’s necessary.
• Repeat or return visitor identification.  Simple measures of recency and frequency, core to media buying and planning and to site optimization, should be a data point available in mobile analytics.
• Conversion and goal metrics.  Do visitors on mobile devices convert better, worse, the same?  Do they reach site goals?  Without tying performance data  and outcomes to mobile visitor activity, I’m left wondering…
• Value scoring for engagement or proxy scoring for revenue and ROI analysis.  I want to be able to score attributes or actions to approximate an engagement score or to identify value or indicate revenue. 
• Non-human traffic and web-browser based detection and reporting.  Mobile pages are full of links.  The ads are links.  Mobile vendors must support detecting, filtering, and reporting, non human and web-based agents from pure mobile agents - otherwise the mobile data gets muddled and skewed.
• Data Export.  Must be able to export reports to Excel or Word, and email them.

So there’s a quick blogviation on Mobile.  Am I right, wrong, what did I miss?  Let me know…

A question any practitioner of Internet-based analytics will be asked by many different stakeholders is “why don’t the numbers match?”  Counts of the identically named metrics from ad servers don’t match the web analytics tool, which don’t match the for-pay third party audience measurement tools, which don’t match the free audience measurement tools, which never match any of the homegrown internal measurement tools.  And none of them ever match each other.

So it’s a good question certainly valid to ask.  The answers are even fairly easy to understand, but the root causes are often difficult to pinpoint and even harder, if possible at all, to remedy.  The fact of the matter is that data discrepancies in analytics result for a multitude of reasons, such as:

• Different data collection methods.  We have a bunch of tools and services that collect web data using various, non-standardized, proprietary data collection methods.  Ad servers use javascript page tags.  Many web analytics tools use page tags too, but it’s not uncommon in web analytics to use additional methods, such as log files or packet sniffers.  Or perhaps a combination of these methods, called hybrid data collection.  And all the tools have different algorithms for processing the data collected.

On the audience measurement side, data is collected from self-selecting panels who install proprietary software (i.e. toolbars and so on) on their computers, perhaps at work or at their university, but most likely at home.  Then, the collected data from different panels is rolled-up and combined, and the limited subset of the Internet population that chooses to be monitored, in exchange for some incentive, is inflated and projected to the entire Internet audience using proprietary statistical methods.  We also have data collected from a limited set of geographically specific ISP’s.  And regardless of whether we’re talking about audience measurement or web analytics, the different data collection methods often, but not always, involve cookies and all their inherent issues of cookie deletion.  

• Unique data models.  Ad servers aren’t focused on counting page views and the other dimension of web analytics (visits, time, and so on).  Rather ad servers focus on serving and counting impressions served (and loads of related derivative calculations, like CTR, CPC, and view–thru).  Metrics are based on an ad request and an ad code.  Ads may or may not be targeted to a page, and instead to various constructs, like a “zone” or “keyword.”  What that means is that the “page” dimension may not even exist in your ad server’s data model.  In other words, you aren’t looking at impressions measured on a page, but rather at the number of impressions served in a different conceptual construct.  That’s one of the reasons why people say metrics and ad-serving systems “don’t measure the same thing.”
• Untagged pages.  Specific to technologies that collect data or serve ads using javascript page tags, there are challenges to ensuring and verifying complete coverage of page tags across every page on a site.  When the pages aren’t all tagged with the different tags for the assorted technologies, guess what?  The numbers won’t come close to falling within tolerable variances.  And questions and skepticism will ensue.
• Non-JS executing clients and ad blocking software.  Let’s imagine for the moment, your site is perfectly tagged for all technologies, so the numbers between your ad server will be close to your web analytics system, right?  Nope, regardless of data model issues, not all browsers execute javascript and many Firefox users have installed Ad Block Plus. 
• Cookie issues.  When you’re counting based on cookies, third-party cookies get blocked (often by privacy software).  Many ad servers and web analytics tools still serve third party cookies, and many corporations have not tricked out their DNS to accommodate this issue.  And we all know how cookie deletion affects unique visitor counts, even if you use first-party cookies.
• Many other issues.  Latency from visitors moving off the page prior to the tag executing to latency in the call to pick up an ad from a third party while your ad server counts the traffic (so your ad count differs from the agency’s count), to refresh rates making it hard to correlate page views and impressions, to no rich media installed and no fallback, to robotic traffic not being filtered from logs or tags, to certain types of user agents (such as mobile devices) not executing javascript… there’s a whole host of other factors that cause data discrepancies.

And of course, there’s always the nebulous issue around the complete lack of consensus-based, enforceable standards for online measurement.  No industry organization can say what vendors or companies “must” do, only what they “should” do… And no industry body is going to get successful companies to change their secret sauce just because they said so…

So what’s a practitioner to do?  Understand the potential sources of discrepancies.  Work with your team (from IT to vendors) to prevent and minimize the root causes when possible.  Educate your team when discrepancies are not remediable.  Ensure you use the different sources of metrics judiciously in the context of your business goals.  Finally, realize that none of the tools are more “correct” than any other.  All of our analytics tools serve different, and sometimes overlapping, business purposes - from counting ads, to influencing media buying, to sizing audiences, to measuring business performance, and to optimizing the site.