Running Multiple Web Analytics Tools has Risks and Rewards…
Running more than one web analytics tool on a site or across a portfolio of sites is an increasingly common practice these days. The majority of the companies that run multiple tools probably run one “for pay” tool and at least one “for free” tool. Based on my experience, the cost of running two “for pay” solutions would be prohibitive for companies still trying to realize the “ROI” from web analytics (but it’s not unheard of in large, solvent, multinational companies). Not surprisingly, the most common “free tool” ran next to “enterprise-level” :-) tools like Omniture, Visual Sciences, and Unica NetInsight is Google Analytics. Data from my pal Eric Peterson’s Vendor Discovery Tool shows that GA and Visual Sciences code were found on 6% of tracked URL’s, GA and Omniture code on 4% of tracked URL’s, GA and WebTrends Hosted code on 4% of tracked URL’s. That’s great for Google and quite an edification of their excellent product!
I’m sure that there are companies running multiple “for free” tools, and/or running multiple big ticket tools (like HBX Analytics and Visual Sciences), and/or multiple homegrown tools built from Business Intelligence technologies and databases (Oracle/Cognos). Yet running multiple tools has risks and rewards.
Some of the risks of running more than one web analytics tools include:
- Lack of control over data. If you trying to foment a data driven culture, nothing could be more frustrating than someone outside of the web analytics team downloading a tag, linking it to their personal account, then questioning why X number in Y tool doesn’t match X number in Z tool. To promote adoption of new technology, running a competing tool has the potential to compromise data believability.
- Numbers not matching across tools. Different vendors “sessionize” differently so numbers will never be identical between tools. Dynamic sites, different underlying site technologies, and unique tool configurations mean numbers won’t match. Never ever. Check out Eric Enge’s highly-recommended 2007 Vendor Shootout. Run two tools and be prepared to answer questions about data discrepancies from those who consume reports on the same site from both tools.
- Conflicting vocabulary. Different tools use different terminology. One tool may use “sessions,” while another may use “visits.” Some tools talk about “views,” while others reference “page views.” Some tools use the term “unique visitors,” and other tools just talk about “visitors.” When you are rolling out “web analytics” to people who need to speak the same language, having multiple vocabularies for expressing the same or similar concepts confuses discussion and muddles actionability.
- Apples and Mangoes Comparisons. Some tools provide only snapshots of aggregated data, while other tools let you drill down, drill up, and slice and dice on detailed level data. Some tools enable you to add metrics on the fly to any report, and then filter and cross dimensions until your heart is content. Two people looking at two tools on the same data may conclude different actions are warranted based on the depth of their analysis. While a good manager can sort that out, it’s a bother.
- Potential to misallocate resources leading to needless redundancy. Companies have limited resources. If I need to apply tags to all my sites so that I can get to the real business of analysis, then why spend valuable time applying multiple sets of tags to enable tools that serve the same purpose.
- Licensing issues. Google Analytics or Quantcast account on a corporate site associated with a personal account whose owner would prefer not to give up the password.
- Training issues. When rolling out systems, training is necessary. Tools take time to learn. Why have resources learn multiple tools that do more or less than same thing when you got real business to take care of?
Some of the rewards from running more than one web analytics tool include:
- Comparative data. If you’re being charged by page views, it’s nice to have an alternate reference point to validate the charges.
- Differentiated reporting. Some tools are just better at custom and ad hoc reporting than others. If you have an inflexible tool not fully loaded with features then maybe it makes sense to get a tool that can do all that stuff a lot cheaper than paying for additional incremental features. Hello GA!
- Potential to enable a different level of integration. Lots of people tell me they download Google Analytics so that they can track their AdWords campaigns.
- Ability to leverage different features. Several major tools are technically and functionally challenged when it comes to simple things like showing the keywords used to drive traffic to the particular page, the number of unique visitors per page, or a bounce rate. Instead of dealing with complexities, sometimes it’s just easier to download and install a free solution like GA that does all of these things at no cost.
- Ability to leverage different data collection methods. Time-based metrics and file downloads are inordinately easier to measure and count using log file tools than using page tag tools, imho. Why fiddle with some esotericisms in tagging when you can just run the logs? Or better, yet, use a hybrid approach in one tool and get the best of both data collection methods.
Is it a good idea, as a site owner or manager of a web analytics team, to run more than one tool? The answer is it depends on your organization’s capability maturity for web analytics and how you balance risks and rewards.
The most mature companies have a centralized web analytics function. That means the company has one “master user” and “strategic owner” for web analytics and related technologies. The centralized web analytics function has its own resources dedicated to “doing web analytics.” Resources may come from other groups within a company, but, regardless, the company executives have identified and placed positional power around a “web analytics champion.” Since you’re reading my blog, you may be this person! Cool!
When you have centralization, you control key elements of doing web analytics:
- Measuring
- Reporting
- Analyzing
- Testing
- Evaluating outcomes
If you’ve centralized your web analytics team, you should select ONE web analytics tool as your Primary Web Analytics Tool.
Then I think it is then I think it’s safe to use more than one web analytics tool along these guidelines:
- Standardize on one tool as your primary tool. This tool should become the ”bible” for web analytics data at your company.
- Give people outside of the web analytics department access to the primary tool and ONLY the primary tool.
- Use the secondary tool within your web analytics team as a supplemental tool for comparing measurements, data reconcilation/verification, or analysis that you can’t accomplish with your primary tool (such as detailed roll-up reporting).
- Keep the overall enterprise standardized on the interface and numbers from your primary web analytics tool. That way you prevent confusion when reporting to stakeholders outside of the web analytics team.
- Do not provide access to the secondary tool to people outside of the web analytics team (unless the numbers match 100%! ;- ) ).
If you have a decentralized web analytics organization, I recommend that you:
- Standardize on a primary tool (whether free or paid). Remember Google Analytics is an awesome place to start!! And so it seems is Microsoft Gatineau!
- Work toward centralization before introducing another tool that has the potential to undermine current measures, reports, analysis, tests, and outcome evaluations.
On a final note, you may have multiple tools for reporting web analytics data (perhaps from companies like Business Objects, Cognos, Microsoft and so on). As long as the data is synchronized with the metrics from your primary web analytics tool, that’s fantastic!
Am I off-base? Absolutely right? Do you run two tools and love it, hate it, don’t care? What do you think?
Jean-Marc Vandenabeele added the following ...
Hi Judah,
I totally agree with what you’re saying here.
Especially the last paragraph about guidelines to run safely more than one tool.
I had to implement both a “for pay” tool and a “for free” tool for a big client, the “for free” tool was (guess it?)… GA. 
It went really well and still today, there is no issue to have both tools on the same site because only the “for pay” tool is the primary and official tool, the second one (GA) is only here to make a kind of benchmark against the official tool and only the Web Analytics team has access to it.
But don’t you think that the big majority of companies running a second free tool does that only to challenge the possibilities of their primary expensive tool? (with the secret hope that they might influence their favorite vendor to take into account their experience for future versions)
Judah added the following ...
Steve: Thanks for sharing information about your tools. Do you end up “going with” the GA numbers across your company, keeping the other three log-file based tools internal to your team? If not, how have your reporting consumers reconciled the mismatching numbers?
As for my book, that’s a great question. I’ve certainly thought about it, and would like to write a book sometime in the near future (next couple years)… We’ll see what the future holds… 
Jean-Marc: You bring up an interesting point. Are powerful free tools like GA causing customers of “for pay” tools to demand new and improved features? I think GA has certainly raised the bar from an interface and reporting perspective. Certain features, like the integration with Adwords and the goals based analysis, are really useful. So I do think the potential exists for clients to point at GA and ask the Omniture’s, Visual Sciences’, and Unica’s of the world, why can’t you do that? Or why can’t you do that as easily or as prettily as Google? Yes, I do think that’s happening. That being said, the enterprise vendors offer products that have many features GA does not have right now (such as control over updates, lookups, decodes, open database with detailed data, custom and ad hoc reporting, hybrid data collection, extensible schema and more), so I also think the same is being asked of Google. I’m curious to see what the fine folks at Google have up their sleeves for V3…
Thanks for commenting! I appreciate your sharing your thoughts and taking the time to read my blog.
Steve added the following ...
Actually the GA numbers are so close to the logs that there isn’t really anything in it to get excited about. With appropriate filtering naturally. From memory it was around 10% out at most, last I looked. Well within acceptable given the technology differences!
Also, as we’ve all “grown” from the logs the inherent … softness? is known and understood.
Having said all that? We only have GA numbers from mid June ‘07. We still need to provide numbers - sometimes - back to 2002. So GA has to be running a *long* time for us to really switch for the historical side.
You know, now I think on it some more. No one in high authority questions the numbers. They get and ask for so many numbers continuously, I suspect they simply trust that we’re doing the right thing and everyone is happy. Make of that what you will.
Plus, some of ‘em still talk in terms of “Hits”. We’re gradually educating upwards, but it’s slow.
But yeah, two of the 3 tools are 98% of the time only used internally.
Cheers!
Mat added the following ...
One of the places where I see confusion being generated here is amongst companies that had log parsing tools for some period of time, and then the issue of page tagging was introduced to them later. I have been working with clients who make such foolish comments as, “We put our log files through product X, and we put our Page Tags through product Y” Which is all fine and well, if you’re looking for a delta of robot/spider traffic, but does nothing for the business sponsor in his quest to segment users based on their actions on individual pages. The system of record, whether it be a free tool or a fee tool, needs to be one system or homogeneous set of tools that complement and integrate with each other rather than a hodge-podge of potential data sources deployed in an incomplete way…
Plusieurs outils de webanalyse sur un même site : une bonne idée ? added the following ...
[…] un excellent article de Judah Phillips qui est un des “gourous” les plus connus de la webanalyse avec Eric Peterson: est-ce […]
Steve added the following ...
Technically we’re running four tools. Three are complimentary and do all the work (no guesses for how awful one is.
). They do different things vs an all in one solution.
GA is the third, and while we sorta use it, it’s main purpose was to demo GA’s capabilities. So as to run on the rest of the Org’s websites.
Given they’re only running monthly reports AND with a multi day turn around on any request for the logs? Well a remote tagging solution simply works betterer.
Cheers!
- Steve
PS Another excellent post. When does your book come out, and can I get an autographed copy?