A successful web analytics practice helps a business manage its performance toward goals.  If you believe the statement, then you understand that in order to manage for site performance, a web site must exist to support one or more goals.  Pretty logical, right?

Then why is that web analytics practitioners tell me they often encounter web site owners who have no clear, measurable goals?  It’s really strange.  In fact, it’s vexing and frustrating because without known goals you OBVIOUSLY can’t manage site performance.  Without goals you can’t really optimize anything and are left to simply tracking trends related to basic metrics and/or derivatives thereof.

Thus, one of the responsibilities of a web analyst is identifying and confirming site goals.  Once you have site goals, you can create KPI’s that you monitor on a regular basis investigating variances, anomalies, and outliers affecting site performance. 

Undoubtedly one of the goals you will identify for your site is some type of conversion.  A conversion is a value-generating transition on your site.  For example, the successful completion of a form that enables a visitor to download a content asset or completing the purchase of a product may be a type of conversion you measure. 

While overall site conversion rates for value-generating events are important to know, real insights come from segmenting visitors or other dimensions by conversion rates.  The best analytic tools enable you to identify the conversion event and slice the conversion by any dimension: page, geography, referrer, keywords and so on. Thus measures of conversion will be instrumental KPI’s when using analytics to manage performance. 

The notion of a “funnel” in web analytics is a sequence of one or more pages that a visitor clicks-through until reaching a final destination, the conversion event.  The “funnel” assists the analyst in understanding the discrete clickstream that led to conversion.  For example,  a pre-defined path through the pages in the subscription or download process could be considered a funnel.  Funnels can be linear and non-linear and are affected by all sorts of things like detours, fall out, abandonment… That’s a post for another topic…  Yet, the metaphor of the funnels is applicable across all sites…  This notion becomes problematic when we consider multichannel attribution of the conversion process (again another blog post when I have time).

So what’s an analyst do when they want to begin to use web analytics to manage performance against goals? Here’s are some tips:

1. Investigate the business’ revenue model. Advertising-based sites generate revenue from selling various types of ad units (rpm/cpm), contextual advertising (rpm/cpc), lead generation programs (rpl/cpl), revenue sharing, and via affiliate syndication and content sharing deals.  Ecommerce sites generate revenue by selling products or brokering services or transactions.  How does your site generate revenue?  If the goal of the site isn’t to generate revenue, then skip this step.
2. Ask key managers to identify business goals.  Top-level managers have a better grasp of the vast ecosystem of suppliers, buyers, and other priorities that you, the analyst, may not be privy to.  Your manager should be helping you put your analytics work in context of the business goals.  So ask your boss what are the site goals?  Don’t accept the answer “to drive revenue.”  Ask how the identified goals support value creation and revenue generation.  The measurement of events supporting business goals should be a focus for performance management and optimization. 
3. Identify the conversion events that support businss goals and the revenue model, including any necessary steps in the funnel.   The actions that satisfy goals are the conversion events. The transition involved when the visitor clicks and makes the site money is your discrete conversion event. The page immediately preceding the conversion event is the last step in the funnel. 
4. Determine the actors on the site.  Actors can be categorized into internal/external actors.  For this exercise, concentrate on identifying the roles and responsibilities of the internal actors who DIRECTLY influence site production.  In other words, who are people modifying the site and what do they do?  The indirect actors, like your boss, also affect the site, so make sure you consider their role and responsibility in advanced site goals and fulfilling the objectives of the business model. 
5. Determine the goals of the actors.  Like site goals based on revenue, all indirect and direct site actors will have goals specific to their jobs.  Actor goals support site goals.  Thus, actor goals can be translated to tactical KPI’s.  For example, the editorial actor may want to ensure that X number of newsletter-referred visitors subscribe to the print magazine, so you create a KPI’s “subscription conversion rate by newsletter” and “number of online subscriptions generated per newsletter.”  Based on the site goals for conversion and the number of subscriptions generated from the online channel, you can start managing editorial performance.
6. Document site goals, actor goals, conversion events, and funnels, including a diagram a hub-and-spoke model of actor roles and responsibilities and flow diagrams of funnels.  In order to establish a process for performance management via web analytics, all the actors must generally agree on roles and responsibilities.  By documenting your investigation, you confirm correctness, identify gaps in business process, and create alignment among actors and management.  You may notice breakpoints in site production processes too.  The end result is a fully-documented operational model of how your site is created, monetized, and deployed.  In the same way that you can’t manage what you don’t measure, you shouldn’t be measuring things you can’t manage.   
7. Have key managers who direct site actors sign-off approval on the documentation.  The holy sign-off confirms you correctly identified the revenue model, site and actor goals, site navigational flows that lead to conversion.  When questions arise you can reference this process artifact to backup the conversion events you defined, the KPI’s you’ve created, and subsequently the performance recommendations you will make and manage.

Then:

1. Configure your web analytics tool to report conversion rates for revenue-generating site transitions and events and to report on funnels.  All tools can do this. 
2. Build KPI’s based on specific functional goals performed by actor’s on the site. Base KPI’s around activities that support the core revenue model and the activities performed by site actors.
3. Review KPI’s with actors.  Bring your documentation and identify how the KPI’s will be used to identify performance, contextualize optimization recommendations, and help each actor be more successful at their job. 
4. Report conversion rates and KPI’s to key managers and site actors.  Optimally these performance metrics should be available for actors and other stakeholders whenever they want them, preferably in the form of dashboards elicitable from your web analytics tool.  The goal should be identified, the target value for the goal, the KPI measurement,  and any deviation from the goal should be noted along with written performance recommendations.
5. Research site performance by segmenting conversion rates and KPI’s and investigating drivers.  KPI’s provide context for understanding the actions that influence site performance.  Overall conversion rate will only tell you so much.  For example, to the SEO conversion by organic search referrer is more informative.  Other actors will require reports on segmented conversion specific to their function. 
6. Make optimization recommendations.  Whether you deliver recommendations via reporting or manage the multivariate testing function at your company, you’ll need identify the events or actions to optimize.  Then you need to get buy-in from various gatekeeping actors.
7. Test and implement optimizations.   Use a multivariate testing vendor to test combinations of content and creative that drive KPI’s and that provide lift in conversion.  Work with site actors to ensure optimization testing and controlled experimentation occurs.
8. Rollout optimization that increase conversion, improve goals, and drive revenue.  Once you are reasonably certain the optimizations you are suggesting will improve performance work with the web or product development team to realize these changes.

Thus web analytics for performance management involves:

• Goal Clarification.  Why does this site exists in the first place?  Don’t be surprised to learn different actors have different goals, and no one is aligned!  From what I hear on the street that’s a common issue!
• Stakeholder Alignment.  Do all stakeholder and actors agree on the reasons why the site exists?  If not, be prepared to mediate.
• Experience Optimization.  How is the visitor interacting with my site, and do those interactions channel visitors to conversion funnels?  Do relevant calls to action and points of resolution work for persuading visitors to convert.  What’s working?  What’s not?  Figure it out.
• Controlled experimentation.  Based on potential optimizations available, what do you test?  Multivariate testing software can help, as can VOC research.  Talk to your research team.  Use the AB testing feature of your web analytics tool…  Whatever you do, you should establish a repeatable process for doing so. 
• Outcomes measurement.  If you set up a KPI dashboard with goals, actual performance, and variances you will be able to answer that question “so I did all this stuff, so what effect did it have?”

Easy right? Now get to managing performance using web analytics!

In wild world of online metrics, it’s a well known fact that metrics from web analytics tools and ad servers never match. Variances can be substantial. 

What I mean is that, given no “refresh rate,” the total impressions for a single ad unit, which should be served on every page request, never matches the number of total page views on the site during the same period of time.  Sigh.

Reasons why identically-named metrics from these two tools (like page views and unique visitors) don’t add up are numerous:

• Different data collection methods.  Ad servers use page tags.  Many web analytics tools use page tags, but it’s not uncommon in web analytics to use additional methods, such as logs or packet sniffers.  The methods have no shared standards for collection or storage of the same data (like visit-level data).  Thus you get apples to strawberries comparisons when attempting to correlate the dimensions from different systems.
• Unique data models.  Ad servers aren’t focused on counting page views and the other dimension of web analytics (visits, time, and so on).  Rather ad servers focus on serving and counting impressions served (and loads of related derivative calculations, like CTR, CPC, and the coolness of view–thru).   Metrics are based on an ad request and an ad code.  Ads aren’t targeted to a page (though that’s possible), but rather to a “zone” or “keyword.” What that means is that “page” dimension may not even exist in your ad server’s schema.  In other words, you aren’t looking at impressions measured on a page, but rather at the number of impressions served in a different conceptual construct.  That’s one of the reasons why people say metrics and ad-serving systems “don’t measure the same thing.” 
• Untagged pages.  Just like analytics implementations suffer from challenges related to complete code coverage of page tags, so do ad serving implementations.  Companies need to determine how to centrally manage the deployment and orchestration of page tags *of all types* and verify all the pages have tags!  Don’t just expect it to work because tagging sounds so easy!  Suspect it won’t work, and determine what you’re going to do *before* you deploy.  Too late?  Time to reengineer. 
• Non-JS executing clients.  Ad servers use page tags.  Not everyone and not all user agents execute javascript.  Everyone needs to realize that page tagging misses traffic as efficiently as it excludes it.  Period.  What percentage of the traffic you miss, you’ll never know… running and filtering your logs may provide an indication…
• Ad blocking software.  Firefox’s Adblock Plus software is a big problem for sites that have a big techie audience, and it affects all sites.  Check your browser reporting and realize a good majority of those Mozilla users may be blocking your ads.  Look at the attitudinal data you have about visitor’s to gauge whether that’s a big issue for your online audience. 
• Cookie issues.  Third-party cookies get blocked (often by privacy software).  Many ad servers still serve third party cookies, and many corporations have not tricked their DNS to accommodate this issue (ahem, CNAME).  We all know how cookie deletion affects unique visitor counts.
• Refresh rates. One page rendered in the browser and many banner “refreshes” makes it really hard to correlate page views and impressions served.
• No rich media installed, and no fallback.  If the client doesn’t have certain plug-ins, and you have no fallback, you miss ad revenue.  Meanwhile the tag executes and you count the traffic.
• Robots, spiders, and crawlers, oh my.  The web is so robotic.  The problem is amazingly understated, especially by companies who want to bill you on page views.  Different data collection methods allow some level of bots to dirty the data.  Logs are harder to efficiently filter.  When the ad server uses tags, and the analytics tool uses logs, you may get some wildly different numbers. 
• Mobile, Mobile, Mobile, Mobile.  Not all Internet-connected mobile devices will display ads, but web analytics tools will track the behavior of mobile visitors.
• Latency.  Visitors who move through the site too quickly may not execute the tag, thus no data is sent back to the server(s).  Ever wonder why vendors tell you to put the tag “high” on the page?

The influence these issues have on your site varies depending on audience.  Investigate factors causing variance and deviation between metrics systems, and educate your audience on why the numbers differ.

The concept of web analytics data sources isn’t discussed nearly as often as web analytics “data collection.”  With that in mind, I’m often asked by people just beginning to explore the wild and wonderful world of web analytics  “where does this metric come from?” 

When people ask me that question I often think back to someone who I may have seen walking in the local village a few miles away if I had lived about 170 years ago.  Ralph Waldo Emerson.  The father of American Transcendentalism once asked “whence is the flower?”  From where did it come from?  Ralphie-boy was questioning the accuracy of long-standing 19th century beliefs.  Keep in mind Mr Emerson parted ways with Harvard for questioning the Trinity (not Avinash’s ;)!

In that spirit, people are really asking me “whence is the online metric?”—from where does the metric come from?  Advertisers and your colleagues want to know because they may be questioning the origin and accuracy of our 21st century metrics! 

While folks who have been “doing web analytics” for some time know how to answer that question, I thought it would useful to blogivate on data origination for folks who are new or aren’t dealing with web analytics in full-time, day-to-day role. 

The answer is that online metrics originate from one of three data sources:  internal data sources, external data sources, and hybrid data sources.  Each source has its own particular challenges to “accuracy.”

• Internal data sources.  Web analytics technology that collects data from websites, mobile phones, or other Internet-connected devices via javascript page tagging, log file processing, or packet sniffing falls into this category.  Major vendors include Unica, Visual Sciences, Omniture, CoreMetrics, Webtrends, Google Analytics, and others.  The issue with Web analytics tools is that two tools will yield different numbers for the same data source.  That’s because each tool has its own “secret sauce” for “sessionization.”  That is, each tool counts traffic in slightly unique ways.  For example, tools may be configured to include or exclude certain filetypes or server responses.  Robotic traffic may or may not be filtered.  Web Analytics tools also depend on cookies for attributing “uniqueness” to visitors; thus, cookie deletion can overinflate unique audience numbers. 
• External data sources.  Data collected from panels, toolbars, and ISP’s–not from the actual site– are examples of external data.  Companies like Comscore, Neilsen Netratings, Compete, and Hitwise provide metrics generated from external data.  These companies provide some sort of incentive (i.e. gifts) or perceived value to entice people to participate in panels and download software that observes their Internet usage.  Self-selecting panelists are monitored, and metrics related to their behavior are projected to the entire online universe using statistical methods.  External metrics are never identical with each other because of differences in consistencies of their panels.  As you’ve probably noticed, Comscore data does not match Nielsen.  Significant divergences across panel-based measurement systems when compared to each other and web analytics tools has led to an audit by the IAB and MRC.  The hope is that auditing will vet methods and identify any potential coverage error or selection bias inherent in sources of external data.  Personally, I’m really curious to see what all the hubba bubba about auditing without guiding standards will really accomplish.
• Hybrid data sources.  Sources of metrics that use some type of internal data collection and some form of external data collection are considered hybrid data sources.  Microsoft Gatineau and Quantcast offer free services that fall into this category.  Reports in the highly anticipated web analytics offering from Microsoft include data collected from Javascript page tags combined with anonymous demographic data from Microsoft Live profiles.  Quantcast’s panel-based measurement system (i.e. external) may be augmented by adding a javascript page tag (i.e. internal) to every web site page.  Behavioral data is collected via javascript and combined with demographic data from the Quantcast panel, then reported to end-users.

In real world practice, companies use many of these different, overlapping data sources to understand their online presence.  Given the free nature of companies like Quantcast and Compete, and the pervasiveness of firms like Neilsen and ComScore in sales and agency cultures, expect that different sources regardless of type will never ever absolutely match site-specific Web analytics tools.  And that’s okay because data from all these data sources have different utility:

• Metrics from internal data sources derived from web analytics tools are very useful to site owners for identifying the behavior of their online audience.  The data can be used for site optimization, to understand what sites are referring traffic to the site, to identify conversion rates for particular marketing campaigns, to understand the broad content themes and particular search keywords driving traffic on your site, to segment and give context for other metrics and attitudinal data and more.  Metrics from site-centric sources should be provided to advertisers for comparison with external data.  Be prepared to discuss why the numbers differ!
• Data generated from external sources are useful to advertisers and agencies.  These metrics can be used for comparing the performance of a site to its competitors and for understanding audience behavior across one or more sites by demographics.  Media buyers and web strategists desiring to understand generalized Internet traffic trends and measures of site popularity use external sources.  Metrics from “audience measurement” sources should be used for comparison with internal data.  But the data won’t match, and that’s okay because you should be looking at the site performance of your competitors, not using that data to optimize your site.  Use the external data for insights into demographic makeup of your audience.  Then compare that information to data from your own internal research teams (who don’t report to web analytics). 
• Numbers from hybrid data sources blend both external and internal metrics together for both site owners and advertisers.  No duh, ay?  New insights about the online audience can be realized from segmenting visitors based on demographic and behavioral data within the same source.  We’re just entering the “early adopter” phase of this market, so I’m curious to see how it all plays out.  How will Microsoft Gatineau differentiate its hybrid analytics service and communicate the value proposition?  Will publishers adopt Quantcast’s hybrid service (and how will they make money)?  One barrier to adoption is that some companies have already combined web analytics behavioral data with audience demographic data using business intelligence and data warehousing technologies (or the more flexible and open web analytics tools that suppor open software standards).   Companies bridging data internally hope to enable a grand orchestration of automatic site optimization and content targeting (with an eye toward behavioral targeting).  Microsoft already seems to be doing this targeting to some degree on based on what I learned about “personamous” content targeting on MSN.com (at Emetrics).  So I’m curious what other online products (for example advertising offerings and site optimization technologies) the gentlemen at MSFT have up their strategic sleeves based on this hybrid source data. 

As you immerse yourself in the wondrous world of overlapping, hardly-standardized, online metrics, it’s critical to consider the source of the metric, bias from the source, and the audience for the metric.   Most critically, understand how a metric, regardless of source, relates to business goals and site objectives before using it as measure for identifying your online performance to internal and external stakeholders and taking action from your insights. 

GO RED SOX! 

I’m contributing a monthly article to MediaPost’s Metrics Insider Column.  My first contribution was published last week, and I’m reposting it here to get your thoughts.  Soon I hope to describe what I mean in more detail… The article was called “The Most Measurable Medium needs and XML Standard.” In case you missed it here it is:

OVER THE LAST TWO WEEKS, my fellow Metrics Insider columnists have correctly pointed out that online metrics are neither standardized nor easily integrated across systems. Vocabulary is muddled. Numbers do not match. Data exists in silos and is isolated from related data. Systems do not adequately or easily talk to each other. Research services, ad servers, and Web analytics tools report similarly named, overlapping and often conflicting metrics. Unfortunately, these problems will not disappear anytime soon, even with emerging “standards” and continued attention paid by the industry to these important issues.

Current industry standards for Web metrics are limited, basic, and come from independent entities. Most recently, the Web Analytics Association released a set of “standards.” The WAA’s standards are elementary definitions of concepts from various periods of Internet measurement. Web 2.0 concepts like “events” are mingled with dated measurements like “hits.” Regardless, these definitions provide a very useful starting point for framing a discussion about metrics. Recently, I’ve learned that the IAB and MRC are developing a set of IAB Reach Measurement Guidelines. Let’s hope the IAB and WAA align their work efforts.

The IAB and MRC are also currently auditing “audience measurement” firms, like Comscore and Nielsen. It’s rather unclear to practitioners what standards the IAB/MRC are applying to the audit. But the hope is that auditing will expose issues of coverage error and selection bias in the black box methodologies used to create the panels and generate the audience measurement data.

It is important to note that the IAB’s audit has two parts. The first is certification, which indicates the company being audited is applying the “standards,” and the second is accreditation, which demonstrates adherence to the IAB standards.

Only time will tell if companies like Hitwise, Compete, and Quantcast will be asked to submit to auditing. It’s worth mentioning that legacy metrics “standards” (and audits) from historic organizations like ABCe still occur and carry weight with publishers and advertisers (especially outside of the United States). It’s entirely possible that newly formed organizations, like the Association for Downloadable Media will offer their perspective on “standards” for online metrics.

The idea of “standards for the standards”–however absurd it sounds on the surface — starts to seem like a good idea when considering that all these parallel efforts aren’t intersecting. Honestly though, I question whether “standards” that are purely “definitional,” even if agreed upon, will solve many of the measurement challenges companies have when trying to understand Web data and take action from it.

Standard definitions are helpful for promoting understanding and creating a controlled vocabulary for discussing online metrics, but they don’t help with what I see as a huge challenge in today ’s metrics technologies. The problem is this: currently available online metrics systems do not adequately separate data from presentation . That’s a huge limitation preventing Web data from being easily integrated with other systems.

Detailed-level Web data (the raw data) is often costly to extract, if available at all. It is nearly impossible to deliver detailed data in real time from Web analytics, ad serving, and research-based technologies in order to feed other systems. The majority of hosted (ASP) metrics systems are closed and do not allow access to key interfaces using open software standards. For the most part, today’s metrics technologies are black boxes where data goes in, but can only be extracted in various file formats after creating a report. Common export formats include csv, pdf, and doc. While XML exports are often available from many vendors, there is no standard XML schema for describing the same type of Web data across different sources!

The industry must begin collaborating and creating a standard XML schema for describing Web data. Creating a widely used, consensus-based, published, and maintained XML standard for online metrics would make it possible to more easily share, transform, and use Web data in other systems.

I firmly believe that current metrics standards must go beyond simple definitions and tackle issues pertaining to data portability and system interoperability. Then we’ll all be in a better position to reuse Web data across the enterprise value chain. Once we all agree on “standard” definitions, I encourage us to start working together to develop a standard Online Metrics Markup Language.

Last week I attended the 1st Annual SEMPhonic XChange and led a collaborative discussion on “deploying measurement systems in distributed companies.” In case you hadn’t heard about SEMPhonic, the company is a boutique consulting firm in Novato, CA (near one of my favorite towns on Earth: San Francisco).  They do some very unique work with web analytics (Functionalism) and other facets of new media technology.  A fine gentleman named Gary Angel leads the group. He’s recently hired smart folks, like long-time web analytics expert, June Dershewitz, and notable blogger and author of the Web Analytics Report, Phil Kemelor.

A few months ago the idea came up for me to lead a “huddle” at their Xchange conference.  I was intrigued with this huddle idea.  It was different. New.  I’d facilitate a discussion on a topic of my choice in a small, Socratic, group setting.  No PPT!

Fast forward to last week, and I found myself sitting in Napa, CA at COPIA: The American Center for Wine, Food, and the Arts. COPIA was a brilliant place for an industry colloquy.  Being a guy who likes culture (and conferences), this venue offered the best of both worlds.  Instead of a hotel, I found myself surrounded by gardens, vineyards, art, food, wine, web analytics, and the brightest in the industry.  Cool!

Thinking back on the event, SEMPhonic Xchange, in my opinion, is a “must-attend conference.”  Not only is the “huddle” format unique and fun in which to participate, it’s also a format that promotes deep discussion that leads to truly actionable insights. It’s a conference based on dialog and collaborative discussions between participants.  The huddle format provides 10 hours of free collaborative consulting (5 huddles) with employed people you can’t hire (and some consultants you can)!  Compare a daily consulting rate to the conference cost, and it’s a no brainer.

My discussion on best practices for the successful deployment of measurement systems lasted a little over two hours.  My group collaborated nicely (so I thought).  It included very intelligent folks like Jared from Intuit, Scott and Christel from Xilinx, Sami and Fred from Adobe, Renata and Matthew from American Express, Aaron from Webtrends, Rupa from Cisco, Amy from JPMorganChase, Jeff from New England Journal of Medicine, and Kevin from Charles Schwab.  Thanks to anyone reading this blog who attended.  Your valuable insights and knowledge sharing contributed to the success of the huddle.

At the ends of the huddle I went over a tips for a successful deployment.  Here are a few:

• Identify business goals, match those goals to business strategy, and align metrics and KPI’s to support those goals.  I always say “metrics and kpi’s have significance in position and relation to a goal.”  You can’t measure against performance unless you’ve identified your business goals.  Goals are supported by strategy.  You create KPI’s for measuring against goals and for guiding work toward objectives.  Every business unit could have different goals.  Management must align and support the standardized goals that you bake into your KPI’s (and any specific derivatives you create to support the goals of differentiated business units).
• Verify the technology implementation and the metrics collected.  A web analytics or other measurement system must be verified to ensure conformance with “standard” best practices based on the company’s goals.  You need to make sure the numbers can be trusted.  That means, validating data and reconciling differences with historic reporting systems or whatever the business demands, understands (and possibly pays for in real dollars, time, or resources allocated), such as ABCe, WAA, IAB, MRC, XYZ, 123, FBCINSA whatever it takes…
• Provide education and training.  While some analysts want to keep all the data within their analytics group, I don’t think that practice scales across a large enterprise unless you are lucky enough to have a large analytics team.   I know I can’t individually and alone satisfy the metrics needs for thousands of stakeholders or hundreds of brands.  Successful companies deploying on a large scale adopt a “train the trainer” approach.  The trainers guide their business units, become local experts, and help foment a data-driven culture.   Let the data be free and the people educated, I say. 
• Consider the corporate culture.  A metrics tool changes organizational culture (for the better or worse).  Suddenly, everyone is being measured and perhaps evaluated on goals implicit in the measurements.  Some will greet the tool with open arms while others will see it as a threat.  Solid management needs to foster buy-in and support for the tool.  That way, organizational resistance is overcome and clarity of mission is realized.
• Help business units ”use the metrics” to improve performance.  My friend Eric Peterson likes to say “web analytics is hard.”  Yes it is!  That means the expert needs to work with global business teams to mentor and teach how to separate signal from noise.   As the tool is used, business units will identify “pain points” that will need to be addressed.  The analytics team should work with business units heal the pain and improve performance.
• Plan to ”get granular” and the “get integrated” with the measurement system.   Additional requirements will come out of the woodwork due to organizational learning after you golive (even if you think you’ve elicited all reqs prior to deploying your tool and building reports).  New requirements are an early sign of success (people are “getting it” after all).   As the business learns, it will become necessary to extend the system.  Consider brainstorming about the possible ways in which the system should be extended prior to rollout, and perhaps create a basic plan for extending the system. When the system is launched, modify that system enhancement plan and execute on it to support business goals.  Employ a project manager to plan and help execute!
• Manage and guide stakeholder expectations to minimize risk.  Be realistic when for guiding and setting expectations so that you can minimize risk (to your overall mission, your job, the company ;).  Risk comes from incorrect metrics, organizatonal issues with adoption, changes in business goals, shifting managerial priorities, technology problems, and issues resulting from resource allocation.  Make sure you manage around these issues or find ways to directly deal with them. 
• Generously explain why the measurement system is being deployed and more.  You must create a well-thought out communication plan for promoting adoption and use of a metrics tool.  The communication plan should focus on answering:
  • Why the tool is being deployed.
  • What people are supposed to do with the tool.
  • How people should use the tool.
  • What key metric/KPI’s people and business units should be looking at to manage their performance.
  • How to go from “insight to action” based on analysis of the metrics.
• Share best practices and lessons learned across the enterprise.  As new insights are realized and the company starts taking action from the metrics, you should provide a way for the enterprise to communicate the “highest and best use” of the tool.  By promoting collaboration and knowledge sharing, the company is more likely to succeed quickly with the measurement tool and realize a demonstrable ROI from it.
• Realize that “premature optimization is the root of all bugs.”  When deploying a measurement system, you need to establish a baseline system before extending the system to get more “granular.”  An implementation must be granular before integrating data from other systems.  While these concepts will overlap in deployment and may occur in close proximity or in a waterfall (i.e granularity may be enabled via integration) you need to ensure you don’t put the cart before the horse.  Make sure you correctly measuring and understanding the basics (like recency, frequency, clickstream, referrers, bounced visits, and depth) then moving forward with more advanced and necessary measurements and reporting (like bounce rate, conversions, view thrus, voice of the customer, and ”engagement.”)   Set clear expectations and guidelines with the consultants.  Don’t move too fast with development.  QA, then segment away, I say… ;-) 

A lot of more was discussed and shared over in Napa and throughout the SEMphonic XChange conference.  Please share your thoughts, if you feel like it!  Thanks for reading!

Running more than one web analytics tool on a site or across a portfolio of sites is an increasingly common practice these days.  The majority of the companies that run multiple tools probably run one “for pay” tool and at least one “for free” tool.  Based on my experience, the cost of running two “for pay” solutions would be prohibitive for companies still trying to realize the “ROI” from web analytics (but it’s not unheard of in large, solvent, multinational companies).  Not surprisingly, the most common “free tool” ran next to “enterprise-level” :-) tools like Omniture, Visual Sciences, and Unica NetInsight is Google Analytics.  Data from my pal Eric Peterson’s Vendor Discovery Tool shows that GA and Visual Sciences code were found on 6% of tracked URL’s, GA and Omniture code on 4% of tracked URL’s, GA and WebTrends Hosted code on 4% of tracked URL’s.  That’s great for Google and quite an edification of their excellent product!

I’m sure that there are companies running multiple “for free” tools, and/or running multiple big ticket tools (like HBX Analytics and Visual Sciences), and/or multiple homegrown tools built from Business Intelligence technologies and databases (Oracle/Cognos).  Yet running multiple tools has risks and rewards.

Some of the risks of running more than one web analytics tools include:

• Lack of control over data. If you trying to foment a data driven culture, nothing could be more frustrating than someone outside of the web analytics team downloading a tag, linking it to their personal account, then questioning why X number in Y tool doesn’t match X number in Z tool.  To promote adoption of new technology, running a competing tool has the potential to compromise data believability.
• Numbers not matching across tools.  Different vendors “sessionize” differently so numbers will never be identical between tools.  Dynamic sites, different underlying site technologies, and unique tool configurations mean numbers won’t match.  Never ever.  Check out Eric Enge’s highly-recommended 2007 Vendor Shootout. Run two tools and be prepared to answer questions about data discrepancies from those who consume reports on the same site from both tools.
• Conflicting vocabulary.  Different tools use different terminology.  One tool may use “sessions,” while another may use “visits.”  Some tools talk about “views,” while others reference “page views.”  Some tools use the term “unique visitors,” and other tools just talk about “visitors.”  When you are rolling out “web analytics” to people who need to speak the same language, having multiple vocabularies for expressing the same or similar concepts confuses discussion and muddles actionability.
• Apples and Mangoes Comparisons.  Some tools provide only snapshots of aggregated data, while other tools let you drill down, drill up, and slice and dice on detailed level data.   Some tools enable you to add metrics on the fly to any report, and then filter and cross dimensions until your heart is content.  Two people looking at two tools on the same data may conclude different actions are warranted based on the depth of their analysis.  While a good manager can sort that out, it’s a bother.
• Potential to misallocate resources leading to needless redundancy.  Companies have limited resources. If I need to apply tags to all my sites so that I can get to the real business of analysis, then why spend valuable time applying multiple sets of tags to enable tools that serve the same purpose.
• Licensing issues.  Google Analytics or Quantcast account on a corporate site associated with a personal account whose owner would prefer not to give up the password.   
• Training issues.  When rolling out systems, training is necessary.  Tools take time to learn.  Why have resources learn multiple tools that do more or less than same thing when you got real business to take care of?

Some of the rewards from running more than one web analytics tool include:

• Comparative data.  If you’re being charged by page views, it’s nice to have an alternate reference point to validate the charges.
• Differentiated reporting.  Some tools are just better at custom and ad hoc reporting than others.  If you have an inflexible tool not fully loaded with features then maybe it makes sense to get a tool that can do all that stuff a lot cheaper than paying for additional incremental features.  Hello GA!
• Potential to enable a different level of integration.  Lots of people tell me they download Google Analytics so that they can track their AdWords campaigns. 
• Ability to leverage different features.  Several major tools are technically and functionally challenged when it comes to simple things like showing the keywords used to drive traffic to the particular page, the number of unique visitors per page, or a bounce rate.  Instead of dealing with complexities, sometimes it’s just easier to download and install a free solution like GA that does all of these things at no cost.
• Ability to leverage different data collection methods.  Time-based metrics and file downloads are inordinately easier to measure and count using log file tools than using page tag tools, imho.   Why fiddle with some esotericisms in tagging when you can just run the logs?  Or better, yet, use a hybrid approach in one tool and get the best of both data collection methods.

Is it a good idea, as a site owner or manager of a web analytics team, to run more than one tool?  The answer is it depends on your organization’s capability maturity for web analytics and how you balance risks and rewards.

The most mature companies have a centralized web analytics function.  That means the company has one “master user” and “strategic owner” for web analytics and related technologies.  The centralized web analytics function has its own resources dedicated to “doing web analytics.”  Resources may come from other groups within a company, but, regardless, the company executives have identified and placed positional power around a “web analytics champion.”  Since you’re reading my blog, you may be this person!  Cool!

When you have centralization, you control key elements of doing web analytics:

1. Measuring
2. Reporting
3. Analyzing
4. Testing
5. Evaluating outcomes

If you’ve centralized your web analytics team, you should select ONE web analytics tool as your Primary Web Analytics Tool. 

Then I think it is then I think it’s safe to use more than one web analytics tool along these guidelines:

• Standardize on one tool as your primary tool.  This tool should become the ”bible” for web analytics data at your company. 
• Give people outside of the web analytics department access to the primary tool and ONLY the primary tool.
• Use the secondary tool within your web analytics team as a supplemental tool for comparing measurements, data reconcilation/verification, or analysis that you can’t accomplish with your primary tool (such as detailed roll-up reporting). 
• Keep the overall enterprise standardized on the interface and numbers from your primary web analytics tool.  That way you prevent confusion when reporting to stakeholders outside of the web analytics team. 
• Do not provide access to the secondary tool to people outside of the web analytics team (unless the numbers match 100%! ;- ) ).

If you have a decentralized web analytics organization, I recommend that you:

• Standardize on a primary tool (whether free or paid).  Remember Google Analytics is an awesome place to start!!  And so it seems is Microsoft Gatineau!
• Work toward centralization before introducing another tool that has the potential to undermine current measures, reports, analysis, tests, and outcome evaluations.

On a final note, you may have multiple tools for reporting web analytics data (perhaps from companies like Business Objects, Cognos, Microsoft and so on).  As long as the data is synchronized with the metrics from your primary web analytics tool, that’s fantastic! 

Am I off-base?  Absolutely right?  Do you run two tools and love it, hate it, don’t care?  What do you think?

As I explained in Part 1 of this lengthy blog post, companies switch web analytics tools due to a combination of issues related to:

• Price (which I covered in Part 1)
• Features (which I also covered in Part 1)
• Customer Service and Support
• Fulfillment
• The Company Itself (not the vendor, us)

Let’s tackle some thoughts on the the last three (Customer Service and Support, Fulfillment, The Company Itself):

• Customer Service (and the post-hoc judgement of it): 
  • Post sales:
    • Unresponsive or inadequate customer support and professional services.  So many items fall into this category.  From missed phone calls and emails not returned, boilerplate/unfocused proposals,  generalized answers from support to other forms of poor communication, to followup and failures in delivery of service level agreements, to vendors not spending enough time talking to clients after the first year as they do during the first year (as Benry pointed out in his comment to Part 1).  All makes a company begin thinking money and time may be spent better elsewhere…
  • Pre-sales: 
    • Sell them what they, make them buy what they need later.  In this bucket fall items that cause upsetting realizations  that lead to talk about ”the switch”well after the sales process ends - from inadequate professional services implementations to the need to buy another one of the vendor’s products to store all of your data or segment it… 
    • Smoke and mirrors.  Pilots and sales demos look slick but the results of your initial implementation may greatly differ.  After all, your team needs to learn a new technology.  When your team realizes that it will take X months and X hours of dedicated resources to create the same type of functionality you saw in a simple, canned pilot, then the seed for switching is sown.  Keep in mind, vendors tune their pilots for performance and to exhibit the best features they have.  They sell you an executable, documentation, and professional services, not the pilot (no matter what they say… otherwise get in writing). 
• Lack of fulfillment:
  • Implementation problems.  Tagging all of your pages or processing your log files is easier to talk about in meetings than do in real life.  If you’ve spent two years trying to tag your pages across all your companies domains, you may just get frustrated and move on to another solution.      
  • Complexity.  xys_435, this=12r, pass this variable, make this request, do this, do that.  Companies without sufficient expertise inhouse and dedicated resources may get sick of the esoteric proprietary nature of a tool and not want to hire vendor professional services, so they look for technology that is easier to wield using internal skill sets.  Couple that with an acute inability to extend the data model, to use API’s, to access the data in an open database, to decode, and to use lookup tables, and viola, the “switcheroo” gets brought up.
  • Not customizable enough to solve business challenges.  Can’t save reports, add metrics on the fly, use whatever filter you want, extend the data model to include custom business dimensions?  Or you can do it, but it costs too much?  Dang.  Maybe it’s time to take a look at what current technology offers?   
  • Limited ability to integrate cross-channel data.  Isolated silos of data are not good.  Integration is necessary for realizing insights across online and offline channels.  Resistance to this fact is futile.  Products that resist this fact won’t last for long in companies that realize it.
• The Company Itself:
  • New or changing goals.  As a company learns how to “do” web analytics, they may realize the tool is incapable of meeting new goals and move on.
  • Current tool doesn’t fit into business process. As companies engage in Business Process Management (BPM) and Master Data Management (MDM) initiatives and develop service-oriented architectures (SOA), web analytics tools need to fit in.  A company may judge their current tool doesn’t support integration with X or enablement with Y, then begin looking for a new vendor.
  • Organizational alignment. If a company isn’t organized in way that promotes the usage of the analytics tool to achieve business goals, or if the deployment of the tool is not managed by a business person who ensures technical development achieves business goals, then the true value of the analytics technology may never be achieved. Business owners will move on and demand a new web analytics tool.
  • Inability to meet internal service level agreements. If the system doesn’t enable the team to quickly adapt to evolving business demands related to internet measurement, SLA’s may slip,  which leads to thinking about technology change.  
  • New management.  Change comes with new management.  Always.  Inevitably.  Sometimes that change comes from standardizing on a tool that management has had success with in the past. 
  • Limited resources to support tool.  Given a hard to identify ROI, the company may limit resources.  As a result the tool withers on the vine and gets replaced.
  • No EXPERTISE.  With a dearth of qualified, experience web analysts, high salaries, and high demand, a company may just not be able to find expertise needed to maintain the current tool and move to something simpler.

The good news is 38% of “best in class” companies haven’t switched (yet), so some vendors are doing it right for some clients.  And 570 Fortune 1000 companies have no idea what I’m talking about yet at all, so that bodes well for you, the analyst or consultant, and you, the vendor, and us, the industry.

If you have switched and feel like sharing why, please do!  Thanks for reading!

Switching web analytics tools is fairly common (see Part 2).  According to a recent survey by analyst John Lovett (now at Jupiter), research suggests that 62% of “best-in-class” companies are currently on their second or third web analytics vendor platform.  An update of Eric Peterson’s Vendor Discovery Tool, due out any day now, indicates that only 43% of the Fortune 1000 have even deployed web analytics technology.  That means only 430 Fortune 1000 companies have deployed web analytics tools.  If Fortune 1000 companies are considered “best-in-class,” then, ouch, web analytics vendors could really improve customer satisfaction and retention!  My friend Robbin Steif over at LunaMetrics recently asked ”why?”  Why do companies switch web analytics tools so often?

The answer is that every company reaches a threshold moment of judgment about a particular technology - the instance the decision is made to either keep the technology or ”move on” and begin searching for new technology to replace the old.  This day of reckoning results from a combination of issues related to:

• Price
• Features
• Customer Service and Supported (covered in Part 2)
• Fulfillment (covered in Part 2)
• The Company Itself (not the vendor, us, and covered in Part 2)

The same goes for web analytic’s tools.  Challenges related to these five items fester until they cause a company to begin thinking about switching.   For example (the list continues in Part 2):

• Price:
  • Licensing or Subscription Costs.  Whether your running an in-house or hosted web analytics environment, it costs money.  Companies care more about things like the capital budgeting, debt service, and cash flow than they do about web analytics tools.  In times of limited resources or when the tool isn’t producing the desired return for a given cost, change may occur.
  • Per seat costs.  Every person who accesses the web analytics application may be thought of as having “a seat.”  How many seats do you need for your team?  your company?  The answer really depends on the scale of your implementation.  Seats aren’t inexpensive, and if you haven’t considered how many people in your organization need access the tool, you may soon find you can’t scale your implementation without significant revenue expenditures.  Per seat costs are generally factored into overall costs (sometimes as a line item).
  • Maintenance or upgrading costs.  Version upgrades or closely-related new products may not be “free.”  And that may make companies, that already feel limited with their current implementation, very ”unhappy” and cause “the switch.”
  • Costs for specific features.  Vendor’s basic pricing-models limit access to data filtering and drill-down features.  Slicing data across more than one dimension may require additional components or entirely new products, which have cost.  Some companies don’t dig deep enough during due diligence, only to buy a tool that they know can show “keywords per page” but hasn’t been configured or costs additional money to do so.   
• Features:
  • Limited features for custom reporting, segmentation, and ad/post hoc analysis by the business user.  Similar to the bullet above, Companies may offer a simple reporting tool with a limited number of ways to filter, segment, or correlate data.  To do more, the company may need to purchase additional features or completely new products not in the budget.  To lower cost, the company starts looking elsewhere for a comparable service. For example, time spent metrics are easy to calculate.  They’ve been in log files for years.  But don’t assume every page tag vendor has the ability to calculate these metrics out-of-the-box.
  • Inadequate data storage. Companies may not have asked “how long is my raw data stored” or if the data after a certain period of time.  It’s a real bummer when you want to backtest and realize you don’t have the data you need. Take for example, URL length, some companies may trim your URL’s unless you ask them not to.  You won’t know that until you realize it’s a problem.  At that point, not having historical data to test may cause managers to question the tool’s capabilities and to start thinking about the switch.
  • Limited data model, lookups, and decodes.  If the company wants deeper insights across more dimensions, the current product may have a limited data model that isn’t extensible to your business.  Decoding parameter or using lookup tables might not be possible.  The model may not enable you to track and report on things like Events, Spider/Bots, IP addresses, and file downloads.  If you can’t extend the schema, you limit your ability to use custom business dimensions already existing in your data warehouse or in other applications. 
  • Closed systems with no API’s or methods for rewriting data.  No API or methods beyond simple exporting of filetypes for getting data out of the system limits companies from achieving a 360 view of online customer experience.  If you want to do cross-channel marketing merging print and online, a simple reporting tool will not suffice (no matter how pretty the reports and trendlines). Companies that wake up to this fact, switch. 
  • Data not separate from presentation.  To integrate data across systems you need to have access to it.  For such needs, data is best in an open database, not only available via a web interface or on a “pay-to-use” basis.   Companies that wake up to this fact, switch.    

This list continues in Part 2.  Please click here to begin reading Part 2!

Facebook’s emergence as the platform du jour for social networking and the Internet marketer made me start thinking about how to “do web analytics” on the Facebook platform.  Even Eric’s moved the Web 2.0 Measurement Group to Facebook.

Apparently Facebook is thinking about�