Every month I write column for MediaPost.  This month I wrote  a short summary piece I thought I’d share with you in case you missed it.  Here it is:

As 2007 ends, I thought it worth looking back, from the practitioner perspective, at just a few of the issues that have shaped Internet measurement and thus online metrics over the last year:

• The Page View is Dead, Long Live the Page View.  During 2007, technologies like AJAX and Flash continued to erode the construct of the page view.  These technologies render content in a browser but do not always make requests to the server to do so.  If the technology you are using to measure behavior requires the page request and you do not have a page request, what do you measure?  The major vendors of online metrics tried to answer that question. 

Various audience measurement companies claimed “total minutes” and other time-based derivatives were better alternatives to measuring the page view.  Web Analytics companies rolled out features for measuring “events” subordinate or equal to the page view (and highlighted already existing time-based metrics).  Ad serving companies made inroads in reconciling view-through to assist advertisers in understanding the latent effect of ad exposure on the purchasing lifecycle.  Yet all these technologies still count and report page views.

• Engagement, Engagement, Engagement.  One of the hot topics in 2007 was a carryover from 2006.  Definitions for “engagement” emerged from the worlds of advertising, social media, online metrics, and more.  Engagement has been described as “turning on a prospect to a brand idea enhanced by the surrounding context” to “repeated, satisfied interactions that strengthen the emotional connection a customer has with the brand” to “apparent interest” to the more metrical “estimate of the degree and depth of visitor interaction against a clearly defined set of goals.” 

“Engagement” is very specific to the site being measured and full of nuance.  This fact has led agencies, consultants, and various companies to create complex engagement indices consisting of measures of key behaviors.  Behaviors are tallied and segmented in order to calculate an engagement metric, which is then used as the basis for site optimization.  These indices go far beyond often-cited simple time-based measures of engagement.  For a well-thought-of example, see Eric Peterson’s Engagement Metric.

• Cookie Deletion, Again!  Jupiter Research, in 2005, first uncovered and quantified how cookie deletion can affect unique visitor numbers in web analytics systems.  The effect of cookie deletion is not quantifiable in the basic way audience measurement companies want to quantify it in 2007 – by only examining cookie deletion rates from self-selecting panelists who visited one portal site and an ad server. 

Cookie deletion behavior varies greatly by audience segment and by site.  It may be as much of an accuracy problem in web analytics as selection bias and coverage errors are in panel measurement.  It is worth noting that some audience measurement firms use cookies to collect panel data. 

• Black Box Audience Measurement.  Many questions were asked about whether audience measurement companies adequately measure “unique visitors” or “unique users” or just the frame of self-selecting “unique panelists.”  In audience measurement, counts of “unique visitors” are generated using complex, black-box mathematics that project observed metrics to the entire online universe.  The projections are always unequal to the same data provided by other audience measurement companies or web analytics tools.  Panel inconsistencies (across at-home, at-work, at-university, or specific to the geography being measured) may cause some level of bias and error. 

Accounting for the difference between actual, observed panel metrics and projected metrics is perhaps even more challenging to clarify and resolve than the measurable effect of cookie deletion. 

• The Continuing Need for Standards Enforcement.  2007 was the year two significant industry bodies continued working on standards related to online metrics: the Internet Advertising Bureau and the Web Analytics Association.  While each organization serves the needs of different constituencies, they both share the inability to enforce standards.  Both bodies can say what you should do, but not what you “must” do. 

Throughout 2007, these issues (and others) brought increased attention and scrutiny to online metrics.  Corporations are inextricably linking online metrics to site and channel strategy and performance, and thus to overall corporate profitability.  The “numbers” are now more important than ever for managing an online business and maximizing online revenue.  Nevertheless, questions are still being asked about accuracy, precision, usage, and sources of online metrics.  We have a ton of collaborative work to do in 2008 to provide the best answers and numbers we can. 

Happy New Year!

If you’re new to web analytics, or the management of web analytics, it’s always helpful to get advice from someone whose been in the industry.  Here are a couple of simple principles I’ve found helpful:

• Investigate and be skeptical of claims by vendors. Upgrading a tool?  Migrating to a new tool?  Completely replacing one?  Adding a tool to your analytics arsenal?   Someone is then trying to sell you something.  Vendors really want your money.  And when it comes down to it, it seems the bigger the vendor and the smaller you are, the less attention you will get, even when you need their help really badly.  Vendors attempt to have their own, differentiated spin on web analytics - one of the reason the WAA developed standards.  Avoid falling into the mental model of any one vendor… push their envelope… champion for standards.  In doing so, make sure to realize that a good relationship with a vendor based on an understanding of mutual strengths and weaknesses will lead to collaboration important to your success.
• Don’t bias yourself when it comes to data collection.  Page tags are imperfect.  Orchestrating and maintaining comprehensive, deep tagging is never easy.  Log file analysis, while often disdained, has utility for situations where page tags are insufficient (say you want to detect how search bots crawl your site) or when tags are impractical (mobile).  Packet sniffing can be beneficial when you can’t tag or access log files.  In fact, some combination of hybrid data collection, for example page tags and logs, might be the best option for a given situation.  It’s up to you to work with your team to determine what method is most appropriate to use to reach your goals. 
• Realize that your IT department is your ally.  Marketers can’t do web analytics without IT.  Whether they help you include page tags, locate and synch your log files, or monitor the implementation of your packet sniffer, you will need to develop and maintain a relationship with IT.  You, the business marketer, may think quite differently about web analytics than IT, so work toward a mutual understanding that supports business and technology goals.
• Reconcile what it means to you when you hear that “Web Analytics isn’t easy… Web Analytics is hard!”  That’s a quote from Eric T Peterson.  He’s right.  You need people, processes, workflows, dedicated resources, teamwork, teamwork, teamwork.  How do you prepare an organization for something that is hard?  How do you prepare a team?   Training, education, evangelism?  Yes, that helps… You can make it easier by reading the books, checking out the blogs, going to the conferences, taking a UBC course, and participating in industy organizations like the Web Analytics Association.   

Web analytics tools can be outgrown, like houses, clothes, shoes, music, books, and ways of thinking about the world…  But how do you know when you’ve outgrown your web analytics tool?  In part 1, I began the list of five symptoms of an outgrown web analytics solution, which was spawned out a preso I recently gave.  The five symptoms include:

• Inadequate segmentation (covered in Part 1)
• Poor visualization (covered in Part 1)
• No custom reporting (covered in Part 1)
• Limited Integration 
• Cost 

So without further adieu, here’s the rest of the list and some thoughts regarding these symptoms (click here for Part 1):

• Limited Integration.  Soon after deploying a web analytics solution, you will become intimately familiar with clickstream data and simple counts of things (like page views) and measurements (like time).  You will hopefully have deployed Key Performance Indicators for understanding how effective your web site is at converting visits and meeting defined business goals.  Depending on the web analytics tool you use you may even have insight into the behavior and KPI’s of visitors from online channels like newsletters, search, and rss because you have applied “tracking codes.”

Soon will come a time when you may ask yourself how do I integrate data from other data sources?  You may want to bring data from an email service provider, CRM system, and registration databases, so that you can see delivery rates next to conversion rates from newsletters, or so that you may pass behavioral information about a visitor who registered on your web site.

You may want to move data out of your web analytics system.  Perhaps you want to feed your data warehouse?  Or you may want to feed web analytics data into a targeting system.   Simple XML-based feeds from a hosted solution may not suffice.  You will need access to your data in a open database. You may even want to stop non-human readable text and character strings from appearing in your reports. To do so you may need to lookup data using various methods in order to make reporting comprehensible.  All of these goals require some level of integration.

If your current web analytics tool can’t:

• Provide insight into all online channels
• Enable you to bring data from other systems into your web analytics platform
• Pass Web Analytics data to other systems.
• Manipulate data by looking up values, resolving urls, and decoding parameters

And do all of this at a reasonable cost in a maintainable way using in-house resources, then you may have outgrown your web analytics solution.

• Cost.  Web analytics done right isn’t cheap.  It costs money to maintain and extend whether you run an in-house solution or external solution.

When running an in-house web analytics costs are spread out across hardware and software and resources:

• Software license.
• Recurring maintenance costs.
• Servers (one or more).  Perhaps you virtualize (it cost money for the virtualization software).
• Database license(s).
• Storage.
• IT resources - people like project managers, application engineers, and dba’s.

As you expand your web analytics operation, all of these resources and technologies will need to scale.  Time will need to be devoted to maintaining it all, and time costs salary dollars. 

Your company will, hopefully, grow.  Then you will have more sites.  These will need to be tracked.  New reports will need to be created, tested, and rolled out into production.  New data and systems integration requirements will spring up.  All this has cost.

On the other hand, if you are using a hosted solution, you will need to extend the page tag and tool configuration when you want use more features or integrate systems and data.  That means spending money to use vendor professional services or consultants, unless you want to dedicate internal resources in IT who may already be overburdened.

At some point you say enough is enough. The COST is too much!  You then decide to invest in well-negotiated vendor solution that provides a lower TCO over some horizon.  When you start to run up against the barrier of cost, you may have outgrown your web analytics solution.

So that’s the list.  I can think of many more reasons why companies outgrow their web analytics tools…  What have I omitted?  What do you think?

Web analytics tools can be outgrown by companies, just like pants can be outgrown by people.  Over time, an analytics tool may no longer fit organizational needs or be well suited to deliver on complex organizational requirements for site optimization and multichannel integration (among other things). 

This topic led me Silicon Valley this week thanks to an invitation from Unica I headed over to Webex headquarters to record a preso in their new LiveStream studios.  A few other folks also participated in the production.  In attendance was Fireclick founder and all around cool guy, Steve O’Brien (also VP Internet Marketing at Unica), my pal and fellow blogger, Avinash Kaushik (of ZQ Insights and MarketMotive), the genial Elana Anderson (founder of NxtERA marketing and former VP Marketing Research at Forrester), the excellent bloke and all around nice guy from across the pond, Dr Alan Hall (Director of Analytics at SCL Analytics), forthcoming author and savvy multichannel marketer, Akin Arikan (Analytics Evangelist at Unica), and the jet-setting smarty Karen Hudgins (marketer at Unica).   We all had a blast getting down to business at Webex studios.  And eating at places like Burk’s and Parcel 104 - both excellent restaurants.

The title of my preso was “Symptoms you’ve Outgrown your Web Analytics Tool,” such as:

• Inadequate segmentation
• Poor visualization
• No custom reporting
• Limited Integration (covered in Part 2)
• Cost (covered in Part 2)

While I did manuscript the speech (hey I was being recorded!), it’s way too long to post, so I figured you all might enjoy me paraphrasing my own content.  So as I sit here on a Jetblue redeye, here it goes:

• Inadequate Segmentation.  Segmentation in web analytics describes the activity of categorizing and dividing your online audience and customers by their various attributes.  For example, you might choose to segment your audience based on their demographic location information to determine if a visitors from a certain geography have a higher conversion rate or behave differently on your site than visitors from another geography.  Or you may choose to use your web analytics tool to define a segment that you want to track such as visitors who clicked on a paid search term and did not convert, but came back to the site within one week.  Sounds easy, right? 

But not all web analytics tools can segment data.  The proprietary tool you run in-house may not be able to segment data.  Your expensive vendor solution may not be able to segment data easily.  Many tools only provide simple reports.  Yet basic reporting is insufficient for web analytics.  In order to understand new data relationships and the effectiveness of marketing campaigns to your massive online audience, you need to a web analytics tool that can segment data.

The idea being to do what I describe in this post on web analytics segmentation:

• Define a segment
• Identify expected segment behavior. 
• Measure current segment behavior. 
• Create “optimization hypotheses.” 
• Optimize content, offerings, user experience, and other site elements. 

How does your web analytics tool fit into the process of segementation that I described?  Does it?  Can your tool assist you in this process?  If not, you may have a nice IT tool that reports web metrics, not a marketing tool that enables you to optimize your site and landing pages to offer the best possible messages to known online segments.

• Poor Visualization.  Pictures are worth a thousand words.  Your stakeholders are already overwhelmed with data before ever presenting reports with a whole bunch of numbers.  Not everyone is quantitative.  Some stakeholders just want to be able to quickly digest data, and they prefer an aesthetically pleasing visualization instead of a spreadsheet. 

Data visualization helps stakeholders interpret important data at a glance.  Visualization helps reporting comprehension.  Good visualizations are important when you want to:

• Highlight key trends in the data
• Compare counts of things
• Identify multidimensional relationships using cube visualizations

If your tool can’t visualize your web analytics data, and you need that visualized data to assist comprehension, act as sales tool in a presentation, or as marketing collateral in a report, you have outgrown your web analytics tool.

• No Custom Reporting.  An acute inability to deliver customized reporting that meets the needs a diverse group of stakeholders is one of the signs you’ve outgrown your current web analytics tool.  The problem manifests itself in sheer frustration because people can’t get the data they need. Over time this will cause people to lose faith in web analytics because the data isn’t relevant to their jobs.  Some of the symptoms include:
  • Problems creating KPI’s.  To manage online performance, you need to be able to define Key Performance Indicators.  For example, you may want to define a view:visit ratio, that is the number of page views generated per visit.  You need to define this equation in your web analytics tool.  If you can’t define such simple KPI’s, you are limiting your success at web analytics.
  • You may not have reporting that identifies conversion rates and allows you to define custom metrics for channels like RSS, Newsletters, and Internal and External Search.  A powerful web analytics tool will be able to build custom reporting for conversion rates and other KPI’s by online channel.
  • You may have a limited ability to build reports with filtered data, such as viewing reporting of top pages on a particular day or combinations of days, or filtering data by referrer, geography, or time.
  • No ability to add core web analytics dimensions to your reports, such as creating and saving a report that shows all referrers, their keywords, conversion and bounce rates for each city in the United States.
  • Quite simply, you may only have one set of reports and you can’t build new ones at all.

In the real world practice of web analytics, you need a web analytics tool that has the ability to build as many custom reports as you want when doing analysis, to filter, add metrics, to use dimensions, do AB tests, and save all that stuff until your heart is content so you can meet business goals.  You can’t be restrained by the inability of your current tool to create custom reports. If you are you may have outgrown your web analytics tool.

I’ll post part 2 this weekend as I recover from the jet lag…     Part 2 is up, click here to view it!