Last week I attended the 1st Annual SEMPhonic XChange and led a collaborative discussion on “deploying measurement systems in distributed companies.” In case you hadn’t heard about SEMPhonic, the company is a boutique consulting firm in Novato, CA (near one of my favorite towns on Earth: San Francisco).  They do some very unique work with web analytics (Functionalism) and other facets of new media technology.  A fine gentleman named Gary Angel leads the group. He’s recently hired smart folks, like long-time web analytics expert, June Dershewitz, and notable blogger and author of the Web Analytics Report, Phil Kemelor.

A few months ago the idea came up for me to lead a “huddle” at their Xchange conference.  I was intrigued with this huddle idea.  It was different. New.  I’d facilitate a discussion on a topic of my choice in a small, Socratic, group setting.  No PPT!

Fast forward to last week, and I found myself sitting in Napa, CA at COPIA: The American Center for Wine, Food, and the Arts. COPIA was a brilliant place for an industry colloquy.  Being a guy who likes culture (and conferences), this venue offered the best of both worlds.  Instead of a hotel, I found myself surrounded by gardens, vineyards, art, food, wine, web analytics, and the brightest in the industry.  Cool!

Thinking back on the event, SEMPhonic Xchange, in my opinion, is a “must-attend conference.”  Not only is the “huddle” format unique and fun in which to participate, it’s also a format that promotes deep discussion that leads to truly actionable insights. It’s a conference based on dialog and collaborative discussions between participants.  The huddle format provides 10 hours of free collaborative consulting (5 huddles) with employed people you can’t hire (and some consultants you can)!  Compare a daily consulting rate to the conference cost, and it’s a no brainer.

My discussion on best practices for the successful deployment of measurement systems lasted a little over two hours.  My group collaborated nicely (so I thought).  It included very intelligent folks like Jared from Intuit, Scott and Christel from Xilinx, Sami and Fred from Adobe, Renata and Matthew from American Express, Aaron from Webtrends, Rupa from Cisco, Amy from JPMorganChase, Jeff from New England Journal of Medicine, and Kevin from Charles Schwab.  Thanks to anyone reading this blog who attended.  Your valuable insights and knowledge sharing contributed to the success of the huddle.

At the ends of the huddle I went over a tips for a successful deployment.  Here are a few:

• Identify business goals, match those goals to business strategy, and align metrics and KPI’s to support those goals.  I always say “metrics and kpi’s have significance in position and relation to a goal.”  You can’t measure against performance unless you’ve identified your business goals.  Goals are supported by strategy.  You create KPI’s for measuring against goals and for guiding work toward objectives.  Every business unit could have different goals.  Management must align and support the standardized goals that you bake into your KPI’s (and any specific derivatives you create to support the goals of differentiated business units).
• Verify the technology implementation and the metrics collected.  A web analytics or other measurement system must be verified to ensure conformance with “standard” best practices based on the company’s goals.  You need to make sure the numbers can be trusted.  That means, validating data and reconciling differences with historic reporting systems or whatever the business demands, understands (and possibly pays for in real dollars, time, or resources allocated), such as ABCe, WAA, IAB, MRC, XYZ, 123, FBCINSA whatever it takes…
• Provide education and training.  While some analysts want to keep all the data within their analytics group, I don’t think that practice scales across a large enterprise unless you are lucky enough to have a large analytics team.   I know I can’t individually and alone satisfy the metrics needs for thousands of stakeholders or hundreds of brands.  Successful companies deploying on a large scale adopt a “train the trainer” approach.  The trainers guide their business units, become local experts, and help foment a data-driven culture.   Let the data be free and the people educated, I say. 
• Consider the corporate culture.  A metrics tool changes organizational culture (for the better or worse).  Suddenly, everyone is being measured and perhaps evaluated on goals implicit in the measurements.  Some will greet the tool with open arms while others will see it as a threat.  Solid management needs to foster buy-in and support for the tool.  That way, organizational resistance is overcome and clarity of mission is realized.
• Help business units ”use the metrics” to improve performance.  My friend Eric Peterson likes to say “web analytics is hard.”  Yes it is!  That means the expert needs to work with global business teams to mentor and teach how to separate signal from noise.   As the tool is used, business units will identify “pain points” that will need to be addressed.  The analytics team should work with business units heal the pain and improve performance.
• Plan to ”get granular” and the “get integrated” with the measurement system.   Additional requirements will come out of the woodwork due to organizational learning after you golive (even if you think you’ve elicited all reqs prior to deploying your tool and building reports).  New requirements are an early sign of success (people are “getting it” after all).   As the business learns, it will become necessary to extend the system.  Consider brainstorming about the possible ways in which the system should be extended prior to rollout, and perhaps create a basic plan for extending the system. When the system is launched, modify that system enhancement plan and execute on it to support business goals.  Employ a project manager to plan and help execute!
• Manage and guide stakeholder expectations to minimize risk.  Be realistic when for guiding and setting expectations so that you can minimize risk (to your overall mission, your job, the company ;).  Risk comes from incorrect metrics, organizatonal issues with adoption, changes in business goals, shifting managerial priorities, technology problems, and issues resulting from resource allocation.  Make sure you manage around these issues or find ways to directly deal with them. 
• Generously explain why the measurement system is being deployed and more.  You must create a well-thought out communication plan for promoting adoption and use of a metrics tool.  The communication plan should focus on answering:
  • Why the tool is being deployed.
  • What people are supposed to do with the tool.
  • How people should use the tool.
  • What key metric/KPI’s people and business units should be looking at to manage their performance.
  • How to go from “insight to action” based on analysis of the metrics.
• Share best practices and lessons learned across the enterprise.  As new insights are realized and the company starts taking action from the metrics, you should provide a way for the enterprise to communicate the “highest and best use” of the tool.  By promoting collaboration and knowledge sharing, the company is more likely to succeed quickly with the measurement tool and realize a demonstrable ROI from it.
• Realize that “premature optimization is the root of all bugs.”  When deploying a measurement system, you need to establish a baseline system before extending the system to get more “granular.”  An implementation must be granular before integrating data from other systems.  While these concepts will overlap in deployment and may occur in close proximity or in a waterfall (i.e granularity may be enabled via integration) you need to ensure you don’t put the cart before the horse.  Make sure you correctly measuring and understanding the basics (like recency, frequency, clickstream, referrers, bounced visits, and depth) then moving forward with more advanced and necessary measurements and reporting (like bounce rate, conversions, view thrus, voice of the customer, and ”engagement.”)   Set clear expectations and guidelines with the consultants.  Don’t move too fast with development.  QA, then segment away, I say… ;-) 

A lot of more was discussed and shared over in Napa and throughout the SEMphonic XChange conference.  Please share your thoughts, if you feel like it!  Thanks for reading!

As I explained in Part 1 of this lengthy blog post, companies switch web analytics tools due to a combination of issues related to:

• Price (which I covered in Part 1)
• Features (which I also covered in Part 1)
• Customer Service and Support
• Fulfillment
• The Company Itself (not the vendor, us)

Let’s tackle some thoughts on the the last three (Customer Service and Support, Fulfillment, The Company Itself):

• Customer Service (and the post-hoc judgement of it): 
  • Post sales:
    • Unresponsive or inadequate customer support and professional services.  So many items fall into this category.  From missed phone calls and emails not returned, boilerplate/unfocused proposals,  generalized answers from support to other forms of poor communication, to followup and failures in delivery of service level agreements, to vendors not spending enough time talking to clients after the first year as they do during the first year (as Benry pointed out in his comment to Part 1).  All makes a company begin thinking money and time may be spent better elsewhere…
  • Pre-sales: 
    • Sell them what they, make them buy what they need later.  In this bucket fall items that cause upsetting realizations  that lead to talk about ”the switch”well after the sales process ends - from inadequate professional services implementations to the need to buy another one of the vendor’s products to store all of your data or segment it… 
    • Smoke and mirrors.  Pilots and sales demos look slick but the results of your initial implementation may greatly differ.  After all, your team needs to learn a new technology.  When your team realizes that it will take X months and X hours of dedicated resources to create the same type of functionality you saw in a simple, canned pilot, then the seed for switching is sown.  Keep in mind, vendors tune their pilots for performance and to exhibit the best features they have.  They sell you an executable, documentation, and professional services, not the pilot (no matter what they say… otherwise get in writing). 
• Lack of fulfillment:
  • Implementation problems.  Tagging all of your pages or processing your log files is easier to talk about in meetings than do in real life.  If you’ve spent two years trying to tag your pages across all your companies domains, you may just get frustrated and move on to another solution.      
  • Complexity.  xys_435, this=12r, pass this variable, make this request, do this, do that.  Companies without sufficient expertise inhouse and dedicated resources may get sick of the esoteric proprietary nature of a tool and not want to hire vendor professional services, so they look for technology that is easier to wield using internal skill sets.  Couple that with an acute inability to extend the data model, to use API’s, to access the data in an open database, to decode, and to use lookup tables, and viola, the “switcheroo” gets brought up.
  • Not customizable enough to solve business challenges.  Can’t save reports, add metrics on the fly, use whatever filter you want, extend the data model to include custom business dimensions?  Or you can do it, but it costs too much?  Dang.  Maybe it’s time to take a look at what current technology offers?   
  • Limited ability to integrate cross-channel data.  Isolated silos of data are not good.  Integration is necessary for realizing insights across online and offline channels.  Resistance to this fact is futile.  Products that resist this fact won’t last for long in companies that realize it.
• The Company Itself:
  • New or changing goals.  As a company learns how to “do” web analytics, they may realize the tool is incapable of meeting new goals and move on.
  • Current tool doesn’t fit into business process. As companies engage in Business Process Management (BPM) and Master Data Management (MDM) initiatives and develop service-oriented architectures (SOA), web analytics tools need to fit in.  A company may judge their current tool doesn’t support integration with X or enablement with Y, then begin looking for a new vendor.
  • Organizational alignment. If a company isn’t organized in way that promotes the usage of the analytics tool to achieve business goals, or if the deployment of the tool is not managed by a business person who ensures technical development achieves business goals, then the true value of the analytics technology may never be achieved. Business owners will move on and demand a new web analytics tool.
  • Inability to meet internal service level agreements. If the system doesn’t enable the team to quickly adapt to evolving business demands related to internet measurement, SLA’s may slip,  which leads to thinking about technology change.  
  • New management.  Change comes with new management.  Always.  Inevitably.  Sometimes that change comes from standardizing on a tool that management has had success with in the past. 
  • Limited resources to support tool.  Given a hard to identify ROI, the company may limit resources.  As a result the tool withers on the vine and gets replaced.
  • No EXPERTISE.  With a dearth of qualified, experience web analysts, high salaries, and high demand, a company may just not be able to find expertise needed to maintain the current tool and move to something simpler.

The good news is 38% of “best in class” companies haven’t switched (yet), so some vendors are doing it right for some clients.  And 570 Fortune 1000 companies have no idea what I’m talking about yet at all, so that bodes well for you, the analyst or consultant, and you, the vendor, and us, the industry.

If you have switched and feel like sharing why, please do!  Thanks for reading!

Switching web analytics tools is fairly common (see Part 2).  According to a recent survey by analyst John Lovett (now at Jupiter), research suggests that 62% of “best-in-class” companies are currently on their second or third web analytics vendor platform.  An update of Eric Peterson’s Vendor Discovery Tool, due out any day now, indicates that only 43% of the Fortune 1000 have even deployed web analytics technology.  That means only 430 Fortune 1000 companies have deployed web analytics tools.  If Fortune 1000 companies are considered “best-in-class,” then, ouch, web analytics vendors could really improve customer satisfaction and retention!  My friend Robbin Steif over at LunaMetrics recently asked ”why?”  Why do companies switch web analytics tools so often?

The answer is that every company reaches a threshold moment of judgment about a particular technology - the instance the decision is made to either keep the technology or ”move on” and begin searching for new technology to replace the old.  This day of reckoning results from a combination of issues related to:

• Price
• Features
• Customer Service and Supported (covered in Part 2)
• Fulfillment (covered in Part 2)
• The Company Itself (not the vendor, us, and covered in Part 2)

The same goes for web analytic’s tools.  Challenges related to these five items fester until they cause a company to begin thinking about switching.   For example (the list continues in Part 2):

• Price:
  • Licensing or Subscription Costs.  Whether your running an in-house or hosted web analytics environment, it costs money.  Companies care more about things like the capital budgeting, debt service, and cash flow than they do about web analytics tools.  In times of limited resources or when the tool isn’t producing the desired return for a given cost, change may occur.
  • Per seat costs.  Every person who accesses the web analytics application may be thought of as having “a seat.”  How many seats do you need for your team?  your company?  The answer really depends on the scale of your implementation.  Seats aren’t inexpensive, and if you haven’t considered how many people in your organization need access the tool, you may soon find you can’t scale your implementation without significant revenue expenditures.  Per seat costs are generally factored into overall costs (sometimes as a line item).
  • Maintenance or upgrading costs.  Version upgrades or closely-related new products may not be “free.”  And that may make companies, that already feel limited with their current implementation, very ”unhappy” and cause “the switch.”
  • Costs for specific features.  Vendor’s basic pricing-models limit access to data filtering and drill-down features.  Slicing data across more than one dimension may require additional components or entirely new products, which have cost.  Some companies don’t dig deep enough during due diligence, only to buy a tool that they know can show “keywords per page” but hasn’t been configured or costs additional money to do so.   
• Features:
  • Limited features for custom reporting, segmentation, and ad/post hoc analysis by the business user.  Similar to the bullet above, Companies may offer a simple reporting tool with a limited number of ways to filter, segment, or correlate data.  To do more, the company may need to purchase additional features or completely new products not in the budget.  To lower cost, the company starts looking elsewhere for a comparable service. For example, time spent metrics are easy to calculate.  They’ve been in log files for years.  But don’t assume every page tag vendor has the ability to calculate these metrics out-of-the-box.
  • Inadequate data storage. Companies may not have asked “how long is my raw data stored” or if the data after a certain period of time.  It’s a real bummer when you want to backtest and realize you don’t have the data you need. Take for example, URL length, some companies may trim your URL’s unless you ask them not to.  You won’t know that until you realize it’s a problem.  At that point, not having historical data to test may cause managers to question the tool’s capabilities and to start thinking about the switch.
  • Limited data model, lookups, and decodes.  If the company wants deeper insights across more dimensions, the current product may have a limited data model that isn’t extensible to your business.  Decoding parameter or using lookup tables might not be possible.  The model may not enable you to track and report on things like Events, Spider/Bots, IP addresses, and file downloads.  If you can’t extend the schema, you limit your ability to use custom business dimensions already existing in your data warehouse or in other applications. 
  • Closed systems with no API’s or methods for rewriting data.  No API or methods beyond simple exporting of filetypes for getting data out of the system limits companies from achieving a 360 view of online customer experience.  If you want to do cross-channel marketing merging print and online, a simple reporting tool will not suffice (no matter how pretty the reports and trendlines). Companies that wake up to this fact, switch. 
  • Data not separate from presentation.  To integrate data across systems you need to have access to it.  For such needs, data is best in an open database, not only available via a web interface or on a “pay-to-use” basis.   Companies that wake up to this fact, switch.    

This list continues in Part 2.  Please click here to begin reading Part 2!

Is web analytics data normally distributed?  That question calls for another question: what web analytics variables are you measuring?  That matters.  Numeric random variables (let’s call them data) are classified into the following types:

• Discrete.  That means you count it.  The data arrives from a counting process.  In web analytics discrete random variables are counts of things like page views, visits, and unique visitors
• Continuous.  That means you measure it.  The data arrives from a measurement process.  In web analytics continuous random variables are time-based metrics.

We do both in web analytics, don’t we?  We count some things.  We measure some stuff.  And if we’re smart and have the autonomy and positional power to do so, we apply process to counting and measuring web analytics data. 

We often talk about “counting” and “measuring” like they are the same activities.  In general day-to-day online business, that’s no big deal for conceptual conversations.  But in statistics, “counting” is different than “measuring.” 

Both discrete and continuous variables may be represented by probability distributions to assess the liklihood of an outcome.  To identify probability for discrete variables, use a “binomial distribution.”  Binomial distributions take into account the probability that an outcome will occur, so you may see some skewing when plotting the data that may make it look a bit “long tail.” 

For continous random variables use the “normal distribution.” Realize your data won’t always look exactly like a bell curve.  If it looks really different and ”long tail” you may be looking at a discrete variable better suited for a binomial distribution.    

So is web analytics data “normally distributed?”  The answer is that it depends on the type of data.  Even then, the answer is “probably not.” In fact, most business data doesn’t follow a perfectly normal distribution.  Yet every day in halls of academia, very book smart people teach statistics and tell you to apply it to business data.  Are they wrong?  Insane?  Misguided?

No they aren’t (well maybe you have to be slightly insane to teach stats).  Academics realize that most distributions are not normal and do not have equal measures of central tendency (i.e. mode, median, mean).  Skewness abounds!  The normal distribution, however, can be used to approximate “real-world” distributions that have different measures of central tendency. 

A theory called the “central limit theorem” states that “if the sum of the variables has a finite variance, then it will be approximately normally distributed ( i.e., following a normal or Gaussian distribution).” In other words as the sample gets larger the distribution of the mean can be approximated by the normal distribution.  And if I remember correctly statisticians have determined that with a sample size of at least thirty, the sample distribution of the mean will be approximately normal.   Fortunately, we web analysts often have millions of data points to use…

Some time ago I actually took average visit duration for one site for which I have real data for thousands of visits and did a Lilliefors Test of Normality.  The test found no evidence that the data wasn’t normally distributed even though it looked a bit odd and the skewness was 0.741426 and the kurtosis was 4.1525665. 

If you’re thinking about applying statistics to web data, make sure you identify whether the data you are looking at is discrete or continuous.  Don’t abandon the normal distribution for certain types of web analytics data just because it doesn’t exactly look like the Liberty Bell.  Test it for normality before applying the Gaussian statistics.  If the data is highly skewed, determine whether the level of error is in acceptable limits.  Look at using other distributions for discrete variables.  

Image from http://www.weibull.com/

In Part 1, I began a long list of reasons why your web analytics data quality may stink.  I’m continuing the list below (make sure you read Part 1 for context and to view the entire list): 

• Storing only visit level data.  May tools don’t have schemas that store raw data at the visitor level.  Instead they provide access to only visit level data.  For example, you may not be able to see all the page views during a single visit per ip address or cookied visitor.   Assess the impact of the vendor’s schema on your goals.  Companies that use analytics data to feed other systems or that want to use visitor attributes for content targeting, segmentation, optimization, or analysis may not be well-served by some vendor schemas.
• Little to no decodes or lookups.  If you use numeric codes and non-human readable naming conventions in your data, they can pass through to your reporting and prevent your colleagues from understanding the reporting.  Strange codes look like hieroglyphics!  Decoding and looking up data can eliminate the problem of non-readability and strange numerical names in your reporting… While some would say this is a reporting issue, not a data issue,  I chose to include it because it’s at the surface… it’s the data your customers see.  Not all tools decode or lookup.  Some tools allow rewriting of data in the database.
• Failure of key services supporting the application.  If you are dependent on page tags, synchronization software, web servers, databases, or any of the wondrous technology that makes it all work, failures are a real bummer.  Make sure you have monitoring and recovery processes in place so you don’t miss data!  When page tag collection fails (perhaps the page tag server went down ay?), the data is gone forever.  If the web server fails, then no logs are written, but no pages are served either - so is traffic missed?  But if the processes supporting log file analysis fail (i.e. data synch), watch out! 
• Inadequate or incorrect implementation.  If you can’t cross dimensions (like finding out what keywords referred traffic to a page), filter all of your data (for example, filtering pages to see only those viewed by the iPhone), easily create new metrics, or if the numbers aren’t adding up, you may have not adequately or correctly implemented your software or communicated your requirements to your vendor’s professional services team. 
• Limited, hard-to-extend data model. Powerful, actionable insights from web analytics are enabled by extending a data model to incorporate business specific dimensions.  For example, if every page has a category and an author, you may want to see a list of all the page views in that category or ranking of pages by most popular author.  To do that you may need to join data at the database level or take advantage of variables you pass in a page tag.  Various tools have different limits on if, how, and to what extent you can extend the data model.

So what do you do when you know your data quality is less than stellar?  Here’s some guidance:

• Don’t worry, be happy. Just by collecting the data you are collecting, you are doing better than a great majority of companies that do business on the Internet.  By asking questions about data and investigating the issues, you have a leg up on your competition.  Work on optimizing the data, expose flaws in site design or architecture that impede data collection, work with your vendor and seek help in the web analytics community if you run into real problems.  The Web Analytics Association’s Forum on Yahoo is a useful place for posting questions.  But whatever you do, stay positive and focused on solving your problems and making your web analytics practice more optimized.  Don’t get frustrated.
• Recognize the limitations in the data and do not go gently into the night.  Ask the hard questions about sampling, schemas, data retention, processing, querying and reporting to understand where the holes and noise could be in your data.  Demand answers from your vendors and quick response times to your questions about data quality.  If you vendor is frustrating you by not being responsive, talk to the boss and the vendor’s bosses, escalate, escalate, escalate until you get resolution.
• Understand the underlying elements of data collection and what can go wrong.  Learn about sessionization and why different tools and data collection methods have limitations.  Explore the more technical components of the backend, like the database and your web analytics schema - all your data is in one (or more)! Talk to your engineers.  Have them explain the technology in terms you understand.
• Evaluate your tools.  Some tools are just better suited for particular business problems than other tools.  Log files tools enable you to constantly change assumptions and reprocess data.  Page tags provide a standard data collection and transport mechanism.

With hard work on your part, you can make you web analytics data smell like roses!  I know you can!

When you want to spend capital to purchase a web analytics tool, the decision is never easy.  There is no standard, scientific way to select a web analytics vendor and tool - it’s a bit of an art.  The decision is full of compromises - no one tool  or fancy family of tools from one brand will be able to do everything you think or want them to be able to do.  Nor will any one tool have all the bells and whistles you want.  

Lots of resources exist for helping you select a web analytics tool and vendor - from Marketing Sherpa to CMS Watch.   Even with good resources, it’s still tough to narrow down the selection and really identify what’s important.  I figured I’d blog on some of my thinking about the subject to add to the knowledge base in our industry.   And I figured I’d call it the “Phillips Framework” because 1) a friend told me not to share it without attribution 2) cool phonetic alliteration.

The first thing I’d recommend before beginning the due diligence process  is asking yourself or your boss the following (relatively) simple questions:

• How much money can I spend?
• What resources do I have?
• Do I have the organizational capability and maturity to run an in-house software solution?
• Do I prefer to eliminate overhead and technology expense by delegating control of my web analytics technology and infrastructure to a hosted solution run out of a vendor’s data center? 
• Do I want to integrate web analytics data with data from offline systems? If so, what systems and what methods (i.e. web services)?

You’ll have a short list of potential vendors rather quickly.  I would recommend framing your vendor evaluation across these dimensions in the context of how they are relevant to your business needs and goals:

1. Company and Technology
2. Product and/or Services
3. Features
4. Vendor Organization
5. Strategic Fit
6. Cost

Since this is Part 1, let’s discuss the first dimension of the assessment:  Company and Technology

I’d recommend creating a matrix so that the attributes presented below are on the left axis and the companies you are selecting are on the top axis.  Fill in the cells with your custom information evaluating a vendor:

Useful attributes for beginning your evaluation of a potential company and technology for web analytics include:

• Company Description. Describe the company using publicly available sources.  How long has the company existed?  How solvent is it?  What do customers say about the company?
• General Technology Description. Explain the technology and how it works. If technology uses OLAP, what happens to the confidence level and confidence interval (i.e. margin of error) when drilling down on the data?  Can I report on every dimension and attribute of available data about a segment or is the reporting limited?  How about when exporting?  
• Product and Service Capabilities. Assess the overall ability of the vendor’s technology and services organization when compared to the industry. What percentage of the company’s customers successfully deploy tags and get complete tag coverage across every page from day one?  Or successfully transfer and correctly parse customized log files from day one?
• Product(s) Required for Solution. List the product or products required to support the full solution. Can I run identical queries and get identical answers across all company technologies?
• Ease of Use. Indicate the complexity of interacting with and navigating through the interface and reports.   Assess the user experience of the GUI from usability and information architecture perspectives. Can I simply find the data I need to gain analytic momentum?
• Product Updates and Difficulty. Indicate difficulty of product updates and general migration path for upgrades. Does taking advantage of new functionality in a release usually require upgrading the code throughout my web site? 
• Real-time reporting latency. Identify the delay or lag in availability of the data within the technology.  Continuous processing?  Batch?
• Time to Implementation.  Indicate the time to deploy the baseline, out-of-the-box solution. What percentage of the company’s customers have successfully tagged all site pages and/or processed logs within 1 month after beginning? 3 month? 6 months? 
• Ease of Implementation. Indicate the difficulty level of implementing the technology. What percentage of the company’s application can I use if no changes are made to the javascript page tag?
• Data Collection Model. Identify data collection methods.  Does the company’s data schema simply rollup and report “unique” counts across time periods and delete the underlying data (even if I don’t buy an additional product)?  Does it cost more money to retain full, unsummarized visitor data for 12 months? 24? Longer?
• Data Retention and Ownership. Indicate if I retain ownership of my data.  If so for how long and at what level of granularity? For what duration does the company retain visitor data?  Is that the same across all applications (not just a data warehousing component)?
• Integration. Identify features and methods for integration with external systems.  API? Web services? Summary extracts?  Just Excel?
• Innovation. Indicate the level of innovation perceived by looking into the company when compared to industry competitors.  What do the analysts say?  How large is the company’s engineering organization?  What percentage of overall expense does the company spend on R&D?  Partnerships?
• Security. Identify the security model. Does the tool support integration with Active Directory or LDAP?  What is cost per seat or license?
• Segmentation.  Identify the flexibility and ease of segmenting data.  What is the total, maximum number of segments available for use “out of the box?”  How much more does it cost if I want to increase segments or filters?

More attributes exist.  More questions should be asked. 

Truly understanding a web analytics technology means asking hard questions and assessing the way a company answers those questions to frame your subsequent analysis and guide your selection. 

I’ll blog on part 2 in the future, probably as a monthly series over the next 5 months.  Thanks for reading!