American’s don’t want to hear about audits in April, so I can only imagine how comScore and Nielsen felt after receiving that letter from the IAB. 

The letter made salient points.  It was “right on.”  While audience panel measurement is a time-tested method for deriving conclusions about “audience,” it must evolve beyond current maturity to fit today’s Internet.  The long tail, cookie usage, and impact on b2b traffic measurement can’t be rolled up into a bunch of people “at-home” visiting one portal site and using one ad network.   “Event” tracking, rich internet, digital video, and connected device measurement make it all even more difficult.

Panel selection must avoid coverage error and selection bias.  And that seems really hard, to me, to do on the modern Internet.  I’ve heard rumors these panels tend to self-select, offer participation based on incentives, or random dial yr digits.  Well, ah, see, I’m too busy. I buy what I want.  I have two cell numbers, three laptops.  Those methods missed me.  And guess what?  I use the Internet too.  A lot. 

So the question must be asked, do existing methods for panel measurement frame judgment samples leading to nonprobable data? 

Well, I don’t really know and I sure hope not.  comScore and Nielsen are good companies, but I’d sure like to learn more about what goes on inside. 

The panel measurement that will “win” in the long run is the company that understands it’s just one, external, metrical input into an online company’s value chain.  The internal metrics are another input (and output), and one much more valuable because I can reconcile them.  If questions arise, I can identify the derivation of my internal numbers and release that data for auditing by IPRO or ABCi or processing by any number of vendors who want my lucre.  External metrics are the Web’s dark matter.

I’ll propose that the smart option for companies with the ambitious goal of measuring the entire Internet for everybody everywhere is to think with “open” in mind.  What that means is take that black box methodology and make sure it’s simply:

-
Radical, I know.  Otherwise, we’re left with the “damned statistics” and the perception of the first part too (lies, lies).  I have no way of knowing if comScore’s recent data is representative of the Internet population or if the data is simply representative of the “frame” they sampled using whatever clever method.  “Trade secret” or “proprietary,” but certainly not standard or open. Even the answers to Eric’s questions left more questions.  

A band named Pavement once sang “questions are the answers to questions in themselves.”  It sure sings true here.