Why Data Rarely Tells the Whole Story

In his book, The Signal and the Noise, statistician and writer Nate Silver said we live in an age where the volume of data available to us has vastly outpaced our knowledge of what to do with it. In no industry is this more evident than web publishing.

Between pageviews, reader engagement, clicks, shares, and numerous other metrics, publishers today have a lot of data to choose from when measuring success. However, measuring value based on individual metrics does not provide a holistic overview of the success of your business — because data rarely tells the whole story.

Prior to my current gig, I taught high-school math and economics in Brownsville, Brooklyn. The NYC Department of Education had just implemented the NYC Progress Report, which used mathematical models to assign a grade to each school in the city.

The program came loaded with controversy, partly because opponents thought the metrics the model used were too focused on test scores. The issue rages on today. The Common Core program has sparked fierce debates, and again, it focuses on what? Measurement.

An aggressive focus on metrics, and particularly on one “golden metric,” creates the illusion that solving for the metric solves for the problem. But, that’s not always the case.

The problem in education is how to help students.

The problem in journalism and media is how to build a sustainable business online. And in publishing, an industry where the value of individual metrics has dropped in recent years, the challenge to measure success in a meaningful way — beyond pageviews or visitors — is what will drive us toward sustainability.

In short, there is no golden metric that tells the whole story of how users are interacting with your website and content. So what metrics actually matter? What role do they play for audience managers as we move  into 2016? How should we approach all the charts and graphs presented by our myriad analytics platforms?

I propose a simple solution: Take a more holistic approach to answering the specific questions you have about your content. Use all of the metrics you have available and combine them in a way that will provide you with insight into specific questions that your site is looking to answer.

This is the type of analysis that site managers who aren’t necessarily data-savvy can perform. Start by making a list of questions you have about your content or website. Then look at your analytics platform to see which available metrics can serve as variables to answer your question.

This analysis will do more to answer the key questions about your business, and you’ll move away from the short-sighted practice of focusing on one “golden metric” to answer everything.

Portions of this post originally appeared in an October 1, 2015 Computerworld article, “The myth of the ‘golden metric,’” by Sachin Kamdar.