As the ad-supported side of the video business looks to supplement panels with much larger data sets, two alternatives have come to the fore: set-top box data and automatic content recognition (ACR) data from smart TVs.

While each data set has its pros and cons, the fact remains that the overall number of set-top boxes is shrinking at a steady clip, while the number of smart TVs is growing at a similarly steady rate, and this is likely to make measurement much trickier to calculate in the years to come.

Let’s start by looking at the value each data set adds.

ACR data from smart TVs provides a second-by-second look at anything that is “on the glass” regardless of input. That makes it an ideal measurement tool for right now in that it can measure both streaming and linear viewing, not to mention over the air. This is key because most viewers are going to be watching both linear and streaming for many years to come.

As per a recent study from Leichtman Research, 71% of U.S. households still have a pay TV subscription of some sort. (That number includes vMVPD subscriptions.)  So while cord-cutting is indeed increasing, it’s still only at a single-digit pace and there’s likely to be a floor too—at TVREV, we’re thinking that somewhere between 30% and 40% of households are never going to willingly give up traditional pay TV.

This means that the video industry will need to measure both linear and streaming for the foreseeable future, at least the next five to 10 years, and ACR data is uniquely positioned to provide that sort of measurement.

On the flip side, ACR data comes from a single TV, not from every TV in the household, which is where set-top box data comes in. Since all of the set-top boxes in a household are connected and track to a single account, it’s easy to measure what that household is viewing.

The downside to set-top box data is that it often reports viewing that happened after the user has shut off the TV and gone to sleep as the user has only shut off the TV set and not the set-top box.

There’s also the issue that if the user has a smart TV, the set-top box and the smart TV are both collecting data on the same viewer. This is why the combined data sets must be cleaned up (de-duplicated) and why one can be used to confirm the other.

Set-tops Boxes Down, Smart TVs Up

The aforementioned Leichtman report also found that just 37% of all TV sets in use today have a traditional pay-TV providers’ set-top box attached– a massive decline from the 58% figure Leichtman recorded in 2016,

Compare that stat to an April 2021 study from Hub Research that found that seven out of 10 U.S. households own at least one smart TV and that 52% of TVs are smart TVs, an increase of 7% from 2020.

Those numbers will only be decreasing (set top boxes) and increasing (smart TVs) in the years to come.

Set-top box numbers are shrinking because the various MVPDs are slowly but surely moving away from them. Set top boxes are notoriously expensive, difficult to upgrade, and the cause of much consumer unhappiness due to missed or delayed tech appointments.

Instead, MVPDs are turning to apps, which can easily be installed on smart TVs and connected devices, including connected devices they themselves provide to their customers in lieu of set top boxes.

Even Comcast, which has been justifiably proud of its X1 set-top box interface, has been slowly shifting away from set top boxes. They’ve made the X1 the basis of their new Xfinity Flex streaming device and they are going to be rolling out their own Comcast branded smart TVs whose user interface will be based on X1.

All of which means that the future of set-top boxes is looking increasingly murky and would seem to be heading further in a direction that is the basis of another common slam, which is that set top box data measures an audience that is increasingly older and less affluent than the general TV viewing population and is thus increasingly less relevant.

Smart TVs, on the other hand, are on the upswing.

Amazon has also rolled out its own line of smart TV sets and Google is pushing Android-branded sets as well.

More than that, it’s all but impossible to buy a new TV set these days that is not a smart TV. So as consumers are replacing their older unconnected TVs, they’re replacing them with smart TVs. That means major ACR providers like LG, Samsung, Vizio and Roku (TCL) are going to be seeing their user bases growing steadily over the next several years. (The average replacement cycle for a TV in the U.S. is around seven years.)

The demise of the dongle is also fueling the growth of smart TVs. While the dongle is far from dead right now, the main reasons to turn to a Roku or Amazon stick—better interface and larger app selection—are no longer valid as smart TV OEMs have dramatically upped their interface games. Roku and Amazon themselves are putting more effort into their own smart TVs too, as they realize that a $300 TV is not as readily swapped out for a competitor as a $29 dongle.

ACR data is not without its own flaws however. Only two of the companies that collect ACR data—VIZIO and Gracenote—actually license that data to third parties, which greatly reduces the data set that can be used to create measurement stats. (The remaining OEMs use the data for their own purposes, to support their ad sales teams and their FASTs.)

In addition, many, if not most households do not exclusively have one brand of TV in the house. That, critics note, can make it difficult to ascertain household data in households with multiple TVs, especially if one TV is the main set and/or if different family members exclusively use different TVs.

All that said, the more viewing that happens on smart TVs, the more valuable ACR data will be as it captures both streaming and linear. While set-top box data has appeal as both a check on ACR data and as a way to understand household viewing, the shrinking size of the set top box database, along with the fact that it does not measure most streaming viewing both serve to lessen its overall value for measurement purposes.

The result is that TV viewing is going to become trickier to accurately measure in the years to come, which is why those companies who are able to figure it out will find themselves in the proverbial catbird seat.

TV measurement is at a crossroads.

Viewers are watching TV (or TV-like content) in more places than ever before: traditional linear TV, streaming TV, VOD, and on social platforms too, where premium video—everything from originals to clips of The Tonight Show—constitutes an increasingly expanding piece of the pie.

For the brands and agencies that service them, this has created a conundrum: There are more places to reach their audiences with high emotional impact video ads than ever before. It’s just that doing any sort of measurement is tricky.

The problem stems from the fact that linear TV, streaming TV and social TV have all traditionally been measured by very different standards and systems ranging from panels (linear) to self-reported views (streaming).

This is not of any real use to anyone, and if anything, it holds the industry back, as brands do not want to spend money on platforms when they can’t accurately determine the ROI of their investments.

Nielsen, the industry default for measurement, has struggled for years to try and figure out the new digital-based ecosystem without much success. That’s why the industry is now turning its attention to a new group of up-and-coming players who are making strides to bridge the gap between the various platforms while improving the quality of the data, the usefulness of the metrics and the level of privacy compliance.

And if that sounds like a difficult task, that’s because it is, and the number of successful companies in the space is gradually being winnowed down to a select few, three of which had major announcements this past month.

First up is Tubular Labs, which has made its name measuring video on Facebook, YouTube and other social platforms. Tubular just announced that it will be rolling out its own version of the Gross Rating Point or GRP, which wast designed specifically for social video.

The goal is to allow advertisers and their agencies to do more of an apples to apples comparison of their buys on social video versus the same target audience on other publishers as well as on TV.

While social video may call to mind cute cats and puppies, platforms like YouTube and Facebook have become major outlets for premium video content from the major TV networks. (Think of all the Saturday Night Live clips you’ve watched on YouTube.)  

Advertisers like social video because it’s very easy to target specific audiences on the platforms and because they are unable to reach those viewers elsewhere in the TV universe. With Tubular Labs’ new GRP ratings, they will finally be able to measure the effectiveness of their spend against those audiences in a way that makes it easy to compare both publishers and platforms. 

The need to be able to measure video across a range of platforms using a range of inputs is why VideoAmp, another of the new breed of measurement companies, received a sizable cash infusion this month, in the form of a $275 million Series F fund raise based on a $1.4 billion evaluation.

VideoAmp measures viewing across a range of sources—linear TV, streaming TV and social—and is one of the companies that have been attracting attention as the industry is looking at alternatives to Nielsen’s traditional panel-based measurement. VideoAmp and other up and coming players pull their data from a much broader pool of inputs, including ACR (automatic content recognition) data from smart TVs, set top box data from MVPDs along with Experian and other data that allows them to track attribution.

Speaking of which, attribution was no doubt a key factor in Viant’s decision to partner with iSpot, another prominent member of the new breed of measurement companies. 

iSpot, whose focus is on real-time ad attribution and measurement, will be providing reach, frequency and business outcome/attribution insights for Viant’s Adelphic DSP (demand side platform) on linear and CTV campaigns, including second-by-second viewership data for commercials.

Attribution, or multi-touch attribution in particular, is increasingly important to brands as it allows them to understand the role different campaigns and different platforms play in moving consumers through the sales funnel.

All three announcements represent positive signs that the greater television industry is realizing that in order to grow it will need to drastically change the way it approaches measurement in the digital era.

The goal, many TV executives have told me, is to make TV data every bit as useful and granular as digital data, which will eliminate the advantage that digital display advertising has long had over TV in terms of the type of reporting brands and agencies were able to get. 

At which point, they feel, the greater emotional impact of the sight, sound and motion offered by video advertising will win out.

Or to quote an oft used adage, “People remember TV commercials they saw twenty years ago. They don’t remember banner ads they saw twenty minutes ago.”

Which is not to say that everything is sunbeams and unicorns.

The industry will need to overcome a tangled web of walled gardens and incompatible and often out of date data sets, the lack of a universal data pool and a rapidly growing host of privacy concerns.

And that’s before tackling the issue that TV viewing is often a group activity that is measured by households and the difficulty of translating those “households” into “persons.”

That is why it’s heartening to see both increased innovation, increased partnership agreements and, of course, increased infusions of cash, all three of which are necessary to solve the industry’s measurement conundrum. 

If there’s one phrase that sums up the state of the media industry these days, it might be: Out with the old, and in with the new. For the first time ever, the industry is speaking in unison and saying we need a new direction in measurement. The antiquated monopoly built for the needs of the 1950s is finally being set aside for a modern approach.   

In a recent post on this site, Nielsen took the curious position that, in a world where concepts like machine learning and blockchain are redefining entire sectors of the global economy, big data and new approaches are something they fear. Perhaps this false agenda is because Nielsen has not been able to incorporate large-scale data into their own methodology in a credible way. They continue to rely on small panels with single-digit response rates that ignore most of the population as the foundation of their television measurements. It’s natural that they would try to refute modern approaches when they are so tied to the past and legacy methods. Simply proclaiming that their panels are “sources of truth” doesn’t make it so. It is good to hear Nielsen acknowledge Comscore as the alternative to this old measurement approach. Our more than 3,000 worldwide customers would agree with them. With Nielsen’s limited experience in this new world, it is misguided for them to discount Comscore’s more than a decade of experience and knowledge in creating stable, reliable, and predictable local and national measurement currencies using census-scale data.   

Given the extent of the fallacies and mischaracterizations in Nielsen’s commentary, it would be tedious to do a point-by-point rebuttal. Instead, let’s take a step back and look at the big picture, in a particularly illuminating example. When the COVID-19 pandemic changed our world in 2020, the effect of it on television viewing was evaluated by Nielsen using small panels, and by Comscore using big-data assets. Nielsen concluded that TV viewing declined during the pandemic, whereas Comscore found the opposite: a substantial increase in TV and connected TV consumption. One was right and one was wrong. Given that history, why would anyone believe Nielsen’s self-proclaimed “source of truth”? The biggest event in generations was measured incorrectly by small panels. 

What is really important when it comes to understanding the future of media measurement is transparency and collaboration. The shift in distribution platforms and advertisers’ interest in audiences — going beyond age and gender demos — requires modern thinking, modern data assets, and modern technology. Advertisers want to use actual, observed consumer information to evaluate their media choices. This approach, based on census-scale data, is the norm in digital, OTT, addressable, and connected TV (CTV) measurement. Nielsen’s commentary challenged the representativeness of return-path data (RPD), but in fact, the data comes directly from subscribers of certain TV services, without a need for high-touch recruitment. Nielsen ought to understand this since they too have begun to use RPD in their measurements. The problem is they are still relying on their unstable small panels as their starting point and not on the tens of millions of household actual observed viewing as Comscore does. Indeed, the RPD data that Comscore uses, comprising more than 35 million households and 70 million devices across all 210 local markets, aligns quite well both geographically and demographically with the country as a whole. These are the facts.

Contrast these facts with the reality of small panels today: most people who are invited to participate refuse to do so, and a few tens of thousands of respondents are inadequate to measure the fragmented media space of the world today. Small panels were a great idea 70 years ago. Today, not so much.

Can Nielsen really refer to their panel as “robust” when their national and local ratings accreditation was suspended after months of scrutiny in large part due to the unreliability of said panel? 

There are fundamental challenges when it comes to being dependent on small panels, something I covered more extensively in a recent blog post. The most obvious problem is the low response rates. This means you are only measuring the rare people who are willing to participate; nothing is known about the refusers (and we know today that most consumers don’t want to be hassled with). Second, ongoing media fragmentation means it is harder to measure with a small panel. For example, a 1 rating is increasingly hard to achieve, and with a panel of, say, 50,000 households, that rating is based on only 500 of them. And it only goes down from there. The resulting estimates are unstable, fluctuating, and unreliable – especially at the local level – with a plethora of “zero cells” where the content audience is too small to be seen on the panel. With billions of cumulative dollars at stake, can advertisers afford to have such a limited and flawed audience measurement? More and more, the industry is saying no. 

The Nielsen commentary suggests that big data and modern methodology cannot measure who is watching what as well as a small set of personal people meter (PPM) panelists. If that were true, then a comparison of Comscore’s estimates to PPM estimates would demonstrate it. In  fact, for content that has a large enough audience that it can be measured by a small panel, the two approaches line up well, as seen for example in the figure below. My colleague Kumar Rao and I have described our methodology in more detail here.

Preparing for the Future of Media

It’s clear we are in a new era of media. To meet the needs of what the industry needs now, not years from now, Comscore has taken a measurement approach with the imperatives of reliability, inclusion, and stability in mind. We are proud to have built what we believe is the best methodology in the market. We have enormous scale – there is one Comscore-measured home in every three U.S. TV homes – and a data-driven/census-like approach that’s impossible to achieve with small samples.

Simply put, we believe that we provide the best insights into how today’s modern, cross-platform audiences consume media. We are proud that more than 1,000 local stations, 150 national networks, 50 station groups, and the world’s 10 largest advertising and media agencies trust us to provide them with actionable audience measurement that drives better business outcomes. 

Nielsen can continue down its lonely path of being a company unwilling to embrace new technologies. Comscore is focused on the future, not the past; after all, that’s what the industry is demanding. As we enter the next generation of media, where agencies and advertisers are taking an audience-centric approach to planning and buying across myriad platforms and requiring impressions to evaluate their success, Comscore believes the time has come to redefine "currency." The future of media has arrived.  

There’s been a lot of energy and excitement in media circles of late about the future of measurement and the promise of big data. At Nielsen, we’ve long understood the value of big data, in fact just last month we announced additional details around how we are adding it to our national TV measurement service. 

We also know that no panel is perfect, as the past few months have demonstrated. 

But when our teams of data scientists hear some of the big, broad claims about big data coming to save the day and fix all the perceived challenges in the industry, it’s hard not to be skeptical.

That’s because, for all its value and amazing potential, the big data sets that the industry currently has access to have very real limitations. 

A Relevant Recent Example

After losing access to Nielsen’s Portable People Meters, Comscore reported that it will now be using data sets from Experian’s ConsumerView to help them identify individual viewers for measurement purposes. Their announcement was framed in the trade press as an advancement — after all, if big data is the future, any shift in that direction must be a good thing. 

Unfortunately for their customers, and for consumers, that’s not the case. 

There are a handful of third-party identity vendors out there who provide the ability to match data sets based on personally identifiable information and provide demographic characteristics, both directly collected and modeled. 

At Nielsen, we regularly check this data. We do it by directly measuring information from our robust panels to validate how accurate these data sets are by: 1) correctly matching to a household and 2) accurately reporting demographics and characteristics. 

What we typically find should give advertisers pause. 

The majority of data sets out there today are built around billing information or online behavior collection, not demographic profiles. They don’t have the rich details about exactly who the people are on their lists — from age, to income to race and ethnicity — the way you do with a robust panel. These data sets, because they’re created by machine-to-machine transfers, also increase the possibility of waste and fraud. 

Because of that, the level of certainty they can provide around who actually lives in a given household is limited. And they have no ability to say who within a given home is watching a given program at a specific time. 

Even when you triangulate that data with other sources, you’re almost guaranteed to have massive gaps and errors in your estimates. This may be acceptable if the use case is targeting, but this data on its own does not provide the accuracy, objectivity and transparency required to deliver measurement. 

Why It Matters

So what does that mean, practically? Well, it has a few implications. 

In the case of Comscore, a shift away from our Personal People Meters, which actually affix microphones to about 100,000 real-life, verified people and track exactly what they’re watching, to a model that uses billing data to provide guesstimates of who within a dwelling might be watching a given program at a given time, will result in a less-accurate read on who is watching what. 

But the possibly bigger implication is that this shift is going to get the industry further away from capturing a true representation of the country. 

We know that many of these types of data sets do a better job of providing data around households when the people living there own their own home and have been there for a long time. And that stands to reason. The problem with that is that long-time homeowners tend to be more White, more affluent and significantly older than the nation as a whole. By design. these data sets undercount Black and Brown people, lower income people and younger people at a time when all of those segments are growing, not shrinking. 

The same is true of data sets built off of set-top box data, which tends to overcount more affluent consumers who are willing to pay more for cable packages and thus disproportionately excludes lower income consumers who are important targets for many marketers. 

The media industry has, rightly, made accurately representing Black and Brown communities a central priority. At Nielsen, our track record on this going back decades hasn’t been perfect, but today we have the most accurate and advanced view of the nation as it truly is. 

Big data-derived measurement tools that aren’t backed by a representative, validated and audited panel can’t make that claim. Nielsen panels can target many demographics within the U.S. Census with 1% variability, but the big data-focused options out there aren’t even close to that. The industry needs to be open and honest with itself about the challenges that big data presents when it comes to representation.

A Wider Problem

To be clear, this is not just a Comscore issue. This is an issue with all the big data sets out there currently. 

In August of 2020 the Association of National Advertisers, in partnership with the Media Research Council and Sequent Partners, used Nielsen data as a benchmark in a study designed to understand the degree to which the multicultural audiences were being accurately represented in media targeting. The study looked at an aggregated collection of high-quality marketing and media data and sought to understand how accurately it was targeting Black, Brown and Asian audiences. The findings were troubling, but not at all surprising to us. 

The study found that the big data sets the industry relies on weren’t up to the task of accurately targeting these critical communities. In part because the data sets weren’t designed to capture rich data about who these consumers truly are, the way robust panels are, there was rampant misrepresentation and underrepresentation in the data. 

Now contrast that with Nielsen's robust panels, which provide a wealth of directly collected information from real-life people, representative of the entire U.S. population. Who lives in the home? How old are they? What race and ethnicity do they identify as? Who is watching the television at a given point in time? Nielsen's panel answers these questions. 

Again, panels on their own aren’t perfect, but there’s a reason other industries, namely pharmaceuticals, use approaches that are similar to panels in approving drugs. That’s because, when the stakes are high, there’s no substitute for real, verified people.  

We know that many industry players are excited about the promise of big data, we are too. But as an industry we need to be honest about what big data can and can’t solve for. And we too understand that the future of media measurement is an approach that combines the reach of big data with the verified personal data of robust panels.

Against a backdrop of Nielsen issues with TV measurement and ongoing advertiser disaffection with the way all media are measured, the Association of National Advertisers has come out with some baseline requirements for measurement.

ANA‘s Media Leadership Growth Council and Cross Media Measurement initiative steering committee together issued the following to "clarify" what its members are looking for:

1. Given that their marketing decision affect billions of dollars worth of transactions, they want “a measurement system that is objective, independent, transparent, neutral, and third-party verified,” which means, given the technology and intelligence available, “a more robust measurement ecosystem that delivers on these aspirations.”

Also Read: Nielsen’s David Kenny Fights Back After Accreditation Loss

2. The Media Rating Council is “indispensable,” as is its third-party accreditation to "justify" the billions of ad dollars spent on paid media annually.

3. Also indispensable is cross-media measurement (CMM), with the ANA saying it is “imperative’ that CMM “becomes the paramount objective of the measurement community.”

4. ANA said the move to advance a focus on measurement (which the Nielsen issues have put a spotlight on), should be led “by a consortium of trade bodies that represent the various industry constituents — including those organizations that are small to medium-sized businesses." ANA said there has to be a commitment to "real change” and the resources needed to overcome past barriers to sound measurement.

5. New measurement standards must be “open to all,” regardless of the size of the media spend.

Telsey Advisory Group media analyst Tom Eagan initiated coverage of the media measurement industry Thursday, slapping an “outperform” rating on Nielsen and Rentrak and gauging comScore a “market perform.”

In his 32-page report, Eagan noted that it has been a tough time for broadcast and cable ratings – broadcast viewership has declined in 10 of the past 14 months at a rate below the compound annual growth rate of the sector for the past four years and cable ratings were down 9% in 2014, triple the rate of the previous year. While it appears that ratings decline is accelerating, Eagan wrote that makes it an even more critical time for measurement.

Several companies are trying to fill the traditional TV ratings gap as more and more viewers watch content via SVOD services, digital video recorders and through online and mobile vdevices and Eagan wrote that developing cross platform solutions will be critical to the industry’s continued health.

While Nielsen (price target $53) has been taking it on the chin lately, Eagan sees upside for the measurement giant, following the momentum of several new service launches and expects the company to generate mid-single digit revenue gain through 2016.

“With Nielsen, the Street is concerned that the new measurement services offered by its fast-moving competitors will eat into its market share,” Eagan wrote. “We believe a more likely scenario is that the measurement market will continue to expand.”

With new products like MMX Multi-Platform, MMX Mobile and vCE, Eagan expects comScore (price target $53) to grow 2015 revenue by 14.5%, adding that greater adoption should boost margins from 22.5% in 2014 to 24.3% in 2015.

At Rentrak (price target $70), which nabbed an equity partner in ad giant WPP Group last year, Eagan wrote that more deals may be on the way.

“We would not be surprised to see other investment in or joint ventures with measurement companies as their increased role in the advertising eco-system is better understood,” he wrote.

Eagan a ls anticipates that Rentrak, which he says has sector-leading revenue growth, will continue that trajectory, with sales rising 38.7% through 2016.

While Nielsen’s sheer size gives it the measurement upper hand at the moment, Eagan wrote that Rentrak and comScore could partially mitigate that advantage through a cross-platform joint venture.

Rentrak has made a few key changes regarding the terms it uses to describe its TV measurement services. 

Rather than continuing to refer to its national and local television measurement services as “census-like,” Rentrak will now describe those services as “Rentrak Local” and “Rentrak National,” respectively. The terminology change comes after consultation with the Media Rating Council, which claimed that "census like" was not an accurate way to describe the TV measurement services Rentrak offers.

The company's theatrical film and VOD measurement services will remain defined as "census-like." 

Rentrak currently measures approximately 13 million return-path TV homes and 29 million TVs across several MVPDs and regions. According to a statement from Rentrak, the company estimates it will provide measurement services for approximately 26 million TV homes and 60 million TVs by 2015.