In an era in which "Nielsen Overnights" have faded away amid the declining linear TV business, and this present Top Gun-dominated holiday weekend aside, the power of the weekend box office report is also dwindling, the weekly streaming horse race has quickly emerged as a go-to destination to satisfy the competitive bloodlust of entertainment consumers.
But the systems in place to determine weekly winners don't appear ready for prime time.
After months of being confounded by different conclusions offered by Nielsen, Netflix and other companies that publicly report audience tallies for streaming, Next TV attempted to take a deep dive into just how this sausage is really made.
Also read: Nielsen and Netflix Bump Heads Again on Programming Metrics
We weren't allowed into the factories -- neither Nielsen or Netflix would speak on the record to us as to how their methodologies work. But we did find out enough to report that conclusive "winners" are hard to determine -- it often depends on which methodology is being used. And a true competitive analysis of the audience can be hard to come by, as well.
Same As It Ever Was?
In terms of measuring audience for streaming, the most prominent voice right now is Nielsen's "Weekly Streaming Top 10" ranker, which started counting viewers for the largest U.S. subscription streaming services in 2020.
Nielsen, an embattled research company just acquired by private equity in March for $16 billion, is looking to sustain hegemony over a TV industry it has ruled for decades, but it now in the streaming era facing a range of challengers.
Each week, Nielsen ranks the leading 10 shows for Netflix, Amazon Prime Video, Hulu, Disney Plus and Apple TV Plus based on total viewing minutes. Nielsen publishes four separate charts: a ranking of overall top performers, along with sub-rankings for original series, off-net streaming TV shows and movies.
Nielsen pulls this data from its SVOD Content Ratings panel of 41,000 homes, and it accounts for multiple users in the same domicile watching the same show together.
But there are key limitations. For one, the finite sampling is "glass-only," meaning viewership on phones, tablets and computers isn't being measured. Netflix, far and away the biggest SVOD presence in the U.S. with around 66 million of its 220 million users worldwide situated here, won't say what percentage of its audience watches shows on mobile devices.
However, a survey conducted by Leichtman Research Group in 2019 found that around 85% of the audience watches Netflix shows on TVs. But again, the conclusions vary -- comScore, for example, determined back in 2014 that 49% of younger 18-34-year-old viewers watch via IP device of some kind.
Nielsen's competitive analysis is also skewed by Netflix's dominance. Unlike broadcast TV, where it can be assumed that all viewers in a given market have the same over-the-air access to all local stations, Netflix still has a lot more viewers than any of the other SVOD services measured in Nielsen's Weekly Streaming Top 10. Thus, most of the shows appearing in Nielsen's rankings belong to Netflix.
Meanwhile, its SVOD rankings are missing key constituents, including HBO Max and Paramount. These services haven't forged agreements with Nielsen to be included in the rankings.
Back in early 2021, this became an issue for the erstwhile WarnerMedia, which asked for a special Nielsen Weekly Streaming Top 10 ranking be made featuring HBO Max and day-and-date streaming release Wonder Woman 1984, which topped Nielsen's charts for Christmas Week 2020. HBO Max hasn't been back since.
To measure audience usage in its content ratings panel, Nielsen uses an audio-signature technology that takes time to tally. This is the main reason why the research company takes around 25 days before it reports performance data on any given week.
There are numerous research companies right now competing to deliver audience data across the broader range of SVOD, AVOD and FAST services. Media outlets like Next TV only receive a small portion of the iceberg that's poking out of the water.
But even in that niche vanity market, the race is competitive. Samba TV, which measures usage on smart TVs, regularly releases household performance data on shows to the press. Reelgood, meanwhile, makes a mobile app that lets users punch in a show or movie to find out all the different platforms for which it is available. And Reelgood regularly publishes non-contextualized rankers of shows based on its customer referrals.
Then there's Demand Media, which has gained traction via a proprietary formula that ranks programs based on, well, their "demand."
But no other third-party measurement scheme has delivered as clean, and as competitively contextualized a ranking as Nielsen's Weekly Streaming Top 10, which Next TV reported on weekly up until December of last year.
Uh, Houston?
In September, Netflix made the splashy introduction of its "Global Top 10" ranker, which expanded on its publicly effacing "Top 10 in the U.S." rankings widget with four detailed weekly charts ranking its top series and movies, both sliced by English and non-English language formats, by hours viewed on a global basis.
Netflix's four weekly ratings charts offer a clean assessment of hours streamed on Netflix's global platform of 220 million-plus user IDs, complete with indicators showing countries in which the programs performed best.
Netflix's Global Top 10 satisfies the thirst for immediacy in a way Nielsen doesn't -- the Global Top 10 comes out on Tuesday, 48 hours after the culmination of the measurement week on the previous Sunday.
There are drawbacks, starting with the fact that a FAANG company is basically reporting on itself, and its rankings provide absolutely no competitive context.
Also, the Global Top 10 counts how many hours of a show were streamed to user IDs -- there's no accounting for co-viewership.
Regardless, after years of being accused of lacking accountability and transparency, Netflix finally developed a hedge.
And despite the limitations of the Global Top 10, it has called into question the accountability and transparency of Nielsen, which had been steadily gaining traction with its SVOD ranker.
In January, for example, Nielsen declared Disney Plus day-and-date animated film Encanto the winner of the Christmas Week streaming race, beating out Netflix's Adam McKay-directed Oscar-nominated satire Don't Look Up by a sizable margin.
Showbiz trades and enthusiast pubs rushed to declare Encanto the winner over Don't Look Up. But it was clear from comparing the Nielsen and Netflix rankers that something was amiss.
Nielsen credited Don't Look Up with only 521 million U.S. viewing minutes for the week of December 20-26, or around 8.7 million hours. Netflix clocked its show at 111 million hours on a global basis. So, in order for both Nielsen and Netflix to both be right, only 8% of the audience for an English-language movie, set in America, with American talent, came from the U.S., a market that controls 30% of Netflix's total audience.
Nielsen didn't respond to Next TV's request for comment. But in a footnote in its ratings report the following week, the research company said it was revising the Don't Look Up audience by a factor of 3X, which would have rendered the movie the Christmas Week winner with a bullet.
By this time, however, Encanto had been declared the big winner of a key holiday frame, the importance of the streaming window greatly enhanced by Omicron-related theater shutdowns, at least in the eyes of the public.
The handling of the Don't Look Up matter calls into question not just Nielsen's transparency in the wake of a counting error, but also the efficacy of its audio-signature tech.
In April, Nielsen declared Disney-Pixar film Turning Red the winner over Netflix's Ryan Reynolds comedy-action film The Adam Project, even though the comparative math conducted by Next TV clearly favored Netflix.
And just last week, Nielsen declared Netflix's Better Call Saul the winner of the April 18-24 subscription streaming race, even though the AMC off-net series didn't even register on Netflix's Global Top 10.
Again, Nielsen executives declined to speak to Next TV on the record. However, individuals familiar with the company's measurement systems told us that discrepancies with Better Call Saul might stem from the fact that Nielsen ranks shows by their full episode count, while Netflix divides them by season.
However, using Netflix's data, even if we cumed all 50 repeat episodes of Better Call Saul available on the streaming platform, we wouldn't come close to the April 18-24 audience performance registered by Netflix original series Anatomy of a Scandal, ranked No. 1 in Netflix's Global Top 10 but only No. 4 by Nielsen.
Sources close to Nielsen add that while numbers can vary based on methodology, it's the competitive context offered by the Weekly Streaming Top 10 that matters.
But again, with limited participation, transparency and accountability, that claim is also suspect.
So, for now, Next TV will continue to report Netflix's Global Top 10 and weigh in on other metrics on an ad hoc basis. The public wants -- the video business needs -- an effective third-party measurement system.
We'll keep observing this area until one evolves.
