We’re in the midst of the political season which means a lot of polling. Surveys aren’t perfect and in fact, are getting harder to conduct all the time.
In this column, I want to explain the concept of sampling error with respect to ratings. You’ve probably heard about measures of sampling error in political polls, for example, a poll result might be plus or minus four percentage points. If that poll shows candidate A with 51 percent and candidate B with 49 percent of the potential vote, the odds are very good that candidate A is somewhere between 47 and 55 percent, while candidate B is between 45 and 53 percent. In other words, a statistical tie.
What about ratings? How close to “reality” are Nielsen Audio’s estimates?
What you need to keep in mind is the size of the estimates. When two candidates face off for an elective office, both will get fairly high percentages, even in a landslide situation (a landslide might mean the winner received 60 percent of the vote). For radio, we’re looking at AQH ratings, which are tiny. If one of your stations is pulling a 0.5 rating in a key demo/daypart combination, that’s good, but the sampling error, on a percentage basis, can be huge.
I’m not a statistician (nor have I played one on TV or stayed at a Holiday Inn Express recently for those of you who remember old ad campaigns), but here’s the simple formula:
What does it mean? In this simple version, p is your rating and q is 100 minus your rating (100 meaning everyone or 100 percent of the population being studied), while n is the sample size. Take the square root of that result and multiply it by 1.96 to get a 95 percent confidence interval. This means you are 95 percent certain that reality is within the result of the little equation. If you’re willing to go with a 90 percent confidence level meaning a little less certainty, multiply the result times 1.65. If you don’t want to multiply, the result will be about a 68 percent confidence level. The Nielsen calculation is far more sophisticated than the simple version here and takes other factors into account to reduce the amount of sampling error.
Let’s try an example with the simple equation above. Use a calculator on your phone, your PC, or in my case, my 40-year-old solar-powered KMC-7000 calculator, purchased at the K-Mart of Okemos, Michigan when I started my Ph.D. program at Michigan State in September 1983. It still works perfectly, although K-Mart is just about dead unless you live in the U.S. Virgin Islands.
Suppose your station had a 0.5 rating with P12+ (let’s use a big demo here for more sample) and the sample size was 2,000 (reasonable in a PPM or bigger diary market). The 95 percent confidence interval would be around 0.3 rating points. We would be 95 percent sure that the true estimate for this station would be a rating between 0.2 and 0.8.
If you put that in percentage terms, we’re looking at plus or minus around 60 percent! How many other stations are in that range in your market? While the Nielsen version will cut that down to perhaps 0.2 or so, it’s still big. Then consider the demos that matter to you, perhaps men 25-54 for a classic rock outlet, or persons 18-49 for a CHR. The sample size is smaller, so the error will be larger.
You might take exception to results like these, but the fact is that unlike political polls that typically have just two choices, listeners have lots of choices when it comes to radio stations. Further, listening levels, whether you refer to them as PUMM or PUR, are low. In most PPM markets, the listening levels in key demos at peak times are in single digits, perhaps as low as a 5 rating, while diary market listening levels are typically higher. As I’ve always explained it, if the listening level is a 5, that means in any given quarter hour for that demo/daypart combination, 95 percent of the population is not listening to radio, which is simply 100 minus 95.
Nielsen does offer a Ratings Reliability Estimator for clients which calculates the sampling error at different confidence levels. I doubt anyone uses it, let alone can find it, but it is a requirement for Media Rating Council accreditation.
Here’s another twist in the simple equation. If you play with it a bit, you’ll discover that to cut the sampling error in half, you must quadruple the sample size. Sample size (the n or sample size in the denominator drives sampling error calculation) is what matters. In my example, to cut the sampling error in half from the ±0.3 I showed you, the sample size would have to be increased to 8,000. Perhaps hell will freeze over some day, but don’t count on Nielsen announcing a four-fold sample increase any time soon.
We can blame Nielsen for a lot of things, but not necessarily this one. Despite sky high cume ratings for radio usage and sometimes for individual stations (and you can do this sample calculation with cume ratings or persons), radio sells AQH ratings and persons, which are small numbers. Nielsen’s sample sizes are based on survey statistics as well as what the market will bear. Placing PPMs in a home is more expensive than diaries, even though PPM is now being used for both audio and video spreading the cost across a larger revenue base. If your company is a subscriber, it’s unlikely that the management wants to pay Nielsen more than they do today in return for a massive increase in sample size.
There’s more to sampling error than this short piece can cover, and you’d likely be bored reading it. I tell people to just go with whatever Nielsen reported. Radio is a tough enough business these days and having to consider that the top five or more stations may be statistically tied is one more headache you don’t want to think about.
Let’s meet again next week.
