Why should Nielsen survey response rates matter? The question is really this: for whatever behavior or attitude you want to measure (radio listening or TV viewing in our case), were the people who responded similar to those who didn’t wish to participate? If the response rate is high, that’s usually not an issue. If it’s very low, it probably matters.
Long ago and far away, Americans were actually impressed that someone wanted to know their opinions and behaviors. In other words, they were asked to respond to a survey. Sometimes the sample frames were less than ideal. For example, the infamous Literary Digest poll that said Alf Landon would defeat Franklin Roosevelt in 1936 (Landon secured a massive eight electoral votes). And there is the famous picture of Harry S Truman holding a copy of the Chicago Tribune with a banner headline proclaiming, “Dewey Defeats Truman”. Those results aside, many Americans used to enjoy doing surveys.
Response rate has always been a key indicator of survey quality. Assuming the sample frame (the list of possible people or households that you might reach) is representative of the population you want to survey, the higher the response rate, the better the quality. Today, response rates are at record lows. Very few surveys use random sampling at this point although Nielsen still does.
Like every other business, the survey business has changed. Gone are the days when landline penetration in the United States was near universal and randomly calling landline phones was the typical way to survey Americans. If you’re interested, the best track of phone status has been the National Health Interview Survey conducted by the National Center for Health Statistics. As of the second half of 2022, the estimate was that 72.6% of all US households were cell phone only. Only 1.3% were landline only! Less than 1% had no phone service at all and the rest had both cell and landline service in their homes. It’s not surprising that barely one out of four US households still has a landline.
Why not just call cell phones? There is a small issue known as the Telephone Consumer Protection Act of 1991. Did you have a mobile phone in 1991? Perhaps a Motorola brick analog phone? What did roaming cost? It was probably around one dollar a minute (in 1991 dollars). Telemarketers started calling cell phones, whether by design or accident. Those charges added up which led to the cry of “There ought to be a law against this!”.
At that time, Congress was better at getting things done, for better or worse, and they passed the TCPA. It forbids companies from using “automatic dialing equipment” to call cell phone numbers. The goal was to stop the telemarketers, but in doing so, legitimate survey research was closed out as well. The law required some form of agreement from the called party to accept the calls.
Nielsen has adapted as best they can. Arbitron started calling cell phone numbers in 2009 and got around the prohibitions by mailing a survey in advance. Those who responded with household information effectively agreed to accept calls from Arbitron on their cell phone. It was expensive, legal, and necessary as we could see the changes coming. Today, many diary households never speak to anyone at Nielsen…the whole process operates by mail.
As these changes were taking place, the willingness of Americans to participate in surveys was declining. Along with my Arbitron colleagues, we tried to find any way possible to improve response rates without breaking the bank. The Media Rating Council had a laser focus on response rates which further kept us working on that aspect. However, we could improve response rates but would hurt proportionality. For example, we could have improved response by getting far more 65+ persons to respond, but the resulting proportionality would, in technical terms, suck.
When PPM began, Arbitron tried to report an honest response rate. Unfortunately, that number was often a single-digit percentage. The suggestion was to use Nielsen’s panel method, a statistic called the Sample Performance Index (SPI).
SPI implies two samples. One is a set of basic households. You do everything you can to bring in a basic household because the SPI is the percentage of basic households in the sample. If a basic household simply won’t cooperate, you choose an alternate. Here’s the rub: you can go after one or 100 alternates with no effect on the SPI. The real response rates may be absolutely pathetic, but the SPI can look reasonable, and in this case, reasonable might be a 20 or higher.
If you’re wondering about your market’s response rate, it’s in the E-Book. PPM markets have the SPI and diary markets report an adjusted response rate. Take a look and you’ll get an idea of how many households or individuals will simply not be part of your market’s ratings.
Let’s meet again next week.
Excellent insight- Salespeople don’t take ratings lying down. Your clients control the narrative if you don’t know how they are created! Long live Dr.Ed.