August has opened some eyes with respect to survey research. First came the Michigan Democratic primary for a vacant U.S. Senate seat on August 3. The polls showed that Dr. Abdul El-Sayed, a left-wing progressive, was a shoo-in for the nomination, expected to win by double digits. He won the nomination over Representative Haley Stevens, but by only a single percentage point.
Last Tuesday, Wisconsin held its primary. The polls showed another left-wing progressive, this time a card-carrying member of the Democratic Socialists of America, State Representative Francesca Hong, would cruise to a win to be the Democratic candidate for governor of Wisconsin, likely by double digits, just as was expected for El-Sayed. When the votes were counted, she lost by a whisker to Terry Crowley, the county executive of Milwaukee County.
When the Polls Miss the Mark
The point here is not to opine about whether progressives or socialists are suddenly in or out of vogue in the Midwest, but rather the question of how the polls could be so far off from election results. This applies to our world, because ratings are dependent on survey research.
Two articles published after the primaries raised some red flags. The Financial Times ran a piece on Wednesday, “Shock Wisconsin Vote Triggers Doubt Over US Election Polling,” and a Nate Cohn column on Friday in the New York Times asked, “Can Polls Measure the True Strength of the Left?”
The articles pointed to multiple issues in political polling, so let’s define a couple of terms used in survey research — not just for political polls, but for ratings as well.
Bias vs. Error: Getting the Terms Right
I’ve always felt that survey research has a terminology issue, using words like “bias” and “error.” To the layperson, “bias” refers to a “lean” in one direction or another — for example, Fox News and Newsmax lean right or conservative, while MS NOW and The New York Times can be said to lean left or liberal. These are political biases and are more deliberate. In survey research, the word “bias” means the results went a particular way that may be misleading, but not through any deliberate attempt by the survey outfit to create a predetermined set of results.
One element is “non-response bias,” which is whether those who responded to a survey held similar views to those who didn’t respond. If the respondents who answered were very different from those who didn’t, the results can be way off, as in Michigan and Wisconsin. The FT article also noted that according to the polls, Andrew Cuomo should be living in Gracie Mansion in New York City today, not Zohran Mamdani. Cuomo was beaten solidly by Mamdani in the NYC Democratic primary.
Then there is “error.” If you think back to the famous play in Game 6 of the 1986 World Series between the Red Sox and the Mets, a ball went between Bill Buckner’s legs as he was trying to field it at first base. That was an “error,” which in baseball means a mistake.
In survey research, “error” refers to a range around the estimates in the survey that would likely include the truth, assuming we could survey everyone in the population we’re trying to measure. It’s not a “mistake” by the people conducting the survey.
The Shrinking Response Rate Problem
For all surveys, it’s harder than ever to reach people. When I started, the gold standard was an in-person interview, which was rarely done because of the expense. Phone surveys worked because most everyone had a landline phone and was likely to respond when randomly chosen.
Then multi-phone households started to appear, which added a “bias” — in this case, if you had two phone numbers, your home was more likely to be called than a home with one number. Next came cell phones and a shift toward individuals having phone numbers rather than households. More bias.
Meanwhile, the Telephone Consumer Protection Act of 1991, which prohibited the use of automatic dialing equipment to call known cell phone numbers, hampered phone surveys. Political calls were exempted, but not legitimate survey research. Arbitron was one of the first companies to start calling known cell phone numbers in 2009, through an expensive two-step process to comply with the law.
Costs became so steep that a lot of survey research went online, with all the problems and pitfalls that entails. Meanwhile, response rates continued to decline, so survey companies were left with a demand for data while having a huge problem collecting it.
Opt-In Surveys and the Rise of Bots
In some cases, the answer has been opt-in online surveys, which carry all kinds of risks.
The FT article suggested that those who respond to these surveys are “more politically engaged than the average citizen,” and this method also raises the issue of bots.
The FT article carried a quote from a polling expert at Georgetown: “They’re a wacky group of people, if they’re people at all.”
Weighting as the Imperfect Fix
The answer, if there is one? More weighting than ever. Arbitron and now Nielsen radio data are weighted for age/sex, race/ethnicity, geography, and language among Hispanics, depending on the market. PPM added two more weighting variables: presence of children and employment status. That’s a lot of weighting, but it may be necessary in most cases.
Political surveys often add other weighting variables: Will the voters who say they support a candidate show up and vote? What is the strength of their support? What is their party affiliation, if any? Is one group of voters more energized before the election and more likely to respond to a political survey, creating a bias (there’s that word again)? Did the respondent vote in previous elections? How about the undecideds? Will they vote, will they stay home, do they lean toward a certain candidate?
I was on a Zoom call last week with one of my former bosses, and we agreed that we’re glad we’re not doing surveys for a living anymore. It’s a very tough business, and all these factors can cause some strange results. Whether it’s a socialist leading by ten points and then losing a primary, or a country station showing up as number one among 25–54 African Americans, this happens. Even though political polls measure intentions and ratings measure behavior, the problems are the same. Caveat emptor.
Barrett Media produces daily content on the music, news, and sports media industries. Sign up for our newsletters to stay updated and get the latest information right in your inbox.
