The upcoming midterms are generating a great deal of discussion, especially on news/talk stations. If you’re watching the polls, there’s an excellent collection of guidelines for opinion polls and surveys that you should check out. It’s published by ESOMAR and WAPOR, and you can find it here. The document is a bit over 20 pages long, and while it’s not the most exciting read, it’s important as we approach November 3.
ESOMAR is a global organization for research, insights, and analytics with members and companies in over 130 countries. WAPOR is the World Association for Public Opinion Research, and it declares itself “the world’s leading professional association advocating for the integrity of public opinion research worldwide.” I’m a former ESOMAR member and have attended their conferences over the years. These are not lightweight organizations.
Both parties and many voters look at publicly released polls to see if their favored candidates are leading or trailing in their states or districts. It’s important that we separate the wheat from the chaff, or in this case, separate legitimate, properly conducted surveys from crap work.
Why Polls Face Scrutiny
Much has been written (including here) about how polls have “missed” badly in the last few elections, especially at the presidential level.
Legitimate pollsters have implemented new procedures to adjust for non-response and other issues.
Further, the candidates and the parties have access to proprietary polling data and qualitative data (focus groups, etc.) that are not available to the public or press.
What the Guidelines Say
Let’s consider some of the guidelines. Under section 3.2, Honesty, a key point is to not misrepresent activities like political telemarketing or digital marketing as opinion research. This includes “push polling,” where negative campaigning is disguised as a political poll.
Part 6 of the guidelines covers “Methodological quality and transparency,” which can be summarized as “disclosure.” Among other things, here’s what ESOMAR and WAPOR say you should expect from pollsters:
- Provide the technical details of the project. Results should be documented accurately, transparently, and objectively.
- Disclose the timing of the fieldwork, the sample size, interviewer training, and the sampling method.
- Take all necessary precautions to identify or remove fabricated or falsified data.
- Note the rules used for data cleaning.
How Polling Is Conducted Today
Most polling in the US is done online, although some outfits may still use “mixed methodologies” (phone and online) to ensure better coverage of the population being measured. There are some probability-based online samples where participants are initially recruited to be in a panel so they can be surveyed later.
Some time ago, I wrote about a panel developed by the Pew Research Center.
It’s safe to say that most online political polling is not probability-based. These are convenience samples that have issues when projected to a population, for example, voters in a statewide election for US Senate.
What News/Talk Hosts Should Check
If you’re a news/talk host, producer, or otherwise involved and you’re planning to talk about polling data, the guidelines suggest that you check the research company’s website for the following:
- Where did the company get the respondents? From a panel or multiple panels? Probability or non-probability based?
- How were the respondents chosen? Quotas? Propensity? Another method?
- What are the response or participation rates (the former for probability samples and the latter for panels)?
- How were the respondents recruited?
- What was the weighting process?
You don’t need to be a statistician or have one on call (and no, I’m also not a statistician). What you should look for are the potential red flags. First, can you find out how the poll was conducted? If not, either contact the research company or ignore the poll. If you can get the information suggested above, consider “face validity.”
For non-researchers, that’s known as the “smell test” (use “face validity” if you want to sound smart). Does the poll look like it was done with some level of standards, or were corners cut?
For example, was the sample screened for registered voters and intention to vote?
Bringing Polls to the Air
This shouldn’t take much time and, in fact, might add to the on-air conversation. The host can give real reasons why some of the polls about your market’s midterms are good or bogus, beyond “How can Candidate X be ahead?” or “You can’t believe any of the polls.” Discussing the background of polling on-air is probably a great way to destroy a quarter hour.
However, being able to say “this poll was well done, right or wrong” or “I don’t trust this poll” adds credibility. And remember, a poll is “a snapshot in time.” Some registered voters don’t end up voting. Others change their minds, even at the last second.
The last decade has been a tough one for political polls. The good companies have done their best yet have missed by a wide margin in some cases. Understanding how to evaluate polls and explain them quickly and simply to listeners will help.
Let’s meet again next week.
Barrett Media produces daily content on the music, news, and sports media industries. Sign up for our newsletters to stay updated and get the latest information right in your inbox.
