Most weeks, this column will write about the Nielsen ratings system and how you can better understand it to make it work to your advantage. This week, I’ll go in a slightly different direction and cover one of my pet peeves regarding surveys or more specifically, survey questions. Many books have been written about questionnaire design and surveys are built in different ways depending on the mode: online, phone (yes, some still exist), mail (that mode still exists as well), or in-person.
Over my career, it’s always been entertaining to watch amateurs write questionnaires, operating under the assumption that anyone can do it. This is a fallacy. Back when I was doing perceptual studies for radio stations, my mantra was that you would be better off with no data rather than bad data. It’s easy to generate bad data, but in the absence of information, the gut of many radio pros is pretty good.
Most of today’s questionnaires are self-administered, in other words, someone is answering without the aid of a knowledgeable person working for the survey operation. Even in the days when much survey research was done by phone, interviewers were often poorly trained and turnover percentages were typically massive.
In other words, you couldn’t be sure that the “professional interviewer” could help the respondent if a question wasn’t clear since tenure was typically less than six months. The exception to this rule is focus groups or one-on-ones where a professional moderator can follow up if a group member or individual doesn’t understand the question.
For example, if you have a Fred Jacobs doing your online focus group, his years of experience will make sure that an off-the-wall answer can be clarified. The only problem is that particular branch of research is considered qualitative and is best followed up with a quantitative study.
There are so many ways to screw up survey questions (and I’ve made my share of mistakes in the past), but let’s start by focusing on one word: “and”. It’s a good starting point because as soon as you add the word “and” to a question, you may be asking two questions.
A very simple example would be “Do you like or dislike President Biden and Vice President Harris?” While it might make sense to ask about the two together as both are Democrats and ran together for their respective offices, a respondent may like one and not the other. How do they answer the question?
In radio research, this comes into play in a couple of ways. The one that I’ve had issues with over the years is music testing. While the Barrett Media websites cover sports and news, many of you have worked in music formats or may be doing double duty today. In music testing, you are typically trying to find out three things about a song:
1. Is the respondent familiar with the song based on the hook?
2. What songs do fans (P1s) of a particular format like best?
3. Are any songs “burnt”?
For me, numbers two and three are two separate questions. When I did music testing back in the days of in-person auditorium tests and scantron sheets, I preferred a 1-7 scale for like/dislike, and a separate place for something to the effect of “Are you tired of hearing this song on your favorite station?” If the respondent wasn’t tired of the song, it was left blank. The scantron forms also allowed for an “I don’t know this song” option. Online testing can use the same version.
The reason for pointing this out is my opposition to the use of variations along the line of “Love it, Like it, Don’t Like, Tired of it, Don’t know it”. If you love the song and are also tired of it, what do you choose? That’s why I’ve always said, “If you want to ask two questions, ask two questions.”
As an aside, the 1-7 scale has two big advantages. First, the fans of a format tend to like that format’s music, so typically, most of the songs score well (the exception was always rock formats). With a 1-7 scale, the choice is essentially “good, better, best” (5, 6, and 7) offering a bit more discrimination.
Next, while we want to know if respondents have any “lean”, they often want to be neutral. Using an even number of choices will lower the scores a bit. With an odd number of options, there is a middle, in this case, using the “4” choice. When doing perceptual studies, I always preferred a seven-point scale. While others like ten-point scales, where 10 is “strongly agree” and 1 is “strongly disagree”, in my view, respondents will see “5” as “neutral” or “not sure” (assuming “not sure” is not offered) and that lowers scores because the true midpoint is 5.5, which isn’t available.
If you work exclusively in spoken word, also consider the polling that you review. Aside from the other survey research issues (sampling, sample size, etc.), always look at the questionnaire. The best polling outfits will typically show you how the questions were worded, but even the best firms and organizations occasionally make mistakes, and that can lead to misleading results.
As someone who has both created and reviewed many questionnaires as well as lots of data in my career, I enjoy looking at how surveys are constructed today. To that end, I’m in a couple of survey panels, in my case, earning airline miles. While I always answer the questions honestly, my interest is in looking at the questions and the questionnaire flow. Can I readily understand what is being asked? Are they asking me to answer something where I have no real knowledge to answer the question? You’d be surprised how often that arises.
Then there is the length of the survey and whether you hit a burnout point. I suggest that you try it as well and you’ll start to get a feel for what will likely work and better yet, what won’t in your research, if you’re fortunate enough to have a budget.
Questionnaire design is both art and science, in many ways, just like media. What works in one situation may not work in another. Do your best to make sure you get the odds on your side.
Let’s meet again next week.
