Eliciting Valid Responses: Follow-Up Probes to “Don’t Knows” in Political Polls

By LinChiat Chang and Keating Holland

“Don’t know” responses are most common among respondents with low knowledge of or interest in a survey topic. In many instances, “don’t know” is a valid and accurate response, because these respondents genuinely lack an opinion on the issue at hand. Other respondents, however, have a valid opinion but withhold it for a variety of reasons:

• They may not feel completely sure of their response, and so are not comfortable or confident enough to pick a valid answer.

• Their beliefs, attitudes, and feelings on the issue may be contradictory—a state of ambivalence that can result in a “don’t know” response because the conflicting attitudes and feelings preclude a clear stand.

• They may be uncomfortable giving out answers they would prefer to keep private. For example, given the apparent decline in presidential approval ratings for George W. Bush, pro-Bush respondents may be more reluctant to state their opinions, but some will do so upon probing. (Indeed, data from the field experiment to be discussed here showed more pro-Bush and pro-Republican responses among responses elicited by probes, implying at least some support for this reasoning.)

• The survey question is unclear or difficult to understand.

• The response options provided are a poor match to respondents' opinions, making it frustrating and difficult to pick an appropriate answer.

• Interviewers who expect to get valid answers are more likely to do so than those who expect respondents to have no opinion on the subject. Many interviewers and field supervisors also attest to a phenomenon where respondents say “don’t know” when they are merely stalling while they think. These respondents will often provide a valid answer if the interviewer waits or probes.

In short, some respondents say “don’t know” because they genuinely have no opinion, or they are reticent about sharing it; others are obstructed from stating their opinions by flaws in question design or interviewer training.

The main objective of using follow-up probes to “don’t know” responses is to elicit valid answers that would otherwise be lost. A problem with probes, however, is that they may induce respondents to select a valid response when they actually have no opinion. Past research has demonstrated that when a survey question is phrased in a way that suggests respondents ought to have an opinion on the issue, some will actually make up an arbitrary answer because they do not wish to appear ignorant. When a probe was used in such studies, the effect was magnified. For example, before any probe was administered, nearly half a national sample of adults said they either agreed or disagreed with repealing the “1975 Public Affairs Act,” a bogus issue. When respondents who initially gave no opinion were subjected to a follow-up probe, an additional 10 percent provided a valid response.

But researchers rarely design surveys around bogus issues to trick their respondents; on the contrary, they strive to obtain the most accurate gauges possible of legitimate variables of interest. In the abovementioned experiments, survey items were intentionally phrased to suggest that respondents should know of those bogus issues and should have opinions on them, thus increasing pressure to provide answers when they had none.

Nevertheless, if probes can elicit opinions on bogus issues, they may also elicit bogus opinions on legitimate issues. It follows that including responses elicited by probes can undermine data quality.

To date, however, research comparing data with and without probes has not supported this reasoning. Instead, evidence has consistently shown that including responses elicited by probes produces data of higher validity than simply accepting “don’t know” responses in the first place. For example, analyses predicting final vote choice on electoral candidate races and issue referenda that included responses elicited by probes consistently yielded higher predictive validity than those depending only on responses offered without probes.