“Don’t know” responses are most common among respondents with low knowledge of or interest in a survey topic. In many instances, “don’t know” is a valid and accurate response, because these respondents genuinely lack an opinion on the issue at hand. Other respondents, however, have a valid opinion but withhold it for a variety of reasons:
In short, some respondents say “don’t know” because they genuinely have no opinion, or they are reticent about sharing it; others are obstructed from stating their opinions by flaws in question design or interviewer training.
The main objective of using follow-up probes to “don’t know” responses is to elicit valid answers that would otherwise be lost. A problem with probes, however, is that they may induce respondents to select a valid response when they actually have no opinion. Past research has demonstrated that when a survey question is phrased in a way that suggests respondents ought to have an opinion on the issue, some will actually make up an arbitrary answer because they do not wish to appear ignorant. When a probe was used in such studies, the effect was magnified. For example, before any probe was administered, nearly half a national sample of adults said they either agreed or disagreed with repealing the “1975 Public Affairs Act,” a bogus issue. When respondents who initially gave no opinion were subjected to a follow-up probe, an additional 10 percent provided a valid response.
But researchers rarely design surveys around bogus issues to trick their respondents; on the contrary, they strive to obtain the most accurate gauges possible of legitimate variables of interest. In the abovementioned experiments, survey items were intentionally phrased to suggest that respondents should know of those bogus issues and should have opinions on them, thus increasing pressure to provide answers when they had none.
Nevertheless, if probes can elicit opinions on bogus issues, they may also elicit bogus opinions on legitimate issues. It follows that including responses elicited by probes can undermine data quality.
To date, however, research comparing data with and without probes has not supported this reasoning. Instead, evidence has consistently shown that including responses elicited by probes produces data of higher validity than simply accepting “don’t know” responses in the first place. For example, analyses predicting final vote choice on electoral candidate races and issue referenda that included responses elicited by probes consistently yielded higher predictive validity than those depending only on responses offered without probes.
To investigate the impact of probing and the effects associated with specific probe wording, Opinion Research USA commissioned a field experiment in which the same nationwide political poll (based on standard CNN poll items) was conducted in three different versions: a control condition with no probe and two conditions with different probe wording. To explore the impact of hard versus soft wording, one probe was worded strongly (“If you had to choose…”) while the other was worded as a mild request (“We are interested in your general inclination…”).
Items included standard polling questions on whether the respondent approved or disapproved of the way George W. Bush was handling his job as president, terrorism, the situation in Iraq, the economy, and immigration. To check for possible approval biases in response, we also added an item on whether the respondent approved or disapproved of the way Bill Clinton handled his job as president when he was in office. A congressional vote choice item asked, “If the elections for Congress were being held today, which party's candidate would you vote for in your congressional district?”; a second item related to the congressional elections was, “If George W. Bush supported a candidate for political office in your area, would you be more likely or less likely to vote for that candidate?”
All three polls were conducted on the same days in June 2006, by the Opinion Research USA call center located in Tucson, Arizona, based on RDD samples of American adults ages 18 or older. We attained a completed sample of 424 respondents in the control condition, 434 respondents in the hard-probe condition, and 428 respondents in the soft-probe condition.
To ensure equivalence, the three completed samples were compared in terms of demographic and operation variables. All three showed highly similar distributions on age, gender, race and ethnicity, highest education attained, annual household income, geographical region, voter registration status, and political ideology (liberal-conservative continuum). The only exception was that there were slightly more Democrats in the hard-probe condition, as compared to the other two conditions.
Table 1 displays the key operation variables. As shown, the three polls had comparable response and cooperation rates, as well as interviewer quality scores. The average length of survey was under five minutes in all three conditions. In sum, the three conditions were roughly equivalent.
Table 2 shows the proportion of “don’t know” responses across the three conditions. As shown in the bottom row in Table 1, the average proportion of “don’t know” responses was 8 percent in the “no-probe” condition, 4 percent in the hard-probe condition, and 3 percent in the soft-probe condition. Hence, both probes were effective in reducing item nonresponse.
We ran various statistical tests to assess the relative effectiveness of the hard versus soft probes, and found no statistically significant difference between the two probes in terms of reducing item nonresponse. In other words, both were equally effective in eliciting valid responses after respondents said “don’t know.”
Ultimately, the objective of a nationwide political poll is to provide survey estimates that gauge the current sentiments of the nation. Hence, the next step in the experiment was to assess whether the overall weighted poll estimates changed significantly if we excluded rather than included responses that were elicited after probing. Each of the three samples was weighted to match census parameters for age, gender, race, education, and geographical region. As shown in Table 3, follow-up probes had minimal impact on substantive poll estimates; none of the shifts before as compared to after probing reached statistical significance.
Table 3 also reveals a marginal but consistent trend on the approval ratings such that slight changes, if present, were always in increments in approval. This trend, which was present for both Bush and Clinton ratings, suggests that respondents who converted from a “don’t know” to a valid answer upon probing (DK converts) were more likely to say “approve” than “disapprove.” Further, there was a marginal but consistent pro-Republican bias among DK converts in the hard-probe condition. We monitored these biases in our subsequent poll and replicated the approval bias but not the pro-Republican bias. In short, DK converts were occasionally more likely to answer “approve” rather than “disapprove.”
The two probes did not work equally well across different demographic groups. Further statistical analysis showed that both probes were more effective with women than men, and the hard probe was more effective in the Northeast and West, while the soft probe was less effective among forty-five- to sixty-four-year age groups and respondents with four-year college degrees, but more effective among respondents with household incomes between $50,000 and $75,000. All these differences were significant when controlling for the impact of all available demographic and political variables.
In conclusion, our field experiment with hard and soft probes showed that, regardless of wording, both were effective in eliciting valid answers from respondents who initially said “don’t know.” Although inclusion of these probe-elicited responses did not have a significant impact on any estimate in this particular poll, the probes did elicit valid responses that would otherwise have been lost. Given the findings from this research, we have implemented the hard probe in all CNN Opinion Research Polls to date. Although both probes were equally effective, the hard probe was selected over the soft probe because the hard probe exhibited a somewhat more consistent impact across different demographics.
LinChiat Chang is a survey methodologist for Opinion Research USA, and Keating Holland is director of the CNN Poll.
The regression models for this analysis were constructed to explore which demographic categories were most or least responsive to the respective probes. Since analyses could not be conducted on each item per se due to the small number of DK converts, all DK conversions across all items were combined for analyses. Logistic binary regressions were run such that the dependent variable was coded 1 for successful conversion by probe and 0 for unsuccessful conversion (that is, remained a DK response); and the independent variables were a series of dummy variables corresponding to demographic groups.
Berinsky, A. J. 1999. The two faces of public opinion. American Journal of Political Science 43:1209-30.
———. 2002. Silent voices: Social welfare policy opinions and political equality in America. American Journal of Political Science 46:276-87.
Bishop, G. F., R. W. Oldendick, A. J. Tuchfarber, and S. E. Bennett. 1980. Pseudo-opinions on public affairs. Public Opinion Quarterly 44:198-209.
Bishop, G. F., A. J. Tuchfarber, and R. W. Oldendick. 1986. Opinions on fictitious issues: The pressure to answer survey questions. Public Opinion Quarterly 50:240-50.
Borgers, N., and J. Hox. 2001. Item non-response in questionnaire research with children. Journal of Official Statistics 17:321-35.
Bradburn, N. M., S. Sudman, and associates. 1981. Improving interview method and questionnaire design. San Francisco: Jossey-Bass.
Converse, J. M. 1976. Predicting no opinion in the polls. Public Opinion Quarterly 40:515-30.
Coombs, C. H., and L. C. Coombs. 1975. “Don’t Know”: Item ambiguity or respondent uncertainty? Public Opinion Quarterly 40:497-514.
Das, A., and J. Deshmukh. 1997. What we know about the “Don’t Knows” (and what we know about what to do with them). WAPOR Newsletter, 12-13.
Ehrlich, H. J. 1964. Instrument error and the study of prejudice. Social Forces 43:197-206.
Faulkenberry, G. D., and R. Mason. 1978. Characteristics of nonopinion and no opinion response groups. Public Opinion Quarterly 42:533-43.
Feick, L. F. 1989. Latent class analysis of survey questions that include don’t know responses. Public Opinion Quarterly 53:525-47.
Gilljam, M., and D. Granberg. 1993. Should we take “Don’t Know” for an answer? Public Opinion Quarterly 57:348-57.
Glynn, C. J., and E. Park. 1997. Reference groups, opinion intensity, and public opinion expression. International Journal of Public opinion Research 9:213-32.
Hawkins, D., and K. Coney. 1981. Uninformed response error in survey research. Journal of Marketing Research 18 (3): 370-74.
Hippler, H., and N. Schwarz. 1986. Not forbidding isn’t allowing: The cognitive basis of the forbid-allow asymmetry. Public Opinion Quarterly 50:87-96.
Klopfer, F. J., and T. M. Madden. 1980. The middlemost choice on attitude items: Ambivalence, neutrality or uncertainty? Personality and Social Psychology Bulletin 6:91–101.
Krosnick, J. A., and L. R. Fabrigar. Forthcoming. Questionnaire design for attitude measurement in social and psychological research. New York, NY: Oxford University Press.
McClendon, M. J. 1986. Unanticipated effects of no opinion filters on attitudes and attitude strength. Sociological Perspectives 29:379-95.
McClendon, M. J., and D. F. Alwin. 1993. No-opinion filters and attitude measurement reliability. Sociological Methods and Research 21:438-64.
Morin, R. 1995. What informed public opinion? A survey trick points out the hazards facing those who take the nation’s pulse. Washington Post National Weekly Edition, April 10-16, e36.
Pickery, J., and G. Loosveldt. 1998. The impact of respondent and interviewer characteristics on the number of “no opinion” answers. Quality and Quantity 32:31-45.
Ready, R. C., Whitehead, J. C., and G. C. Blomquist. 1995. Contingent valuation when respondents are ambivalent. Journal of Environmental Economics and Management 21:181-96.
Schuman, H., and S. Presser. 1981. Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic Press.
Singer E., M. R. Frankel, and M. B. Glassman. 1983. The effect of interviewer characteristics and expectations on response. Public Opinion Quarterly 47:84-95.
Visser, P. S., J. A., Krosnick, J. Marquette, and M. Curtin. 2000. Improving election forecasting: Allocation of undecided respondents, identification of likely voters, and response order effects. In Election Polls, the News Media, and Democracy, ed. P. Lavrakas and M. Traugott. New York, NY: Chatham House.