Ultimately, the objective of a nationwide political poll is to provide survey estimates that gauge the current sentiments of the nation. Hence, the next step in the experiment was to assess whether the overall weighted poll estimates changed significantly if we excluded rather than included responses that were elicited after probing. Each of the three samples was weighted to match census parameters for age, gender, race, education, and geographical region. As shown in Table 3, follow-up probes had minimal impact on substantive poll estimates; none of the shifts before as compared to after probing reached statistical significance.

Table 3 also reveals a marginal but consistent trend on the approval ratings such that slight changes, if present, were always in increments in approval. This trend, which was present for both Bush and Clinton ratings, suggests that respondents who converted from a “don’t know” to a valid answer upon probing (DK converts) were more likely to say “approve” than “disapprove.” Further, there was a marginal but consistent pro-Republican bias among DK converts in the hard-probe condition. We monitored these biases in our subsequent poll and replicated the approval bias but not the pro-Republican bias. In short, DK converts were occasionally more likely to answer “approve” rather than “disapprove.”

The two probes did not work equally well across different demographic groups. Further statistical analysis showed that both probes were more effective with women than men, and the hard probe was more effective in the Northeast and West, while the soft probe was less effective among forty-five- to sixty-four-year age groups and respondents with four-year college degrees, but more effective among respondents with household incomes between $50,000 and $75,000. All these differences were significant when controlling for the impact of all available demographic and political variables.

In conclusion, our field experiment with hard and soft probes showed that, regardless of wording, both were effective in eliciting valid answers from respondents who initially said “don’t know.” Although inclusion of these probe-elicited responses did not have a significant impact on any estimate in this particular poll, the probes did elicit valid responses that would otherwise have been lost. Given the findings from this research, we have implemented the hard probe in all CNN Opinion Research Polls to date. Although both probes were equally effective, the hard probe was selected over the soft probe because the hard probe exhibited a somewhat more consistent impact across different demographics.

LinChiat Chang is a survey methodologist for Opinion Research USA, and Keating Holland is director of the CNN Poll.