To investigate the impact of probing and the effects associated with specific probe wording, Opinion Research USA commissioned a field experiment in which the same nationwide political poll (based on standard CNN poll items) was conducted in three different versions: a control condition with no probe and two conditions with different probe wording. To explore the impact of hard versus soft wording, one probe was worded strongly (“If you had to choose…”) while the other was worded as a mild request (“We are interested in your general inclination…”).

Items included standard polling questions on whether the respondent approved or disapproved of the way George W. Bush was handling his job as president, terrorism, the situation in Iraq, the economy, and immigration. To check for possible approval biases in response, we also added an item on whether the respondent approved or disapproved of the way Bill Clinton handled his job as president when he was in office. A congressional vote choice item asked, “If the elections for Congress were being held today, which party's candidate would you vote for in your congressional district?”; a second item related to the congressional elections was, “If George W. Bush supported a candidate for political office in your area, would you be more likely or less likely to vote for that candidate?”

All three polls were conducted on the same days in June 2006, by the Opinion Research USA call center located in Tucson, Arizona, based on RDD samples of American adults ages 18 or older. We attained a completed sample of 424 respondents in the control condition, 434 respondents in the hard-probe condition, and 428 respondents in the soft-probe condition.

To ensure equivalence, the three completed samples were compared in terms of demographic and operation variables. All three showed highly similar distributions on age, gender, race and ethnicity, highest education attained, annual household income, geographical region, voter registration status, and political ideology (liberal-conservative continuum). The only exception was that there were slightly more Democrats in the hard-probe condition, as compared to the other two conditions.

Table 1 displays the key operation variables. As shown, the three polls had comparable response and cooperation rates, as well as interviewer quality scores. The average length of survey was under five minutes in all three conditions. In sum, the three conditions were roughly equivalent.

Table 2 shows the proportion of “don’t know” responses across the three conditions. As shown in the bottom row in Table 1, the average proportion of “don’t know” responses was 8 percent in the “no-probe” condition, 4 percent in the hard-probe condition, and 3 percent in the soft-probe condition. Hence, both probes were effective in reducing item nonresponse.

We ran various statistical tests to assess the relative effectiveness of the hard versus soft probes, and found no statistically significant difference between the two probes in terms of reducing item nonresponse. In other words, both were equally effective in eliciting valid responses after respondents said “don’t know.”