Table 2 also shows a within-format learning effect. On the KNS within-subjects experimental design, people were first asked seventeen standard questions, followed by the first eleven of these with variant wordings. For the GSS and KNS-A, B, and D, Don’t Knows were lowest for the first eleven standard items, a bit higher for the six standard items that did not have variant versions, and intermediate for the eleven variant items. But on KNS-C, with its follow-up prompt to those initially saying Don't Know, the DK level before the probe fell by over 50 percent from the initial eleven items to the six other standard items (from 10.6 percent to 5.0 percent), and was about the same for the eleven variant items (4.8 percent). This strongly suggests a learning effect, in which people reduced mentions of DKs to avoid the follow-up prompts once they realized that their initial DK responses would be probed.
This finding was further supported by looking at the DK levels on KNS-D and KNS-B. For the no-probe version (KNS-D), DKs for the first six standard items averaged 10.9 percent and for the other five 11.1 percent. For KNS-C, with probes, the initial (pre-probe) DK levels averaged 13.2 percent for the first six items and 7.5 percent for the next five. Thus, in the no-probe condition, DKs rose by a trivial 0.2 percentage points, but under the probe treatment, initial DKs dropped by 5.7 percentage points.
We next sought to find out how results to the surveys would differ with Don’t Know responses excluded. We discovered that with Don't Knows removed, most differences between support for spending levels were small and usually not statistically significant. The mean absolute differences in “too little” on the GSS and KNS were 5.2 percentage points for KNS-A, 5.0 for KNS-B, and 6.5 points for KNS-C.
However, there was a systematic direction to most differences. Figure 1 plots the similarities as well as the differences in the “too little” rates from the GSS and KNS-A. The x-axis shows the percentages of respondents who selected “too little” in the GSS, and the y-axis shows the percentages who selected “too little” in KNS-A. If an item had equal percentages for “too little” in the GSS and KNS-A (the KNS treatment closest to the GSS in Don’t Know levels and with the largest sample size), then the plotted symbol would fall on the diagonal.

The figure shows that overall, the differences are fairly small, but the GSS tends to have somewhat lower percentages for “too little.” These results are consistent with experiments by Jon A. Krosnick and colleagues in 2002, which found that many respondents initially reporting no opinion were capable of providing substantive answers with the same reliability and validity as those who initially expressed opinions when a no-opinion option was provided.
|