Online vs. In-Person By Tom W. Smith and J. Michael Dennis page4

Besides the small but systematic differences among the items as a whole, there were moderate to large differences in one distinctive subset of items. GSS respondents were notably more likely to favor more spending than KNS panelists for big cities (the percentage responding “not too much” in the GSS minus the percentage saying “not too much” in the KNS equaled +6.3 percentage points), drug addiction (+7.0 points), blacks (+13.8 points), foreign affairs (+8.5 points), and welfare (+9.7 points). These results were similar to the differences found in 2000, when the GSS showed notably greater support for spending for the same five areas, four of which had to do with the problems of the urban underclass and the fifth, foreign aid, with another disadvantaged population.

Several theories have been offered to explain this now replicated finding. One suggests that it results from the difference between the KNS's self-administered mode and the GSS’s being administered by interviewers. Proponents of this view characterize the items as being "potentially controversial and sensitive issues" for which the presence of an interviewer may create a social desirability effect, encouraging more sympathetic, pro-spending responses by GSS respondents. Previous research has shown that self-administered formats typically reduce social desirability effects on the reporting of sensitive behaviors.

However, these items are not about behaviors, and it is not clear that being pro-spending is necessarily the socially desirable response. There is a large anti-spending majority for foreign aid; and being in favor of more spending to deal with drug addiction correlates with other items as a punitive rather than a helping and caring response, such as spending more for blacks and big cities.

A second theory is that KNS panelists are more conservative on social welfare issues, perhaps because such people are more likely to join and maintain membership in this incented panel. But in 2003, researchers J. Michael Dennis and colleagues found a slight liberal tendency in the KN panel.

Third, the differences in spending priorities could have to do with differences in survey response rates, conditioning, or other factors.

Finally, we can rule out demographic differences as a likely cause, since the weighted GSS and KNS profiles on background variables are close.

Since 1984, the GSS has included experiments on the wordings of the spending items. They represent the longest and largest-scale series of wording experiments and replications ever conducted. Within the GSS the wording effects are very stable across time, and moderate to large on some items. As in the 2000 experiments, the 2002 GSS and KNS showed wording effects on the same items. But while the direction always agreed, the size of the differences often varied. Spending for assistance to the poor was more strongly supported than welfare spending (GSS +45.9 percentage points; KNS-A +26.4 points). Support for solving the problems of big cities was greater than for assistance to large cities (GSS +27.8 points; KNS-A +11.1), and spending for dealing with drug addiction was supported more than for drug rehabilitation (GSS +4.8 points; KNS-A +10.9 points). No ready explanation for these variations in magnitude is apparent.

The 2002 GSS/KNS experiments replicated key findings from the earlier 2000 GSS/KNS research in that differences in Don’t Know levels were confirmed when the DK response option was explicit in the KNS mode; some spending differences, especially those dealing with the urban underclass and the disadvantaged, were robust; and question wording effects were consistent in direction across time and mode, but variant in magnitude by mode.

The 2002 experiments also extended earlier results by showing that the level of DKs on e-surveys varies by format, that a web-based format can be devised that produces similar DK levels to those of in-person surveys, and that conditioning effects can occur after just a few items on web-based surveys. While some measurements were shown to be highly comparable across mode (such as DK levels when using appropriate formats and most distributions on the spending items), in other cases the results varied notably (such as support for spending on the urban underclass and the magnitude of wording effects).

Taken together, the results of these experiments indicate that differences across modes may tend to be stable, and that steps can be taken to reduce or eliminate some variation, but much more information is needed on differences in measurement across modes before results from web-based surveys and surveys using other modes can be considered generally comparable.

Tom W. Smith is director of the General Social Survey, National Opinion Research Center. J. Michael Dennis is vice president and managing director, Government and Academic Research, Knowledge Networks. The authors wish to thank Rick Li of Knowledge Networks (KN) for his direction of the KN data collection and his assistance with analysis.