From George Bishop

Professor of political science and director of the Graduate Certificate Program in Public Opinion and Survey Research, University of Cincinnati

To begin with, I would maintain that there is, as yet, no satisfactory theory that provides a relatively complete explanation and prediction of any of the question effects identified by Howard Schuman and Stanley Presser in their 1981 book. The various cognitive models that have been developed by researchers such as Schwarz et al., Krosnick et al., and Tourangeau et al. provide more or less useful heuristic models for conceptualizing how respondents might answer survey questions and postulate seemingly plausible accounts of mediating processes (such as assimilation-contrast, priming, and belief-sampling). But none of these so-called models systematically identifies a set of causal variables and interrelated propositions from which we can derive other propositions that can be operationalized and tested in a way that would permit us to say we have explained and predicted a given question effect. What we seem to have at present is a loose mix of taxonomies and conceptual frameworks of various effects and processes and a handful of more or less established empirical generalizations.

To look at it from just a simple explained-variance perspective, I cannot think of a single effect for which I could say that the following set of independent and moderating/conditional variables accounts for 50 percent of the variance in, say, question order and context effects.

In what sense, then, has anything been explained thus far? Schuman and Presser's work and the research literature that has followed seems to me to provide mostly a plausible collection of what one might call middle-range hypotheses with varying degrees of support, but far from anything I would regard as an integrated theoretical explanation yielding reliable and valid predictions of different types of question effects under different conditions. Like a lot of other things, the so-called explanations that have been promulgated over the past twenty-five years since the publication of Questions & Answers (especially the cognitive “models”) strike me as more apparent than real theoretical systems. As I have argued in my recent book, The Illusion of Public Opinion, even the supposedly “axiomatic” model of the survey response developed by Zaller and Feldman, which dominates contemporary theorizing in political science, turns out to be another artifactual account of how respondents answer survey questions. Instead we often find ourselves struggling to explain question effects and artifacts generated by, among other things, the inherent vagueness and ambiguity of the language used in so many of our survey questions.

Additional Readings

Bishop, George F. 2005. The illusion of public opinion: Fact and artifact in American public opinion polls. Rowman & Littlefield Publishers.

———. 2003. Order Effects. In The Sage encyclopedia of social science research methods, ed. Michael Lewis-Beck, Alan Bryman, and Tim Futing Liao. Thousand Oaks, CA.: Sage Publications.

——— . 2005. Question order and context effects. In Polling America: An encyclopedia of public opinion, ed. Samuel Best and Benjamin Radcliff. Westport, CT: Greenwood Press.

Bishop, George F., and Andrew Smith. 2001. Response-order effects and the early Gallup split-ballots. Public Opinion Quarterly 65:479-505.

To Norman Bradburn

To first page