Questions & Answers: Experts Comment on a Public Opinion Classic
In 2006 Gary Langer, the president of the New York chapter of the American Association for Public Opinion Research, invited Howard Schuman and Stanley Presser to present reflections on their book, Questions & Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. The book had been published a quarter of a century earlier in 1981, and this seemed to Schuman and Presser a good point at which to obtain an assessment of progress as seen by seven researchers who have dealt extensively with the question-answer process over the past twenty-five years. The request they sent and the replies they received are presented below. Readers should bear in mind that the replies, given here in alphabetical order, were informal email responses, and that the writers have developed their views more fully in their various publications, including some cited below.
Text of the Letter
In the way of minor anniversaries, this is the 25th year since the publication of our book Questions & Answers in Attitude Surveys, and Stanley Presser & I are giving a talk labeled "Reflections" late this fall [2006] in New York. We would like to note changes since our now ancient work, and that leads to the following inquiry: Of the problems we considered in the 1981 book (the table of contents is attached), is there one or more on which you believe there is now a fairly complete solution—i.e., good theory that explains most past results and allows reasonably confident prediction to future ones?
We would also appreciate important recent references to your writing that are directly relevant to this inquiry, especially something that goes beyond a specific experiment to consider more broadly a type of issue, such as the ones included in the attached [table of] contents.
Responses from:
From George Bishop
To begin with, I would maintain that there is, as yet, no satisfactory theory that provides a relatively complete explanation and prediction of any of the question effects identified by Howard Schuman and Stanley Presser in their 1981 book. The various cognitive models that have been developed by researchers such as Schwarz et al., Krosnick et al., and Tourangeau et al. provide more or less useful heuristic models for conceptualizing how respondents might answer survey questions and postulate seemingly plausible accounts of mediating processes (such as assimilation-contrast, priming, and belief-sampling). But none of these so-called models systematically identifies a set of causal variables and interrelated propositions from which we can derive other propositions that can be operationalized and tested in a way that would permit us to say we have explained and predicted a given question effect. What we seem to have at present is a loose mix of taxonomies and conceptual frameworks of various effects and processes and a handful of more or less established empirical generalizations.
To look at it from just a simple explained-variance perspective, I cannot think of a single effect for which I could say that the following set of independent and moderating/conditional variables accounts for 50 percent of the variance in, say, question order and context effects.
In what sense, then, has anything been explained thus far? Schuman and Presser's work and the research literature that has followed seems to me to provide mostly a plausible collection of what one might call middle-range hypotheses with varying degrees of support, but far from anything I would regard as an integrated theoretical explanation yielding reliable and valid predictions of different types of question effects under different conditions. Like a lot of other things, the so-called explanations that have been promulgated over the past twenty-five years since the publication of Questions & Answers (especially the cognitive “models”) strike me as more apparent than real theoretical systems. As I have argued in my recent book, The Illusion of Public Opinion, even the supposedly “axiomatic” model of the survey response developed by Zaller and Feldman, which dominates contemporary theorizing in political science, turns out to be another artifactual account of how respondents answer survey questions. Instead we often find ourselves struggling to explain question effects and artifacts generated by, among other things, the inherent vagueness and ambiguity of the language used in so many of our survey questions.
Additional Readings
Bishop, George F. 2005. The illusion of public opinion: Fact and artifact in American public opinion polls. Rowman & Littlefield Publishers.
———. 2003. Order Effects. In The Sage encyclopedia of social science research methods, ed. Michael Lewis-Beck, Alan Bryman, and Tim Futing Liao. Thousand Oaks, CA.: Sage Publications.
——— . 2005. Question order and context effects. In Polling America: An encyclopedia of public opinion, ed. Samuel Best and Benjamin Radcliff. Westport, CT: Greenwood Press.
Bishop, George F., and Andrew Smith. 2001. Response-order effects and the early Gallup split-ballots. Public Opinion Quarterly 65:479-505.
From Norman Bradburn
A lot of theoretical and empirical work has gone on the past twenty-five years, and I think it fair to say that we have pretty good explanations for some of the phenomena Schuman and Presser discuss in Questions & Answers, but there is still a lot of tidying up to do. The basic theoretic structure is a cognitive one which has evolved from Cannell et al., through Strack and Martin, Tourangeau and Rasinski to the latest version in Tourangeau, Rips, and Rasinski. Krosnick's satisficing model (1991) is somewhat to the side, but I think adds a motivational component that complements the others.
Let me go through the table of contents:
- Question order. I think we are in pretty good shape here. The various theories do a good job in demonstrating how context affects understanding of the question and activates various schema. Question order is an important part of the context, and, depending on the content of the questions, the order can affect comprehension or retrieval. More empirical work is needed before we identify exactly which questions will create important order effects, but the broad outlines are pretty clear: General questions—e.g., abortion—are affected by more specific questions—e.g., abortion if mother's life is in danger—but not vice versa. Questions that invoke a familiar norm—e.g., reciprocity—will affect later questions that imply the norm, but with a less favorable object—e.g., fighting for the Nazis. Chapters 4-6 of Sudman, Bradburn, and Schwarz, Thinking About Answers, goes over a lot of this ground, including order of answer category effects.
- Open versus closed questions. Less has been done here, since open questions have virtually disappeared from practice. I think the various theories that draw on cognitive theories of memory and activation of associations can pretty well explain what is going on. There are few empirical demonstrations but, I think the 1989 Tourangeau, Rasinski, Bradburn, and D'Andrade article, "Belief Accessibility and Context Effects in Attitude Measurement," demonstrates how it would work.
- No opinion. I don't know that much has been done in giving a good explanation for the effect of giving or not giving an NA option, although I think the argument and studies about order effects in the response alternatives literature would point one in the right direction (see chapter 6 of Thinking about Answers).
- Attitudes versus nonattitudes. I think the Tourangeau, Rips, and Rasinski belief-sampling model, described in chapter 6 of The Psychology of Survey Response, does a pretty good job of making sense of this issue.
- Measuring middle positions. I think less has been done in explaining what goes on here. Chapter 8 in The Psychology of Survey Response is relevant, and here Krosnick's satisficing model could come into play.
- Balance and imbalance in questions. I think this is a subtopic of the context literature and works rather like the mechanisms talked about in explaining the order effects.
- Acquiescence. Carol Stocking did a good job in showing that a lot of what we think of as acquiescence is really about the normative structure surrounding the interview, or at least it was some time ago when surveys were not so common. Given the declining response rates, acquiescence may be replaced by refusals, soft or hard. Still it continues to exist. My explanation is a social interaction model based on role theory, but that model has not been picked up by others—although I haven't seen anyone argue against it, either, so I don't know if that means that people accept it, or whether it is just seen as irrelevant.
- Intensity, strength, and crystallization. I think this bundle of issues is pretty well accounted for by the belief-sampling theory and related views about chronically accessible as opposed to temporarily accessible thoughts.
I guess the bottom line is that I think the developing cognitive theories are bringing considerable order into the explanations of the phenomena Schuman and Presser brought to prominent attention twenty-five years ago. Some of the things that looked like disparate issues really are the result of similar underlying cognitive processes, and, as we do more work on them, we will get a firmer handle on them in ways that lead to improvements in questionnaires and being able to know in advance when questions are going to cause problems. I wrote a paper a few years ago for Survey Methodology that summarizes most of what I know on these issues.
From Jon Krosnick
There is good theory that explains most past results and allows reasonably confident prediction to future ones in all the areas addressed by Schuman and Presser. At the time it was published, serious, in-depth psychological analysis had been done of some response effects Questions & Answers examined (such as acquiescence), and much work had been done on other response effects the book examined, but it hadn't yet been integrated into coordinated theoretical accounts drawing heavily on psychology.
My reading of the literature before and after Questions & Answers, pulling it all together in light of the conceptual framework that Schuman and Presser built, reveals a remarkable array of theoretical explanations for nearly all the response effects examined. The theoretical accounts don't explain all the variance, but they explain a lot. It is wonderful to see how Questions & Answers later inspired both original scholarship and careful looks backward at what was there but had never been coordinated into a single view. Very exciting for the field!
From Michael Schober
Although I believe that real progress has been made since Schuman and Presser's seminal work first came out, I'm not personally convinced that we have complete solutions to any of the problems considered there. I think we have more data points, and I think we have some useful systematizations that have moved our thinking forward. I'm a fan of many of the psychological formulations in Tourangeau, Rips, and Rasinski's book, The Psychology of Survey Response, and consider them, as well as the general thrust of formulating the questions in terms of the psychology of respondents (the CASM movement), to be steps forward.
But I can't say, as I look down the book's table of contents, that I feel I could confidently tell, say, a novice survey designer that we now understand the dynamics of any of those issues so straightforwardly that we can recommend how, exactly, a particular question should be worded on a particular survey. What we do have is a much better sense of the many potentially relevant considerations that survey designers need to think about.
My own view (developed with Fred Conrad, Herb Clark, and other colleagues) is that in working toward a general framework that makes sense of Schuman and Presser’s phenomena, it is fruitful to think of the question-answer process not only from an individual cognitivist view, but also from a collaborative interactivist view. The idea is that when respondents answer survey questions, either as presented by human interviewers or in self-administered formats, they are engaged in a complex form of dialogue, the features of which can help account for some of Schuman and Presser’s effects. Effects of prior questions can thus, for example, be conceived as related to the effects of the buildup of common ground in nonsurvey conversations. Various scholars have made versions of this argument, including Norbert Schwarz and colleagues and a number of writers in the Wiley Standardization and Tacit Knowledge volume. Fred Conrad and I have carried out some studies investigating interaction in interviews that reflect this view, and the papers below present overviews of the position:
Schober, M. F. 1999. Making sense of questions: An interactional approach. In Cognition and survey research, ed. M. G. Sirken, D. J. Hermann, S. Schechter, N. Schwarz, J. M. Tanur, and R. Tourangeau. New York: John Wiley & Sons, 77-93.
Schober, M. F., and F. G. Conrad. 2002. A collaborative view of standardized survey interviews. In Standardization and tacit knowledge: Interaction and practice in the survey interview, ed. D. Maynard, H. Houtkoop-Steenstra, N. C. Schaeffer, and J. van der Zouwen. New York: John Wiley & Sons, 67-94.
Schober, M. F. 2005. Conceptual alignment in conversation. In Other minds: How humans bridge the divide between self and others, ed. B. F. Malle and S. D. Hodges. New York: Guilford Press, 239-52.
Conrad, F. G., M. F. Schober, and T. Coiner. 2007. Bringing features of dialogue to web surveys. Applied Cognitive Psychology, in press.
Conrad, F. G., M. F. Schober, and W. Dijkstra. In press. Cues of communication difficulty in telephone interviews. In Advances in telephone survey methodology, ed. J. M. Lepkowski, C. Tucker, M. Brick, E. de Leeuw, L. Japec, P. Lavrakas, M. Link, and R. Sangster. New York: Wiley.
From Norbert Schwarz
I’ve been thinking about this interesting question for the last couple of days: What have we learned since Questions & Answers? Which issues have been “solved,” and which new ones have been added? Unfortunately, I’ve read too much Popper as a student to think we’ll ever have final answers on anything, but that’s a different story.
First of all, the question once again reminded me how well Schuman and Presser (1981) prepared the field for the cognitive turn it took a few years later. The book presented us with a clearly defined set of phenomena—some puzzling and some rather “obvious” from a social cognition perspective (which was a new baby at the time, with Wyer and Carlston’s 1979 book, Social Cognition, Inference, and Attribution, as the first “bible”). Without Questions & Answers, Hippler, Strack, and I—and probably many others—would not have been drawn to the interface of psychology and survey research. The book was the well-defined problem statement that gets an area going.
Second, thinking about this also made salient my disappointment that we haven’t really discovered “new” survey phenomena in the attitude domain that were not part of the book. There’s some new stuff on behavioral reports that came out of CASM, but as far as attitudes go, Schuman and Presser (1981) pretty much identified all the big issues for survey methodology, and later work mostly provided accounts of the underlying psychological processes.
So what do we know about how people answer attitude questions? Below I cite overviews, most of which are at my website. Here’s my (idiosyncratic) take:
- General Perspective. People form attitude judgments on the spot, mostly by constructing mental representations of the target and of a standard on the basis of information that comes to mind at that time. Which information that is depends on (1) how they construe the meaning of the question (a Gricean conversational story), (2) what was brought to mind by preceding questions (mostly a cognitive accessibility story), and (3) the order in which they consider the response alternatives (which prompt thoughts about the issue). None of this is an “artifact,” and all of it happens pretty much in the same way in the wild. The influence of natural contexts parallels the influence of question order; interviewing people at work, for example, produces similar answers as preceding the target question with work-related questions—in both cases, it’s work that’s on their minds, and this influences answers to all questions to which it may be relevant. The problem for surveys is merely that we mistake context-dependent answers for general ones—we learned how they think about X “when at work,” but treat it as if it were how they “always” think about X.
I’ve recently been arguing that, from the actor’s perspective, this “context dependency” is better framed as “context sensitivity,” and that it is highly beneficial. Any evaluative system that’s supposed to guide behavior needs to be context-sensitive (for the actor), but the observer doesn’t like that because it undermines prediction of the actor’s behavior. In my eyes, we’ve been doing attitude research from the observer’s perspective, and it’s time to try the actor’s perspective. That actor-observer distinction is actually something Schuman (1983) hinted at in Personality and Social Psychology Bulletin. I discuss this in a piece on “Attitude Construction” (written for a special issue of Social Cognition, forthcoming).
- Question order. I think we have gotten a pretty good handle on most question order effects. Some question order effects are primarily comprehension effects, and the meaning of the question shifts as a function of the context. That’s the Gricean story of my 1996 book, Cognition and Communication, and it also applies to graphical layouts, scale formats, and similar issues.
Other question order effects are a function of substantively related material that is brought to mind by earlier questions. I think we understand the conditions of assimilation (carry-over) and contrast (backfire), the determinants of their size, and the conditions under which the effects are limited to one attitude object or generalize over several objects. That’s the Schwarz and Bless inclusion/exclusion model (in Martin and Tesser, The Construction of Social Judgments, 1992), which is very much in agreement with Tourangeau’s model as far as assimilation effects are concerned, but adds an account of contrast effects that’s missing in the latter. Tourangeau discusses these issues in his chapter in Sirken et al. (1999), and a recent Schwarz and Bless piece is an update of our 1992 model.
Let me emphasize that Questions & Answers had an important impact here: Schuman and Presser's part-whole contrast effect identified a phenomenon that required the joint operation of cognitive and communicative processes, which is something that wasn’t known before. The story in Schwarz, Strack, and Mai's "Assimilation and Contrast Effects in Part-Whole Question Sequences" illustrates this interplay. So, in many ways, conversational and cognitive processes work hand in hand in question order effects.
We recently also learned that there is a nice cultural difference in the emergence of part-whole contrast effects. A short summary of that is in a 2003 piece on “Self-Reports across Cohorts and Cultures.”
Finally, preceding questions may bring norms to mind, as Questions & Answers illustrated with its norm-of-evenhandedness studies. I’m not aware of new work on that.
- Response order. This is a frustrating issue. In Thinking about Answers, we described a model that builds on Krosnick and Alwin and predicts an order x mode x item plausibility interaction. The model works fine if we write the response alternatives to be agreeable/plausible or disagreeable/implausible for most respondents. But most regular surveys present a mix of options that are differentially agreeable/plausible for different respondents, and we can’t make clear predictions for the sample as a whole. It’s an illustration for how a model that seems right theoretically, and works under tightly controlled conditions, is relatively useless for practitioners. As a psychologist it doesn't bother me (the mind isn't designed to make surveys easy), but as a survey methodologist it's very dissatisfying.
Making things worse, many survey tests of our model have dropped the “plausibility” variable. They essentially test what is theoretically a three-way interaction (order x mode x plausibility) as if it were a two-way (order x mode). The results are confusing—as they should be when you gloss over one crucial variable. George Bishop and Andrew Smith’s 2001 Public Opinion Quarterly piece illustrates this problem: The authors test a two-way interaction and conclude that a model that predicts a three-way interaction doesn’t work (see also Bishop's comments in this exchange). Unfortunately, such errors of analysis provide neither a basis for theoretical conclusions nor for applied advice.
The new thing we learned is that response order effects increase dramatically with age, in particular on the phone; see Knauper's 1999 study, "The Impact of Age and Education on Response Order Effects in Attitude Measurement."
- Nonattitudes/attitude strength. From an attitude construction perspective, the distinction between attitudes and nonattitudes is not theoretically meaningful. Moreover, there's no empirical relationship between attitude strength and susceptibility to context effects, despite popular beliefs to the opposite—Krosnick and Schuman (1988) put the nail into that coffin with a Journal of Personality and Social Psychology piece. But it leaves an interesting paradox: self-reported attitude strength does predict resistance to persuasion, but it does not predict susceptibility to context effects. I think I have an answer to why that is, and I discuss it in the “attitude construction” manuscript in a section on attitude strength (Schwarz, in press).
A special twist are questions about fictitious issues. In my reading, that was an artifact produced by researchers. Respondents have no reason to assume that we’d ask a meaningless question and peruse the context to make sense of it. Once they made sense of it, they can give a meaningful answer, as Schuman and Kalton (1985) noted. Our demonstration of that is Strack, Schwarz, and Wänke's Social Cognition, and I review it in chapter 4 of Cognition and Communication. The answers to questions about fictitious issues are not meaningless—only the questions are.
- Open versus closed. What we know is what Schuman and Presser (1981) said, and there are no qualifications to be added, at least none I’m aware of. I’m couching the differences in the language of Gricean conversational processes, where the response alternatives clarify what the researcher wants to know (chapter 5 of Cognition and Communication). This makes it look less “odd” (in fact, it’s perfectly consistent with proper conversational conduct) but doesn’t change anything about the robust effect.
- Don't know and middle alternative. Same thing. Nothing to be added to what Schuman and Presser (1981) said, I think. Except, perhaps, that “full filters” can convey the expectation that you should only answer when you are an expert. Hence, they discourage people from reporting opinions they would otherwise report (reviewed in Cognition and Communication, chapter 5).
- Tone of wording. Aside from Hippler and Schwarz (1986) on forbid/allow, we’ve never done much on that, nor have I seen much other work. Conceptually, I think that different tones convey differentially extreme meanings, which triggers a search for different information. For example, “What do you dislike about Bush?” versus “What do you hate about Bush?” would do so, and once that stuff has been brought to mind, you use it later on.
There are a few other topics in Questions & Answers that I have nothing to say on. Eric Knowles has done some interesting work on acquiescence, which is worth looking at.
That’s it. What really impresses me is how much Schuman and Presser (1981) set the agenda twenty-five years ago. What disappoints me is that while we did well in providing more process-oriented accounts of known phenomena, we didn’t put many new phenomena on the map (aside from extensions of familiar stuff, some age- and culture-related differences, some direction of comparison effects, and so on, but that’s all perfectly within the paradigm). So looking back, I feel that the effort was useful and certainly had considerable applied impact; but its pay-off in terms of novel psychological phenomena was more limited than I once hoped.
From Tom W. Smith
My initial reaction is that language, cognition, and communication are so complex that comprehensive and highly predictive understanding of any major aspect (question wording, context effects, and so forth) is likely to elude us for a very long time. We do know a large number of important, smaller things, like that double negatives confuse people and increase error in measurement. (But even that is probably language specific [that is, a major problem in English, but perhaps not in some other languages], and it may well interact with education and other factors.) Thus, while I think we know a lot more about good question design, measurement effects, and how people understand, process, and respond to questions than we did when Questions & Answers was published, I don't think we are near to achieving "lawfulness" in our understanding.
From Roger Tourangeau
Here are my very scattered thoughts:
I think we've made real progress on question order and response order effects. I'd cite the following key pieces on response order:
Knauper, B. 1998. The impact of age and education on response order effects in attitude measurement. Public Opinion Quarterly 63:347-70.
Krosnick, J. A., and D. Alwin. 1987. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly 51:201-19.
Narayan, S., and J. Krosnick. 1996. Education moderates some response effects in attitude measurement. Public Opinion Quarterly 60:58-88.
Schwarz, N., H. Hippler, and E. Noelle-Neumann. 1991. A cognitive model of response-order effects in survey measurement. In Context effects in social and psychological research, ed. N. Schwarz and S. Sudman. New York: Springer-Verlag, 187-201.
And these on question order:
Schwarz, N., and H. Bless. 1992. Constructing reality and its alternatives: Assimilation and contrasts effects in social judgment. In The Construction of Social Judgments, ed. L. L. Martin and A. Tesser. Hillsdale, NJ: Erlbaum, 217-45.
Schwarz, N., and H. Bless. 1992. Scandals and public trust in politicians: Assimilation and contrast effects.Personality and Social Psychology Bulletin 18:574-79.
Tourangeau, R. 1999. Contexts effects on answers to attitude questions. In Cognition and Survey Research, ed. M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. Tanur, and R. Tourangeau. New York: John Wiley & Sons, 111-31.
Tourangeau, R., and K. Rasinski. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin 103:299 314.
I don't have a strong sense of good progress on open versus closed questions, assessing no opinion, and measuring a middle position.
On attitudes and nonattitudes, I think there is real progress here. To the extent that most attitudes are constructed on the spot, the distinction between attitudes and nonattitudes may be overdrawn. Here are some papers I regard as important on this point:
Schwarz, N., and H. Bless. 1992. Constructing reality and its alternatives: Assimilation and contrasts effects in social judgment. In The Construction of Social Judgments, ed. L. L. Martin and A. Tesser. Hillsdale, N.J.: Erlbaum, 217-45.
Smith, T. W. 1984. Non-attitudes: A review and evaluation. In Surveying Subjective Phenomena, ed. C. F. Turner and E. Martin. New York: Russell Sage, 2:215-55.
Tourangeau, R., L. Rips, and K. Rasinski. 2000. The Psychology of Survey Response. Cambridge University Press, chapter 6.
Wilson, T. D., and S. Hodges. 1992. Attitudes as temporary constructions. In The Construction of Social Judgments, ed. L. Martin and A. Tesser. New York: Springer Verlag, 37-66.
Regarding acquiescence and the need for balance, I'm not sure we've made much headway; I know Jon [Krosnick] thinks we have, but I haven't seen the evidence.
I think we have made some progress on passionate attitudes and the meaning of crystallization; I'd cite:
Petty, R. E., and J. Krosnick. 1995. Attitude strength: Antecedents and consequences. Hillsdale, NJ: Erlbaum.
Also, Abelson's various pieces, for example,
Abelson, R. P. 1999. Conviction. American Psychologist 43: 267-75.
Krosnick, J. A., and R. P. Abelson. 1992. The case for measuring attitude strength in surveys. In Questions about questions: Inquiries into the cognitive bases of surveys, ed. J. Tanur. New York, NY: Russell Sage Foundation, 177-203.
I'm not sure we've gotten very far with tone of wording.
As for the general enterprise, Schuman and Presser remains a classic. Some of the issues seem a little dated (tone of wording is one), but many of them (acquiescence, balance, question order, and response order, among others) are just as important today as they were twenty-five years ago.
Afterword by Howard Schuman and Stanley Presser
Taken as a whole, the responses to our letter asking seven researchers to comment on Questions & Answers provide a valuable set of perspectives on recent research on the question-answer process. They convey a range of views as to how much the research on questions and answers has progressed over the past quarter of a century. At one end of the range is the belief that “there is good theory that explains most past results and allows reasonably confident prediction to future ones in all the areas” covered by the table of contents shown at the beginning of this presentation. At the other end of the range is the belief that there is “no satisfactory theory that provides a relatively complete explanation and prediction of any of the question effects” identified in that table of contents. The other statements can be seen as leaning toward one or the other of these two positions.
One possible explanation for the variation may have to do with how much weight is given to the understanding achieved with regard to well-studied past response effects versus how much precise guidance can be provided at this point to those constructing a new set of survey questions. Other readers may have different explanations. Whatever their interpretation, we believe those interested in the question-answer process will benefit greatly from both the ideas stated here and the accompanying references.