Questions & Answers: Twenty-Five Years of Relevance

Excerpts from Questions & Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context, by Howard Schuman and Stanley Presser, presented by POP as one in its occasional series of Public Opinion Classics.

Editor's note: In 2005, Questions & Answers in Attitude Surveys won the Philip Converse Award for the best book published at least five years previously in the field of elections, public opinion, and voting behavior, by the American Political Science Association's Section in that area. APSA's official award statement called it “one of those rare books that became an instant classic and deservedly so.” A quarter of a century after its publication, Questions & Answers is now a true classic in every sense of the word. To cap off our special presentation marking the twenty-fifth anniversary of Howard Schuman and Stanley Presser’s “essential point of reference for anyone who designs surveys or uses survey data,” the selections below from the book itself explain, first, what it was all about, and second, why it remained relevant at the time of its 1996 reprinting—explanations that apply just as readily today.

Scope and Method of the Original Work (from Chapter 1, 1981)

She said to me early in the afternoon,

“What is the answer?” I was silent.

“In that case,” she said,

“What is the question?”

                                          Alice B. Toklas, What Is Remembered

Despite repeated criticisms, polls and surveys continue to flourish in number and influence. The reason for their success is simple. They combine two things: the ancient but extremely efficient method of obtaining information from people by asking questions; and modern random sampling procedures that allow a relatively small number of such people to represent a much larger population. When the speed of computer processing and the power of multivariate analysis are added to the more basic ingredients of questions and samples, the whole can yield information and insights impossible to obtain in any other way about a large population. Yet all the technical developments of sampling theory, computing, and analysis are meaningful only as they facilitate use of inquiries not fundamentally different in nature from our most ordinary attempts to satisfy curiosity by asking someone something.

This book reports an extensive investigation of the kinds of questions asked in surveys, especially surveys dealing with what Turner and Krauss (1978) refer to as “subjective phenomena.” Although this focus means that we will ordinarily not treat questions about “facts” as problematic, it is well to keep in mind that many questions commonly regarded as factual have a large subjective component… Thus, although our main concern in this volume is with questions dealing with attitudes, opinions, beliefs, values, preferences, and so on—for shorthand we shall often refer to all of these as attitude questions—the research has implications for virtually all questions asked in polls and surveys.

Our main goal is to determine how the ways in which attitude questions are asked in surveys affect the results derived from these same surveys. Such results may be in the form of simple response distributions, as with the overall percentages saying “favor” and “oppose” on an issue, or they may involve associations between responses to two or more questions. The latter are particularly important when surveys are used as research tools, and much of our effort has gone into investigating the effects of question form on associations. Changes in single variables are also of interest, however, and we shall be concerned with these as well.

Questions can be asked in infinitely varied ways about an infinite number of subjects. It is not possible, except in a purposive or else purely metaphoric way, to sample the universe of all survey questions. Instead, our strategy has been to identify a small number of important ways in which questions vary in form, and investigate whether these variations have systematic effects that are detectable regardless of exact wording or subject matter. For example, almost any attitude question can include an explicit “don’t know” alternative, but most survey questions do not provide this option for the respondent…

Our primary way of proceeding has been to carry out experiments within surveys. Since our concern is to throw light on large-scale surveys, we have generally not used student or other special samples, but rather have worked with probability samples drawn from the national population, or, in a few cases, from the population of metropolitan Detroit. Within these surveys we ordinarily employ a between-subjects experimental design, randomly dividing the total sample into two or more subsamples, each of which is administered one form of a question. In public opinion research this method has traditionally been called the “split-ballot experiment,” and we sometimes use that term… In certain cases we also employ “panel data,” a form of within-subjects design where the same respondents are interviewed on two or more occasions. Whichever design was used, our aim was to combine the testing of causal hypotheses through experimental manipulation and randomization (“internal validity”) with the ability to generalize to the national population (“external validity”).

There is one other characteristic of our work that distinguishes it from most reports using survey data. Rather than testing many hypotheses once in a single effort, we have developed our ideas step by step, starting with a relatively few experiments, then attempting to profit from the results (and our mistakes) to design a new set of experiments, and so on. Altogether we have worked within more than 30 separate surveys over a 6-year period. Furthermore, we have frequently used replication to test the reliability of results, especially those that are unusual or counterintuitive. Although most social scientists recognize that statistical significance testing is of uncertain meaning in the context of extensive analysis of a single data set, the great cost and effort of carrying out a major survey usually precludes systematic replication. Our own approach, which involved buying small blocks of time in ongoing surveys, together with assumptions and experiences that emphasized the importance of replicability, led us in the opposite direction.

This then is the basic nature of the research to be reported. There are many important problems and results not hinted at, but the broadest outlines of the research have been laid out.

Still Relevant: Revised Preface to the 1996 Edition

There are several reasons why [Questions & Answers], first published in 1981, remains relevant today. First, Q&A, as we have come to call it, stimulated a considerable amount of later research on the question-answer process in surveys. It focused attention on a number of now common concepts such as “the subtraction effect,” “part-whole consistency,” “form-resistant correlations,” and “don’t know filters”; introduced some well-known experimental items (“the Agricultural Trade Act,” “the abortion context effect”); and raised still broader issues like the impact of attitude intensity on question effects and the difference between general and specific attitudes. By providing a theoretically organized approach to survey questioning and stressing associations among variables rather than single variable effects, the book reinvigorated a tradition of experimentation in surveys that had languished since the days of Cantril, Gallup, Hyman, and others.

Second, the experiments we report… are almost all based on carefully designed probability samples of the U.S. national population. Current theory in the areas we deal with, including developments under the now common heading of “cognitive research,” must be able to encompass the results from such general population studies. Thus our findings continue to provide “data” for those interested in exploring the question-answer process, and in that sense this remains a sourcebook for those wishing to explain effects due to question form, wording, or context. There have already been a number of reanalyses of our experiments, made easier because the original data are available through the Inter-university Consortium for Political and Social Research, and we hope others will see ways of looking at this evidence that go beyond our own analyses.

Moreover, it becomes increasingly possible to repeat earlier experiments with an eye to change over time. Some of our most interesting findings occurred when we replicated research carried out by others three or four decades before we came on the scene... No doubt many of our own experiments can now be treated as earlier time points for new replications that investigate the impact of social and linguistic change on question wording effects. Experiments in survey research—as in all social research—can never be timeless but must always allow for changes in the meaning of questions to new generations.

Finally we think that the method pursued here is also instructive. Rather than the one-shot experiment, we were able to replicate and develop results from one experiment to another. This frequently showed that earlier findings were not reliable or were inadequately interpreted. Our experience has made us skeptical of the quickly published finding that is intuitively (or even counterintuitively) persuasive but has not been tested in realistic surveys through both literal and construct replication.

We continue to find questions about questions a fascinating area for research, one that is important to survey methodologists, to cognitive and attitude theorists, and to both producers and users of survey results. Having returned to this book on numerous occasions in our own later research, we hope it will continue to be of value to others as well.

Excerpted from Questions & Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context, by Howard Schuman and Stanley Presser. Reprint edition, copyright © 2004 by Sage Publications. Reprinted by permission.

More classics of public opinion research