In Print: Psychometric Theories of Survey Response

An excerpt from The Psychology of Survey Response, winner of the 2006 AAPOR Book Award.

Perhaps the first models of the survey response process were the ones developed by psychometricians—Guttman, Guilford, Likert, and their colleagues. This research produced the first systematic attitude measurement techniques, techniques that quantified the strength of a person’s conviction in an opinion (e.g., How strongly do you favor increasing aid to the homeless?). Of these early pioneers, Louis Thurstone was perhaps the leader in describing the underlying psychological processes that made attitude measurement possible. Thurstone’s goal was to develop mathematical models that described the outcome of comparisons among several stimuli, including several statements about an issue (e.g., Aid to the homeless should be increased; Society already does enough for the homeless; The homeless are a nuisance).

Thurstone’s formal models rested on a psychological theory of judgment: the idea that people represent stimuli as points or regions on an internal dimension (e.g., the dimension of strength of agreement with a position). Thus, his psychological models distinguished several component processes, including the identification of the dimension of judgment (or, as Thurstone referred to it, the psychological continuum), the judge’s reaction to the stimulus (the discriminal process), the assignment of scale values, and the comparison of pairs of stimuli (the calculation of their discriminal difference). In the case of an attitude question, Thurstonian theory predicts that people compare their own position with what they take to be the position implied by each statement in the scale. The result of the comparison determines which statements they endorse.

For a variety of reasons, these psychometric models have not played a prominent role in discussion of survey error. One major problem is that, even though their emphasis is statistical, the psychometricians’ mathematical models are not easily translated into the terms that have come to dominate discussions of survey error; the survey error models are extensions of those used to analyze random sampling error, and they focus on factual questions, for which it is possible (at least in principle) to measure the accuracy of the answer. In addition, from our current vantage point, the earlier psychometric models are hamstrung by the absence of a detailed description of the judgment process. Although Thurstone’s model presents one view of how people choose among response options, it is silent on questions of how they identify the psychological continuum on the basis of the survey question, how they recruit relevant attitude information from memory, and many other crucial issues.

The model proposed by Cannell, Miller, and Oksenberg was perhaps the first model of the survey response process to reflect the new cognitive outlook within psychology. Their model distinguished two routes to an answer, one based on relatively careful processing of the question and the other based on superficial features of the interview situation, such as the interviewer’s appearance. Careful answers are, according to this model, the product of the five sets of processes:

. . . Cannell, Miller, and Oksenberg view these five sets of processes as more or less sequential, although their model explicitly allowed for the possibility that respondents might cycle back to an earlier stage if they judged their preliminary answer not to be accurate enough. Prior to the fifth stage, when respondents give an accurate (or at least adequate) answer, respondents could switch to the parallel track and alter their answer based on relatively superficial cues available in the interview situation—cues such as the interviewer’s appearance or the implied direction of the question. Responses based on such cues were likely to be biased by acquiescence (the tendency to agree) or social desirability (the need to present oneself in a favorable light).

The model by Cannell and his colleagues has many attractive features, and it has spawned many related approaches. The notion that respondents might take different routes to arrive at an answer is an appealing one, and the specific routes in the Cannell model—one based on systematic processing of the question and the other based on more superficial processing—have a number of parallels in the psychological literature. For example, discussions of attitude change have identified central and peripheral routes to persuasion. Hastie and Park’s distinction between memory-based judgments and on-line (i.e., situation-based) judgments also bears similarities to the two [routes] . . . Another attractive feature of the model is its explicit concern with the respondent’s motivation, including such motives as the desire to provide accurate information, to appear agreeable, and to avoid embarrassment.

From our viewpoint, the model suffers from two major drawbacks. The first is that, because the model never assumed a central place in Cannell’s work, it was never worked out in much detail. The most complete exposition of the model runs no more than three pages. Most of the research inspired by the model has focused on improving respondent motivation in a general way rather than on testing predictions regarding the model’s specific components. The second drawback is related to the first. It is the model’s rather sketchy treatment of the cognitive processes involved in responding to a question, which the model’s second stage lumps together. By contrast, the model distinguishes several stages in describing what happens after the respondent derives a preliminary answer. The respondent evaluates the initial answer in terms of its accuracy, then in terms of its compatibility with other goals, and finally may modify or discard it based on these earlier assessments. The model seems to assume that respondents could answer questions accurately if only they wanted to and concentrates on whether they decide to answer accurately or not. We favor a different emphasis.

We [present] a model that divides the survey response process into four major components—comprehension of the item, retrieval of relevant information, use of that information to make required judgments, and selection and reporting of an answer. [The table below] lists each of these components along with specific mental processes that they might include.

In describing these processes, we don’t mean to suggest that respondents necessarily perform them all when they answer a survey question. Although some processes may be mandatory, others are clearly optional—a set of cognitive tools that respondents can use in constructing their answer. Exactly which set of processes they carry out will depend on how accurate they want their answer to be, on how quickly they need to produce it, and on many other factors. In this respect, the theory presented in [the table] resembles approaches to decision making that emphasize the array of strategies that people bring to bear on a problem. . .

Each of these components can give rise to response effects; respondents may, for example, misinterpret the question, forget crucial information, make erroneous inferences based on what they do retrieve, or map their answers onto an inappropriate response category. Both psychological and survey research provide ample evidence of the errors each component produces, and to understand how the response process can go awry, we need to take a closer look at them.