In Print: Why Do Survey Questions Produce Response Error? A Sociolinguistic Perspective by Gordon B. Willis

An excerpt from Cognitive Interviewing: A Tool for Improving Questionnaire Design

There are a variety of reasons why survey questions produce response error. People may not have the information we seek, or they may not remember it, or it may be something they would rather not talk about. Each is an important factor, and it may not be fair to single out any one as the most villainous. I propose, however, that one particular characteristic of survey questions consistently acts to subvert our best intentions and has emerged repeatedly in the literature: the failure to achieve the fundamental goal of clear communication between the parties. This problem can have a number of sources. For one, we may simply ask questions that are too difficult to understand. In fact, there is some evidence that we routinely overestimate the ability of information recipients to comprehend what we are attempting to convey. [W.A.] Belson describes research that assessed audience comprehension of radio addresses intended for the general public and reported "a considerable failure to grasp either major or minor points in these talks, the average score for the 26 talks being 27 per cent." The lesson, simply, is that simplicity is a virtue.

Survey questions fail in a second major way that is subtle but particularly vexing: they exhibit unacceptable vagueness in expressing the designer's intentions, or what is referred to as the pragmatic (rather than literal) meaning of the question. If there is room for varied interpretation, exactly what information are we asking our respondents for? It may be surprising that communicating our intentions should be so difficult. Given that most people have a lifetime of experience in asking questions of others, we presumably should be adept at employing this accumulated expertise when crafting survey questions.

However, sociolinguistic analysis suggests that the mechanisms of successful questioning behavior in everyday life are actually very complex and not easily transferable to the survey environment. Researchers who analyze real-life conversations have pointed out that conversers follow implicit rules of interaction, such as the assumption that the questioner will not ask for the same information more than once. A particularly relevant observation is that people normally communicate through the critical process of establishing common grounding. In essence, the argument holds that the solitary, unelaborated query (that is, asking a question and expecting a straightforward answer) is often inadequate, because the strings of words that constitute natural language are inherently ambiguous. Rather, conversations entail additional activities that result in a state of shared interpretation, especially with respect to the speaker's information needs.

A key postulate is that conversation is negotiated and terminated only when the participants have determined that the desired state of common meaning has been achieved, even if this ending state is achieved via a circuitous route. If the inquisitor asks, "How far is it from Huddersfield to Coventry?" and the respondent replies, "About a hundred miles," the former can then add, "No, I mean how long does it take to get there," if his or her objective is to obtain information in terms of time as opposed to distance. Speakers rely widely on such repair mechanisms to put the conversation back on track when there are indications that an initial interpretation was inadequate.

Perhaps closer to home for some, consider the following hypothetical, but hopefully realistic, morning exchange between wife and husband:

SHE: What are you doing today?

HE: Um-working. It's Monday...

SHE: I mean, are you working at home or going in? I have a late meeting so I thought if you were here... you could pick up Susie from school.

HE: Yeah, I'm going in, but I can leave early. I'll get her.

As in the previous example, this interchange can be characterized as a "closing in" on a state of shared meaning and purpose, especially as the initial inquiry gives rise to multiple potential interpretations. Additionally, the opening query in this example is vague, and even worse, is only peripherally related to the key item of information required. Rather, it sets the basis for further progression of the conversation towards common ground, and ultimately to the desired outcome: to obtain an answer to the question "Will you pick up Susie from school today?" Of course, there is no guarantee that such adaptive speech mechanisms always work. As may be obvious to married couples, nontrivial misunderstandings between conversers are rife and have caused all forms of calamity throughout history including, presumably, failure to pick up children from school.

In contrast to conversational speech, survey questions usually do not allow for the flexible interactions that establish grounding. This is true by definition for self-administered questionnaires, for which the questions are preassembled and administered in the absence of any questioner. But this situation also applies to interviewer-administered surveys. By scripting a series of survey questions as a type of artificial conversation, we plan the entire interchange before the fact and attempt to anticipate every move the respondent can make. Such scripting can be a simple series of mostly unrelated questions, but it is often exceedingly complex, such as when the answers to questions have implications concerning the question that should be asked next (that is, they affect skip patterns, or sequencing instructions).

As an example of the prototype questionnaire, consider two questions from a U.S. Census Bureau Survey of Income and Program Participation:

• Do you have a physical, mental, or other health condition which limits the kind or amount of work you can do? (YES/NO)

• Medicare is a health insurance program for disabled persons and persons 65 or over. People covered by Medicare have a card that looks like this (SHOW FLASHCARD). During this four-month period, were you covered by Medicare? (YES/NO)

These formulations do not resemble a natural exchange, such as between spouses. Note in particular that a significant burden is imposed by inflexibility in wording and especially by the assumption that the question will be answered without further cycles of interchange that progress toward a common ground. Consequently, questionnaire designers attempt to design questions that by their nature achieve grounding, rather than leave the interviewer and the respondent to establish shared meaning via flexible forms of adaptive speech. In fact, common survey procedure requires that if a respondent asks "What does [term] mean?" the appropriate response is "Whatever it means to you"; this could be viewed as a somewhat perverse attempt to avoid common grounding.

Because of such departures from normal conversation, several authors have advocated the nonstandardized interview, which allows the interviewer to deviate from a set script and to effectively interpret and communicate the investigator's objectives. This notion is not novel. The "nondirective interview" was introduced during the 1930s, and it is common practice in some fields, including medical research and social work, to rely on nonstandardized semi-structured interviews, which typically involve an interview guide as opposed to a fully scripted questionnaire. However, the nonstandard interview has not, at this point, taken hold within the conduct of most large-scale surveys.