All in the Timing: In-the-Field Testing of a Post-Disaster Questionnaire by Stephanie Willson, Kristen Miller, and Karen Whitaker

Another problem associated with the cognitive testing timeframe not matching the survey timeframe arose with these two questions:

After major disasters, people often have to deal with many new problems of daily life. What are some problems you most urgently need help with right away?

What are the most practical actions that your city or state government could take to be helpful to you right now?

In each case, respondents tried to decipher whether the question was asking about their current situation, or their situation right after the hurricane. A typical response was, "You mean right now?" While the question literally asks "right now" or "right away," respondents accurately assumed that its true intent was to ascertain their needs immediately following the hurricane. However, the answers they provided were a mix of what they currently needed (such as roof repairs and new furniture) with what they thought they needed a month ago (electricity, ice, bottled water). The goal of this question is not to have respondents reconstruct their past, but rather to think about their current situations, so testing it outside of the exact timeframe assumed by the survey only served to introduce error and confound results.

The timeframe issue set up context-effect problems as well. We found that even when a question in no way referenced or even alluded to the hurricane, respondents still supplied answers that reflected their hurricane experience. One question at the end of the survey asked, "How would you currently rate your overall hope for the future?" Rather than focusing their attention on the word "overall" as intended, some respondents understood this as asking directly about the hurricane. As a result, they made comments such as, "Everyone will be better prepared"; "Since the hurricane and since answering these questions, I can do what I've got to do and overpower the fear [of hurricanes]"; and "This has been the worst hurricane season people have been through, so I'm hopeful that next year won't be this bad." It is not the intent of the question to capture respondents' beliefs about how they feel about future hurricanes, yet that's the meaning many test participants gleaned from the question.

The choice of fieldwork allows a great deal of control over when and where cognitive testing will be conducted, and researchers should put equal thought and consideration into both aspects. In the case of the current questionnaire, the precise timing of fieldwork was to some extent beyond our control due to the unpredictable nature of the disaster, yet it had an important impact on the cognitive testing results. This lesson has implications for the administration of the actual questionnaire, as the amount of control over the administration of the survey itself may also be somewhat unpredictable. Designers of surveys with any event-based topic should strongly consider the implications of when the instrument can be fielded vis-à-vis the timeframe demands of its questions. In post-hurricane Florida, factors related to a natural disaster prevented us from interviewing within the ideal timeframe; however, any number of factors (such as staffing, funding, or other logistics) may potentially affect when surveys are fielded.

In most cognitive testing projects, much effort is made to select respondents that meet the population criteria for a study, even if interviewing the appropriate subjects necessitates fieldwork. However, timing is also an important, though perhaps underappreciated, aspect of the cognitive evaluation process. Not testing questions within the timeframe assumed by the survey can have a confounding effect on cognitive testing results. In essence, the questions as they appear in the instrument will not be the ones tested if respondents are confused by the difference between the intended timeframe and the actual one. Care should be taken in deciding when to embark on fieldwork when the survey is time-sensitive, event-based, or both.

Stephanie Willson and Kristen Miller are Research Survey Statisticians and Karen Whitaker is Questionnaire Design Research Laboratory Manager at the National Center for Health Statistics.