All in the Timing: In-the-Field Testing of a Post-Disaster Questionnaire

By Stephanie Willson, Kristen Miller, and Karen Whitaker

In the past decade, cognitive testing has emerged as a pre-testing standard for new survey questions. Cognitive evaluation of survey questions can improve the analyst's knowledge of how respondents interpret questions and whether interpretations match the designer's intent, as well as provide insight for changes to question wording that would ultimately improve construct validity. As a result, federal agencies such as the National Center for Health Statistics (NCHS), the Census Bureau, and the Bureau of Labor Statistics now conduct regular cognitive testing and have established special facilities to conduct cognitive interviews on the premises. Sometimes, however, reaching suitable respondents on whom to test survey questions can be a challenge.

The NCHS has been conducting a project to test survey questions intended to be fielded immediately after a natural disaster or terrorist attack. The questions are part of an interagency effort to develop methods and measures for assessing and monitoring the psychosocial needs following extreme events. The goal of the instrument discussed here is to ascertain the extent of losses resulting from disasters and other emergencies and to track mental health and related practical needs of the community over time. To assess these questions, staff of the NCHS Questionnaire Design Research Laboratory (QDRL) conducted twenty-four cognitive interviews in post-hurricane Florida last fall.

Typically, cognitive testing takes place in a laboratory and involves in-depth, face-to-face interviews with respondents who are similar to the intended survey population. The QDRL's location in a major metropolitan area (just outside of Washington, DC) is frequently an asset in the recruitment of respondents. Participants with a variety of demographic characteristics and experiences can usually be identified, providing staff with good information while also saving time and money.

This generally efficient method, however, cannot always be used. Some projects involve specific target groups that either are not accessible in the DC metropolitan area or, if identified, cannot be brought to the lab. For example, particular demographics (such as the rural poor, or American Indians living on reservations) or groups with particular experiences (those affected by a disaster; those who donate plasma) may be geographically unavailable.

Given the intended population of the current survey, two conceivable strategies could have been used to evaluate questions in the lab. First, since a natural disaster has not occurred in the DC area in recent memory, we could have asked hypothetical questions of respondents. However, asking respondents hypothetical questions puts them in a difficult position. We would be testing questions for which there is no "actual" answer, which makes identifying response error very difficult.

Another approach could have been to ask respondents about a past disaster they actually did experience. Unfortunately, significant memory loss is associated with experiences that occurred long ago. Although there is some evidence that memory of significant and rare events is more reliable than that of routine common ones, respondents could not be expected to retain the level of detail required by the questionnaire. Given the limitations of these approaches, we decided that fieldwork-going offsite to recruit and interview the appropriate participants for cognitive interviewing-would be a more productive and appropriate strategy than testing in the lab.