The importance of the contexts in which multicultural surveys are being conducted becomes more apparent when one considers the range of situations in which these surveys are done. While many of the issues described here also occur in surveys conducted in the United States, research techniques that have proved effective with U.S. population groups do not necessarily have the same results when applied to foreign populations. This makes a systematic approach to instrument development and testing especially important, but the literature suggests that many researchers do not currently take full advantage of available techniques, particularly pretesting.

For example, we have previously proposed the process quality approach for systematizing the stages involved in multicultural instrument development. As in manufacturing, the quality of the product is improved by improving each stage of the process. The process quality approach ensures thorough and systematic coverage of the tasks, issues, and potential problems associated with instrument development, including testing parameters, cultural concerns, and specific defects in questions and supporting materials, with research and cultural experts collaborating to ensure proper coverage. Since survey instruments often are flawed for reasons that, in retrospect, appear quite simple and apparent, it is important to be sure that “all the rocks have been turned over.”

In a process quality approach to questionnaire development, the researcher systematically considers the particular set of cultural problems that might arise in a study. This set of problem areas guides the selection of pretesting techniques, how they might need to be adapted, and the sequence of their use.  The goal is to make sure that the full choice of survey procedures fits the culture, with any changes pretested in the new cultural context.

Harmonization is another approach for adapting existing techniques for multicultural research. Harmonization strategies typically are applied at the beginning of research to make surveys comparable across cultures and are used to implement standards, such as definitions or concepts, and data collection protocols for standardized interviewing. 

Harmonization can also be used retroactively, at the statistical or data level. When a measurement procedure for a common variable is made valid in both national and international contexts, then the technique is called “ex-ante output harmonization.”

In a recent paper, one researcher outlines the tightrope-like act of “balancing standardization with appropriate/necessary local adaptations” when using harmonization methods to deal with sampling, design, staffing, procedures, local customs, and other such issues in a large-scale, cross-sectional mental health study in twenty-eight countries and in thirty-five languages. The challenge of harmonization is about maintaining the balance between standardization and adaptation to the local culture, particularly when selecting samples, dealing with human subject review boards, and implementing quality control processes and procedures.

Calibration is an example of a method that can be used to address the problem of being able to replicate measures when designing and administering instruments in multiple languages. Some researchers calibrate response scales so that a scale designed for one culture will provide consistent measurement in another. Special attention is given to how the actual response categories affect the results and to ways of measuring the intensity (or strength) of the response categories. One form of calibration is to ask respondents to rate terms, for instance, from weaker to stronger, that define each point on a scale. Another approach examines the impact of miscalibrations, or differences between intervals assumed to be equal in a scale, since different respondents might think of the intervals in different ways. It is important to recognize this is no easy task; it is a complex effort that requires special methodological expertise.

As we have shown, many issues that are apparent in other countries are the same as or similar to those found in the United States, although some issues might be more intensified or more sensitive in a different cultural setting. This suggests that an additional range of cultural and communicative issues should be addressed when designing surveys within foreign contexts. These types of issues, including norms and behaviors, often are unintentionally overlooked when surveys are designed for another culture or meant to serve as a standard core that can be adapted to multiple cultures. Perhaps applying one or more of the structured approaches described could be an effective way to reduce the likelihood of missing or underestimating the importance of such culture-specific factors in future research studies.

Johnny Blair is principal scientist and Linda Piccinino is scientist/associate at Abt Associates Inc., Bethesda, Maryland.