Current Issues in Multicultural Survey Methodology
By Johnny Blair and Linda Piccinino
Multicultural, cross-national, or cross-cultural surveys raise issues that the current state of survey methodology is only partly able to address. As surveys that traverse cultural boundaries become more prevalent, questions of comparability of results, new sources of measurement error, and sampling and nonresponse bias become more pressing, especially where the data are used in support of such vital matters as government policies, business decisions, program evaluation, and investigations of human rights. What issues should be considered when a survey must accommodate multiple cultures, or when a survey designed in one cultural context must be applied to another? Different language groups are an obvious starting point for such a discussion, but one that can be oversimplified by limiting consideration to accurate translation of the survey questionnaire. For one thing, the way people interpret things they hear often goes beyond the literal meaning of the words in a statement or question. The occurrences of question order effects also can vary from one culture to another. The respondents’ understanding of the general intent of a survey, what the data will be used for, and assurances of confidentiality may all affect how they understand and respond to survey questions. Even when respondents are multilingual, a survey conducted in a language other than their first presents risks of miscomprehension that differ from those affecting native speakers. Going beyond considerations of language are features of cognition and socialization that affect response behaviors and response effects. Social desirability may manifest itself differently for respondents from different cultural backgrounds, for example. Differences in how respondents process some types of questions—both in terms of comprehension and response formation—can vary by racial or ethnic groups. Immigrant groups might not share assumptions that are sometimes implicitly part of survey instruments, pertaining, for instance, to how they decide when to seek health care or their perceptions of people’s rights in the workplace. Socioeconomic status might affect surveys, if, for instance, there is a large class difference between the interviewer and the respondent.
The fact that a culture rarely can be completely defined or described by a single factor that a researcher can take into account makes the design of multicultural surveys a difficult and complex undertaking. Since researchers designing surveys for their own cultures and countries might, almost automatically, account for many cultural factors, particularly societal norms for behaviors and interactions, they sometimes unconsciously overlook them when designing a survey for another culture, transferring a survey between cultures, or designing a single survey meant to be adaptable to multiple cultures. For this reason, researchers need to apply a systematic approach to designing multicultural surveys. With regard to sampling, we may begin by noting that, for many countries, the use of a random sample survey is not a feasible option. Technical difficulties aside, sampling may be complicated by issues of equity, ethics, and limited resources. For example, a survey may be used to evaluate a program for which the treatment areas have been “hand-picked” because of political and geographic preferences, or where staff and funds for probability sampling are scarce. Sampling frames in many countries often can prove difficult to locate, or are incomplete or out-of-date. Data specific to cultural subpopulations that could aid sample design frequently do not exist, are hard to find and obtain, or are of unknown accuracy. Nonsampling error, which may also have serious effects on survey results, can easily go unnoticed altogether. Respondents sometimes answer questions even if the questions are not entirely clear or are unsuited to their lives in some way. Differences in socioeconomic status or doubts about confidentiality can affect responses without the slightest indication that error is being introduced into the survey. Both sampling and nonsampling error can have serious consequences for the ultimate use of data for policymaking or for reporting on conditions such as the impact of natural and manmade disasters.
|
Levels of respondent cooperation may be affected by factors specific to different cultures, and some situations may need to be handled differently depending on the group being approached. For instance, in some surveys, such as HIV/AIDS studies, sensitive or invasive questions often need to be asked before or at the beginning of the interview process to determine eligibility for the survey. In such cases, the cultural context may dictate whether respondent-recruiting and screening activities can be conducted in an open manner or should be kept clandestine. Asking some cultural subgroups about their sexual preferences or HIV status may be very different when conducting a full-length health or sexual behavior survey as compared to a short screener instrument. The level of cooperation can be limited by fear of reprisals or the lack of, or uncertainty about, confidentiality and privacy assurances. In countries where the level of risk behaviors or disease infection rates is high, the target population can become fatigued or reluctant to participate in surveys as the result of being overstudied. On the other hand, response can be overwhelmingly participatory, perhaps from habits of compliance left over from past regimes, as in some former dictatorships where cooperation in government-run projects was mandatory. Other problems that have not been widely addressed in the literature are often encountered in developing questionnaires for multicultural surveys. Researchers designing instruments “from scratch” for use in cultures not their own face difficulties when an instrument already administered in one cultural setting is required to provide comparable measurements in another. Or an instrument might need to be designed for use across different cultures, either in a single survey or in multiple independent surveys.
Multicultural survey research becomes even more challenging where special circumstances, such as hazardous locations or difficult topics of research, add another layer of complexity to an already complicated task. Researchers planning and conducting surveys in war-torn or impoverished areas have safety and logistical issues to contend with. For example, in a study on mortality after the 2003 invasion of Iraq, researcher Gilbert Burnham and his colleagues found they could not replicate a sampling strategy they had used only two years before, involving the use of global positioning system devices, because their lives might be endangered by being seen with the GPS units. They were forced to adopt a suboptimal but less risky sampling strategy. Human rights research is severely hampered when existing information systems to aid data collection are slow, overloaded, and chaotic because of human rights emergencies. These extraordinary surveys will have all of the multicultural problems we have outlined, and others as well. Beyond taking whatever steps are feasible to mitigate the impact of unfavorable circumstances, researchers have a particular obligation to note (and quantify, if possible) factors beyond their control that might affect results, acknowledging the survey’s limitations in their methodological reports even when their impact cannot be precisely known.
|
The importance of the contexts in which multicultural surveys are being conducted becomes more apparent when one considers the range of situations in which these surveys are done. While many of the issues described here also occur in surveys conducted in the United States, research techniques that have proved effective with U.S. population groups do not necessarily have the same results when applied to foreign populations. This makes a systematic approach to instrument development and testing especially important, but the literature suggests that many researchers do not currently take full advantage of available techniques, particularly pretesting. For example, we have previously proposed the process quality approach for systematizing the stages involved in multicultural instrument development. As in manufacturing, the quality of the product is improved by improving each stage of the process. The process quality approach ensures thorough and systematic coverage of the tasks, issues, and potential problems associated with instrument development, including testing parameters, cultural concerns, and specific defects in questions and supporting materials, with research and cultural experts collaborating to ensure proper coverage. Since survey instruments often are flawed for reasons that, in retrospect, appear quite simple and apparent, it is important to be sure that “all the rocks have been turned over.” In a process quality approach to questionnaire development, the researcher systematically considers the particular set of cultural problems that might arise in a study. This set of problem areas guides the selection of pretesting techniques, how they might need to be adapted, and the sequence of their use. The goal is to make sure that the full choice of survey procedures fits the culture, with any changes pretested in the new cultural context.
Harmonization is another approach for adapting existing techniques for multicultural research. Harmonization strategies typically are applied at the beginning of research to make surveys comparable across cultures and are used to implement standards, such as definitions or concepts, and data collection protocols for standardized interviewing. Harmonization can also be used retroactively, at the statistical or data level. When a measurement procedure for a common variable is made valid in both national and international contexts, then the technique is called “ex-ante output harmonization.” In a recent paper, one researcher outlines the tightrope-like act of “balancing standardization with appropriate/necessary local adaptations” when using harmonization methods to deal with sampling, design, staffing, procedures, local customs, and other such issues in a large-scale, cross-sectional mental health study in twenty-eight countries and in thirty-five languages. The challenge of harmonization is about maintaining the balance between standardization and adaptation to the local culture, particularly when selecting samples, dealing with human subject review boards, and implementing quality control processes and procedures. Calibration is an example of a method that can be used to address the problem of being able to replicate measures when designing and administering instruments in multiple languages. Some researchers calibrate response scales so that a scale designed for one culture will provide consistent measurement in another. Special attention is given to how the actual response categories affect the results and to ways of measuring the intensity (or strength) of the response categories. One form of calibration is to ask respondents to rate terms, for instance, from weaker to stronger, that define each point on a scale. Another approach examines the impact of miscalibrations, or differences between intervals assumed to be equal in a scale, since different respondents might think of the intervals in different ways. It is important to recognize this is no easy task; it is a complex effort that requires special methodological expertise. As we have shown, many issues that are apparent in other countries are the same as or similar to those found in the United States, although some issues might be more intensified or more sensitive in a different cultural setting. This suggests that an additional range of cultural and communicative issues should be addressed when designing surveys within foreign contexts. These types of issues, including norms and behaviors, often are unintentionally overlooked when surveys are designed for another culture or meant to serve as a standard core that can be adapted to multiple cultures. Perhaps applying one or more of the structured approaches described could be an effective way to reduce the likelihood of missing or underestimating the importance of such culture-specific factors in future research studies.
Johnny Blair is principal scientist and Linda Piccinino is scientist/associate at Abt Associates Inc., Bethesda, Maryland.
|