Current Issues in Multicultural Survey Methodology

By Johnny Blair and Linda Piccinino

Multicultural, cross-national, or cross-cultural surveys raise issues that the current state of survey methodology is only partly able to address. As surveys that traverse cultural boundaries become more prevalent, questions of comparability of results, new sources of measurement error, and sampling and nonresponse bias become more pressing, especially where the data are used in support of such vital matters as government policies, business decisions, program evaluation, and investigations of human rights. What issues should be considered when a survey must accommodate multiple cultures, or when a survey designed in one cultural context must be applied to another?

Different language groups are an obvious starting point for such a discussion, but one that can be oversimplified by limiting consideration to accurate translation of the survey questionnaire.

For one thing, the way people interpret things they hear often goes beyond the literal meaning of the words in a statement or question. The occurrences of question order effects also can vary from one culture to another. The respondents’ understanding of the general intent of a survey, what the data will be used for, and assurances of confidentiality may all affect how they understand and respond to survey questions. Even when respondents are multilingual, a survey conducted in a language other than their first presents risks of miscomprehension that differ from those affecting native speakers.

Going beyond considerations of language are features of cognition and socialization that affect response behaviors and response effects. Social desirability may manifest itself differently for respondents from different cultural backgrounds, for example. Differences in how respondents process some types of questions—both in terms of comprehension and response formation—can vary by racial or ethnic groups. Immigrant groups might not share assumptions that are sometimes implicitly part of survey instruments, pertaining, for instance, to how they decide when to seek health care or their perceptions of people’s rights in the workplace. Socioeconomic status might affect surveys, if, for instance, there is a large class difference between the interviewer and the respondent.

The fact that a culture rarely can be completely defined or described by a single factor that a researcher can take into account makes the design of multicultural surveys a difficult and complex undertaking. Since researchers designing surveys for their own cultures and countries might, almost automatically, account for many cultural factors, particularly societal norms for behaviors and interactions, they sometimes unconsciously overlook them when designing a survey for another culture, transferring a survey between cultures, or designing a single survey meant to be adaptable to multiple cultures. For this reason, researchers need to apply a systematic approach to designing multicultural surveys.

With regard to sampling, we may begin by noting that, for many countries, the use of a random sample survey is not a feasible option. Technical difficulties aside, sampling may be complicated by issues of equity, ethics, and limited resources. For example, a survey may be used to evaluate a program for which the treatment areas have been “hand-picked” because of political and geographic preferences, or where staff and funds for probability sampling are scarce. Sampling frames in many countries often can prove difficult to locate, or are incomplete or out-of-date. Data specific to cultural subpopulations that could aid sample design frequently do not exist, are hard to find and obtain, or are of unknown accuracy.

Nonsampling error, which may also have serious effects on survey results, can easily go unnoticed altogether. Respondents sometimes answer questions even if the questions are not entirely clear or are unsuited to their lives in some way. Differences in socioeconomic status or doubts about confidentiality can affect responses without the slightest indication that error is being introduced into the survey. Both sampling and nonsampling error can have serious consequences for the ultimate use of data for policymaking or for reporting on conditions such as the impact of natural and manmade disasters.