Starting from Scratch: Making Research a Reality in Afghanistan by Matthew Warshaw

Test 2 looks for patterns of consistent responses across respondents within an interviewer's pool of respondents. While it is possible that an interviewer with fifteen to twenty respondents might legitimately have all of them answering identically to a number of questions, it suggests there is a problem. Of course, a poorly designed question or a highly skewed public agenda may produce a respondent pool with zero variance for one or more questions. We take that into account. However, an interviewer who has much less variance in responses across his or her interview pool compared to peers will be singled out for attention.

While this particular problem has not been a major issue in Afghanistan, the temptation is sometimes strong for an interviewer who has traveled several hours to a rural location and failed to complete many interviews to cheat and complete them on his or her own. Consistent back-checking of work and additional incentives to interviewers to reach the truly hard-to-find locations help to avoid this problem.

Test 3, for patterns of nonresponse, seeks to identify interviewers who have substantially higher average rates of nonresponse across their interviewer pools compared to their peers. A high rate across the board may indicate a poorly designed questionnaire (too sensitive, too difficult) or poor timing-perhaps the survey is being conducted at a time when people are distracted (such as an important holiday like Ramadan) or when certain questions are too sensitive (such as in the run-up to an election). However, a statistically significant higher rate of nonresponse compared to an interviewer's peers merits management attention.

Finally, test 4 for interviewer productivity is both a test and a diagnostic. While field managers want productive interviewers, we tend to question those who are too productive. We look at two indicators: average length of interview and number of completed interviews per interviewing day.

Interviewers whose interviews are much shorter than the average or who complete many more interviews per day (generally measured in terms of one or two standard deviations above the norm) are singled out for management attention. While one could be concerned about interviewers who work too slowly, because we pay by the completed interview we focus on ones that are too short, or too many to be completed in one day. These productivity measures may also explain the other problems, in whole or in part. Interviewers who are moving too fast through the questionnaire may stimulate patterned responding, or they may start skipping responses and coding the results themselves.

Interviewing is difficult work in Afghanistan. The interviewers face skepticism and refusals to participate, and in some cases danger in the field. Wandering about Afghan neighborhoods could attract unwanted attention from residents or even local officials who are suspicious of outsiders. The desire and rationale to move quickly in the field is understood, but we monitor the interviewers to make sure efforts to finish fieldwork quickly do not adversely affect the results of the survey.

Interviewers who meet conditions that fall within the above descriptions are issued either a "Yellow" or a "Red Card" depending on the frequency or seriousness of the problems uncovered in our statistical review. Interviewers with yellow or red cards are singled out for additional training and more extensive supervision on their next survey. Depending upon the reason for them, an interviewer may be excluded from D3 surveys after receiving one or two red cards. If we start seeing too many interviewers with yellow or red cards, we begin looking for common explanations: working in the same region, ethnicity, or perhaps a common supervisor. Sometimes it is necessary to retrain or replace supervisors whose interviewers are consistently flagged with yellow or red cards.

ACSOR is overcoming the many obstacles it faces in Afghanistan. The quality of the fieldwork grows with each survey and the quality of the data has been excellent. As an example, D3 Systems conducted a survey from September 8-18, 2004, of 2,156 Afghans ages eighteen and over that included questions about the rights of women. Respondents were asked about women voting, pursuing university degrees, working outside the home, and holding political offices, as well as Afghan women already having all the rights they need and the impact of Western culture on them. The survey yielded some expected results, such as concerns that Western culture has a negative influence on Afghan women, but it also challenged some of the conventional wisdom. For example, it was revealed that both men and women in Afghanistan are open to the concepts of women voting, holding political office, and working outside the home, despite the contrary having been the imposed law during the many years of Taliban rule.

Market research is possible in Afghanistan. It requires patience, making incremental improvements in the fieldwork, and a willingness to participate in the development of capacity. While D3 Systems is moving to CATI surveys in other Muslim countries, such as Saudi Arabia, where telephone lines and economic conditions make it possible, we will continue to rely on paper, pencils, and the tenacity of our local field teams to get the job done in Afghanistan.