Tough Calls: Potential Nonresponse Bias from Hard-to-Reach Respondents
By
Peyton M. Craighill and Michael Dimock
Are respondents in telephone surveys who are more difficult to reach different from those more easily reached?
A common concern in the survey research industry is that efforts to sample the population it attempts to study may be unreliable due to error in nonresponse. If hard-to-reach respondents are substantially different than those who are more willing and easily reachable, then surveys with more limited calling designs may be less fully representative.
In 1997 and 2003, the Pew Research Center conducted two major methodological experiments comparing a standard five-night survey with an extended "rigorous" study in which every effort was made (including mailings and incentives) to maximize response rates over a three-month period. In both experiments, the increased effort involved in the rigorous design yielded few significant differences in data quality from standard surveys.
Because these analyses were limited to two special studies, it was hard to say what would happen within the range of a normal survey. To further this research, in 2004 the Pew Research Center compiled data throughout the year on how many call attempts were necessary to complete each interview in our national surveys. By combining data from nineteen surveys, we increased our sample size, allowing us to pinpoint small but relevant differences between easy- and hard-to-reach respondents.
The basic distribution of call attempts indicated that almost six in ten completed interviews (59 percent) were achieved within the first two attempts (see box). These were the easy-to-reach respondents. Another fifth of the interviews were completed on the third (13 percent) and fourth (9 percent) attempts. Only 19 percent of the entire sample required five or more call attempts--the hard-to-reach respondents.
| |
Call Attempt Basics |
|
| |
Completes |
|
| Attempts |
% |
N |
| 1 |
37 |
11,723 |
| 2 |
22 |
6,886 |
| 3 |
13 |
4,246 |
| 4 |
9 |
2,804 |
| 5 |
6 |
1,814 |
| 6 |
4 |
1,309 |
| 7 |
3 |
833 |
| 8 |
2 |
537 |
| 9 |
1 |
408 |
| 10 or more |
3 |
1,063 |
| |
100 |
31,623 |
The easier to reach respondents primarily consisted of older Americans, whites, people not employed full-time, and those living in the Midwest and rural areas. Among the harder to reach respondents were younger people, minorities, parents with children in their homes, and those living in the West.
Figure 1 provides a relative ranking of mean calling efforts needed to reach various groups.
Demographic, rather than behavioral or attitudinal variables, generally fell further from the mean, which was about three calls. Lifestyle indicators (such as online use and employment) and lifecycle indicators (age, parenthood) appeared to account for more differences in availability. Attitudes were correlated with these factors, but not perfectly so. Factors such as race, parental status, employment, age, and region of the country were more determinative of the effort to complete an interview than, for example, political party identification, ideology, or church attendance.
|
