It should, of course, be noted that identifying question wording switches, interview mode switches, and house switches was somewhat laborious, and in most cases entailed checking the actual reported poll questions, sources, and figures against published results, such as those in the Roper Center archives, the American National Election Study, Gallup poll, or other published sources. In only a few cases did authors provide the actual poll sources. The resulting sample of convention papers, journal articles, and books should therefore be treated as a convenience sample, with the observations cited here not definitive. They are, however, strongly suggestive of a need for secondary researchers to deal with these issues.
And how have they dealt with switches in question wording, interview mode, or polling houses, and the effects they might have on their analyses? The answer, unfortunately, is that they often haven't. As Figure 6 indicates, two-thirds of recent secondary research includes at least one of the three issues discussed. In nearly two-thirds of those cases, they were not even acknowledged in a way that readers could identify, whether adjacent to the table or figure, in the text, or at least in a reference or an appendix. In most such instances a reader could not tell a switch had even occurred.
Beyond simply acknowledging them, a further question is whether the research controlled for such effects statistically. Question wording switches, interview mode switches, and house effects are not random sampling effects, and are not controlled for by confidence levels , so raising the confidence level would do little to sort out their importance. Another approach would be to include dummy variables for any of these three switches in the equation. Even for studies using only two polls this is problematic, and, as noted earlier, most secondary research reports more than two polls, with an average of five used. This sample of fifty-four convention papers, journal articles, and books did not identify a single instance in which statistical controls for question wording switches, interview mode switches, or house effects were used.
That these factors can cause differences in poll results at or above statistical significance levels is, of course, not news. A vast body of literature on primary survey research advises the researcher how to deal with them. Yet published secondary survey research apparently largely ignores these concerns. As a result, it is quite possible that many, perhaps most, secondary survey research models and conclusions may be contaminated by these issues. Raising that threshold to the 99 percent confidence level, particularly for large samples, might reduce the importance of substantively unimportant predictors, but would seem to do little to unscramble the effects of assigning question wording effects, interview mode effects, or house effects.
The exact consequences of these issues are difficult to determine. With no way to control for the switches, researchers may miss true change because wording, interview mode, or house effects obscure it. In other cases, they may falsely observe change when none has occurred. Statistical explanatory models may be wrongly specified, perhaps most often because the effects of uncontrolled polling artifacts are assigned to other variables (such as time or events or demographics or other attitudes). Type I errors, Type II errors, or wrongly specified models may influence future research based on past conclusions. When applied to funding programs or future research in the social sciences, biomedicine, government, or business, such errors could be costly.
Doubtlessly these issues will continue to be significant, since secondary survey research is already common and is likely to be more so in the future, as the number and diversity of survey data collections and their availability increases and as survey methods continually change. To the extent these concerns are not addressed, the credibility of survey research is to some degree at risk. What, then, is to be done? |
