Every once in awhile a controversy erupts on AAPORnet—a listserv for members of the American Association for Public Opinion Research—that really strikes a chord (or hits a nerve) with a lot of people in the polling community, and for days to follow, the list is unusually animated by the spirited volleys of conflicting pollster opinion. Such was the case recently with the subject of online polling, as participants debated the value (or lack thereof) of this mode of conducting surveys. Indeed, I haven’t seen such excitement in a long time—there were posts and ripostes, a few tempers flaring, and a number of arguments that got downright esoteric, punctuated by the occasional entreaty, “Can someone please take me off this list?”
The timing of this discussion was uncomfortably fortuitous, coming right as Public Opinion Pros was preparing the second of two features we have presented in the past five months involving a process called “deliberative discussion.” If you use POP’s access to Public Perspective Online and search on the word “deliberative,” you will retrieve a flurry of articles from 1996 on what was then known as the “deliberative poll.” This was an experiment in what originator James Fishkin described as “a means of giving voice to public views that represent all the people under conditions where they can also think about the issues in question.”
I think it was the words “represent” and “all the people” that really got folks’ blood up at the time—not to mention the word “poll”—much as they seem to be at the crux of this month’s online polling donnybrook. The studies we have had in POP on deliberative discussion claim to be attempting no such thing. Still, the subject remains a touchy one, if not nearly as sore as that of internet surveys.
Having published articles on both, I suppose I should be taking some sort of position on these and other “nontraditional” survey or survey-related methods, such as cognitive interviewing; but I feel rather unequal to that task, not being a methodologist, nor even a statistician. My attitude is something like, “Here are these studies; serious researchers are devoting a lot of time and energy to them; what can we learn from them?”
Of course, I proceed from a different point of departure than most of the AAPOR members who weighed in on the online polling furor. By both training and predisposition, I am a humanist, not a scientist. When you’re researching, say, the effects of French political propaganda on absolutist theory in Elizabethan England and everybody directly involved has been dead for several hundred years, you don’t have much control over the evidence—however much of it you are lucky enough to find. In a field where the entire extant literature might consist of a paltry few thousands of documents spread out over a period of many years, you have to squeeze as much insight as you possibly can out of whatever has happened to survive, using every possible tool of reasoning at your disposal. In these circumstances, the rules of evidence become a bit spongy.
And that is okay, I think, as long as we don’t represent our findings as something they are not, or use them for purposes to which they are not suited. Readers of my book (all three and a half dozen of them) understand—or they should understand—that I cannot travel back in time to gather evidence about sixteenth-century European politics, and so they need to weigh my conclusions against the extent of my inability to find any kind of truth. Further, they need to place them in the context of “truths” that other researchers have done their best to formulate on the same subject.
Can we conduct a meaningful survey using cognitive interviewing? I don’t know; but I do know that when I have read transcripts of some of these interviews and learned a little bit about how minds other than my own work, they have suggested all kinds of possibilities for survey questions and ways of asking them that otherwise might not have occurred to me. Are the survey results coming from very small samples of students at Rider University and the American University in Cairo representative of the views of any population? I know they are not; but I also know that when I look at those numbers in the light of what the students had to say in the course of their “deliberative discussions” on terrorism, I find in them both fascinating and thought-provoking clues as to differences of opinion between people from two deeply divided parts of the world, on an issue critical to us all. There is very definitely value in this.
Where we draw the line between findings that have value and those that don’t, and how much bias we are willing to allow the limitations of our evidence and our methods to impose on us, is a larger question. Warren Mitofsky hit it on the head, I think, in the course of the AAPORnet discussion, when he said, “We can have a debate some other time about whether some information, no matter how biased the source, is better than no information.”
I hope that other time comes eventually, and that survey researchers address it with the same fervor they have directed these past few weeks toward the relative merits of online polling. In the meantime, we leave the table set for food for thought, and hope that readers will see this as an invitation to dinner.
—Lisa Ferraro Parmelee, Editor
|
