During the years I worked at the Roper Center for Public Opinion Research, I spent many happy hours immersed in iPOLL, the Center’s amazing archival database of thousands of polls going back to the 1930s. My job often required delving into historical data, and every once in awhile I was struck by the thought: They sure don’t write survey questions like they used to.
This is not necessarily a bad thing. Frankly, some of those old questions were really awful. But some were fascinating, too—and not just for what they revealed about public opinion. The curiosity of the researchers themselves shone through the questions they asked, along with their interests, their enthusiasm, their prejudices—in short, their worldviews.
The notion of a worldview that shapes and biases all our perceptions, whether we are conscious of it or not, presents a problem that haunts every field of social inquiry. It is a basic tenet of historiography that historical study often tells us more about the scholars engaged in it, and the times in which they lived, than it does about the subjects they are studying. Some accounts of the Roman Empire written during the 1950s, for instance, advanced views about Roman government that bore clearly the mark of the Cold War on the writers’ thinking.
Closer to home was a poll conducted in May 1943, by the firm of Elmo Roper for Fortune magazine, of 2,787 women ages twenty to thirty-four. One day, while perusing the documentation for the dataset in the Roper Center's archives, I stumbled upon this directive from the researchers to their interviewers:
Some of the male members involved in the survey think it will be very interesting to classify answers to some of the questions depending on whether the respondent is attractive, unattractive, or just middling . . . Under “good” record the attractive women, under “fair” record those who are so-so and under “poor” record those unattractive ones.
Intrigued by the fact that the interviewers had actually followed instructions and recorded data for this variable, I ran a crosstab between it and the rest of the questions in the survey. The results seemed to show that for women at that time, appearance was a robust determinant of attitudes across a broad range of social indicators.
Of course, it was not that simple. Further investigation revealed a powerful relationship between the interviewers’ rating of appearance and the respondents’ economic class, which in turn was related to the women’s social attitudes. In an article that was eventually published in the Chicago Tribune, I speculated that perhaps hardship took a toll on some poor respondents’ looks, or perhaps the interviewers were predisposed to see beauty in affluent women while overlooking it in poor ones, or perhaps both. In any case, it seemed to me that the classification of these women according to their looks told us far more about the mindsets of the pollsters than it did about the opinions of the polled.
In 2001, in his presidential address to the American Association of Public Opinion Research (AAPOR), Murray Edelman said:
If our surveys give voice to the people, the answers to survey questions are the language through which that voice speaks, and this language is conditioned by our worldview as the researchers . . . In deciding what questions to ask and how to ask them, we researchers have to be selective. This is in the nature of conducting a survey. But let us also recognize the power we have in defining political and personal realities and be more aware of the limitations of our worldviews.
Sometimes I wonder what, decades from now, the surveys of today will have to say about the people who conducted them. In a featured op-ed piece in this month’s issue of Public Opinion Pros, Steve Farkas talks about what he sees as serious problems pervading the field of public opinion research, mostly having to do with the surveys themselves—the questions that are asked, the findings that are reported, the reasons for doing them—and he cites, among other things, the pressures often imposed on pollsters by agendas, pecuniary or political, that might not permit much latitude for looking for the truth, or making discoveries about society that are truly meaningful.
I think the future will judge survey researchers today not just by the limitations of our worldviews, but by the extent to which we allow themselves to be hamstrung by the reasons for doing the research in the first place. Some practical realities cannot be ignored—like who is paying for the survey, or what is likely to boost ratings for the six o’clock news—but within those boundaries there should still be some room for a spirit of inquiry to flourish, if only we can muster the desire and the initiative to encourage it.
We should take every opportunity to do so. Indeed, as researchers, we have an obligation to do so. Like that crew of sexist guys in 1943 whose activities now seem more amusing than malign, history may forgive us for thinking within the confines of our worldviews. It may look less kindly, however, on diffidence, complacence, or sheer lack of imagination if it finds itself frustrated with what we could have learned but didn’t, and the opportunity forever lost.
—Lisa Ferraro Parmelee, Editor |
