Methods And Standards: A Challenge For Change

By Warren J. Mitofsky

Warren Mitofsky delivered the following presidential address at the forty-fourth annual conference of the American Association for Public Opinion Research (AAPOR). Mitofsky’s analysis of a situation that was current then seems prescient today. The divide between methods and disclosure described by him seventeen years ago continues to persist within the survey community. The conflicts he presented as quota versus probability sampling and telephone versus in-person interviewing now manifest themselves as telephone versus internet interviewing, with the question of how to generalize off an internet sample. And, of course, the problems of poorly designed polls and biased question wording and question order effects haven't gone away, nor have concerns about acknowledgement of limitation or issues surrounding polling in nondemocratic countries. Presented here in its entirety, Mitofsky's speech is still just as important in January 2007 as when it appeared in the journal Public Opinion Quarterly in Autumn 1989.

Ever since I joined AAPOR, some 25 years ago, I have been struck by the tension that seems to exist between a relatively small group of survey practitioners on one side and more formally trained methodologists on the other. But it is easier to describe the conflict than the participants.

It is a conflict between the old and the new. It concerns methods and practices that were popular during another era, but they are not for today. It permeates conference sessions at AAPOR. It affects the things we say to each other about our colleagues. It even has found its way into other AAPOR presidential addresses.

For a while I thought the resistance to progress came mostly from older practitioners and those in the commercial sector of our organization. But it does not. Some are in institutions of higher learning. Many have been in the field since the early days of AAPOR. But there are younger people in their ranks also.

All of the people on both sides of the conflict have a deep commitment to survey research. And all share a love for AAPOR. Yet they take their positions at the barricades, like combatants at the Alamo, ready to capture or defend it, every time standards and practices are discussed.

I don’t want to characterize this tension as a dominant theme of our meetings. That would be far from an accurate picture. The tension is more of a persistent undercurrent that keeps surfacing. I don’t know if this same tension exists in other social science associations. But I do know we experience it at AAPOR.

For my own part, few who know me can have any doubt on which side of this controversy I stand. To me, the issue is whether we benefit from methodological advances, or whether we persist in using tools that were state-of-the-art too many years ago.

Would our public poll mentors—men such as George Gallup, Elmo Roper, Archibald Crossley—if they were alive and working today, still be doing what they did in days gone by, or would they adapt to the current methodological age? Of course these men of vision would be leading the way. I cannot imagine them stuck in the past. It is these men who formed the National Council on Public Polls. They did it partly because of the conflict within AAPOR over formulating standards and partly to head off government regulation of polling. This new organization was supposed to be an instrument for self-policing of the polls, and to educate the news media and the public about polls.

And yet AAPOR today still has no unambiguous mechanism in its Code of Professional Ethics and Practices for condemning polls that are outright garbage, let alone so-called legitimate work that cuts corners and lacks scientific rigor.

In the sample design area I am referring specifically to surveys with known biases in sample selection and estimation, surveys that still rely on quota methods at some stage of the design rather than probabilities at all stages, or surveys that ignore unequal selection probabilities when it comes to estimating. We should also include surveys based on biased samples selected only from listed phone numbers.

Usually there is an attempt at legitimizing quota sampling. It is described in reports as a “modified probability” design. (I must tell you, every time I hear that phrase “modified probability,” I am reminded of the expression “a little pregnant.”)

And taking account of unequal sample selection probabilities, such as those we get when we select one person per household in our sample, requires weighting if we are to avoid a bias. But in some eyes weighting, I am told, is “cooking the data.” Any discussion of reducing the sampling error by use of ratio or regression estimates I suspect would move us, in the eyes of those who object, from a venial sin and punishment in purgatory to a mortal sin and condemnation to a survey hell.

There are techniques used for asking questions that introduce response bias. This seldom gets the same attention as sample design issues. I am thinking of survey questions that purport to set the agenda for the respondents’ frame of reference by providing background information on a complex or unfamiliar subject. Or questions that are deliberately given a context by placing them after other subject matter in the survey. An example is the intentional placement of presidential preference questions near the end of a questionnaire. Presumably, the issue questions that precede it are a mini-agenda for the campaign, which cannot help but bias the presidential choice.

Fifteen to 20 years ago, the more common method of sampling and interviewing switched from in-home and face-to-face to telephone. When this happened there were speeches by elder statesmen to the press and to marketing directors saying that they could not rely on interviews conducted by telephone. They gave no credence to telephone interviewing’s advantages of direct supervision of interviewers or the ability to select unbiased probability samples of the telephone household population.

The methodological issues we dispute today are not new. They were with us in 1946 at Central City and at the first AAPOR meeting at Williamstown a year later.

At Central City there was a session devoted to “probability” versus “quota” sampling. Morris Hansen of the Census Bureau was a lonely advocate of probability sampling. Representing nonprobability sampling, or quota sampling, were Norman Meier of the University of Iowa and a consultant to the Iowa Poll, Lucien Warner of Life magazine research, and Elmo Wilson, director of research for the Columbia Broadcasting System. The argument was mainly over the greater cost of the new method. At the conclusion of the discussion there was no consensus. There was no meeting of the minds among the participants.

Another panel discussion at Central City included some of the most distinguished researchers of the day: George Gallup, Julian Woodward, Clyde Hart, and Harry Field, discussing “Technical and Ethical Standards in Public Opinion Research.” Woodward argued that the people who do polls “must conduct themselves in such a way as to justify the responsibilities which will increasingly be theirs and to deserve the respect with which the public will regard them.” He was talking mostly about public polls, but others at the meeting quickly added market research to the discussion. Woodward saw polls as a “public utility.” He was concerned that cost alone should not deter improvement in methods. He wanted “a committee on standards that would ensure that no poll that did not live up to the standards agreed upon should have membership in the association. Such standards would include adequate sampling methods, competent interviewing staff, well-designed questionnaires, and the observance of certain ethical as well as technical standards.”

This set the stage for what Paul Sheatsley describes as the “ensuing tensions between [the] academic and commercial constituents.” He describes the position of the “protagonists” in the only debate that took place during the adoption of AAPOR’s constitution. It was over standards: “The commercial researchers… were on guard against any attempts by academics or others to restrict their research freedom.”

From the history, it is clear that the debate over the scientific integrity of methods used in the conduct of surveys is at least as old as this organization. Whatever mechanism for improving standards that was envisioned but not enacted as Williamstown still remains nonexistent. The conflict is unresolved today. Thus, the tension remains.