Online vs. In-Person By Tom W. Smith and J. Michael Dennis

Social scientists and others have become increasingly interested in using email and the internet to conduct surveys. E-surveys are attractive because of their low costs, the fact that they are self-administered, and their ability to incorporate visual images. But major factors impede their general and widespread use, including limited population coverage, differential noncoverage, the difficulty of conducting random or probability sampling, and low response rates.

Moreover, it is unclear how comparable e-surveys can be to those utilizing more traditional survey modes. Even when issues of coverage and probability sampling can be dealt with, e-surveys still differ from the other major modes in various ways. Compared to telephone and in-person surveys, they are visual rather than oral, read rather than listened to, and use self-completion rather than interviewers. Compared to traditional, mailed, self-completion surveys, they use computers rather than printed questionnaires, have very different visual appearances, and can be interactive rather than static. By depending on some familiarity and comfort with computers, e-surveys present different cognitive demands than traditional modes. The social demands and task orientation probably differ on them as well, in part because they use self-completion, and for such other reasons as different social norms and expectations associated with the internet and/or computers.

Little empirical evidence has been collected on how the measurement features and error structures of e-surveys compare to those of other modes. To begin to fill this gap, we designed a series of mode experiments to examine the similarities and differences between two 2002 surveys: a web-based study by Knowledge Networks (KN) and the in-person General Social Survey (GSS). Our aim was to replicate and extend experiments done previously on the 2000 GSS and a 2000 KN study.

In brief, the GSS and KNS are both probability samples of adults living in households in the United States who, with a few exceptions, were asked identically worded questions in the same order, having to do with respondents’ preferences on government spending. This allowed us to compare the two in a number of respects that would demonstrate the effects of presentation, mode, and format of the questions on survey results.

Our first objective was to compare Don’t Know/No Opinion responses to the two surveys. Previous research on item nonresponse has shown that DK levels differ greatly by the presentation, mode, and format of questions. For example, Don’t Knows will be lower

Mode, in particular, can make a major difference, in that oral presentations can accommodate DKs, even if they are not explicitly mentioned, as unread but pre-coded responses. Self-completion modes, on the other hand, need to offer DKs explicitly along with other response options to capture them clearly.

With e-surveys, research indicates Don’t Know levels are lower when prompts are used and when no decline-to-answer or skip-out option is offered. They also go down when radio buttons are used to capture substantive and DK responses rather than using open boxes for both, and when a visual line does not separate the DK response from the substantive options; but not necessarily when the visual prominence of DK responses is varied in other ways. Further, DKs are higher on e-surveys if questions are cognitively more demanding (for instance, because of respondents’ limited reading ability or the lack of an interviewer to explain items or offer reassurance).

Online vs. In-Person: Experiments with Mode, Format, and Question Wordings