Publication Type: Paper/Poster Proposal
Abstract: Placing related questions together can alter the associations between answers to them. We conducted two experiments that varied whether related questions were presented together on a single screen in a web survey. The first experiment replicated that inter-item associations and scale reliability were highest when the questions were presented together. However, a structural equation model revealed that these higher associations reflected correlated measurement error and decreased rather than increased construct validity. We carried out a second experiment to test three possible mechanisms for the heightened correlations, but reduced validity. First, the questions may be perceived as being multiple measures of the same construct, inducing more similar interpretations of the items. Secondly, when no actions are needed to get to the next question, the same material may be retrieved from working memory in answering all the questions. Thirdly, respondents may be minimizing effort by clicking response options in the same columns and paying less attention to the individual questions when they are presented in a grid. Our second experiment used a factorial design in a web survey with 2,694 respondents. Respondents answered 4 questions on diet and 4 on exercise, where the layout (together in a grid, together on a screen but listed separately, in separate screens), the accompanying instructions (related, independent, no instructions), and the order of the questions (by topic, intermixed) were varied randomly. We also expected these manipulations to interact with the location of the experiment in the questionnaire and randomly assigned its placement. Respondents’ Body Mass Index was calculated in order to estimate and compare measurement error properties and validity of the diet and exercise constructs. The findings will allow us to understand the mechanisms generating differences in responses to questions on the same topic and guide survey design decisions that affect measurement error, nonresponse, and cost.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: Intuitively, an attractive questionnaire cover design should enhance the response to a mail survey, but attempts to develop a theory, or explanation, of what constitutes an effective cover design are based on the assumption that such a goal is ultimately attainable. A different approach is to accept that this may not be possible. In other words, though we may know a ‘good’ questionnaire cover design when we see one, we may not be able to explain how to achieve this.


Thus, rather than attempt to solve the problem of how to design an effective questionnaire cover, we can draw instead on the experience of advertising, which has a similar problem, but also has a solution – likeability. Regardless of how advertisements are created, more likeable advertisements are generally more effective than less likeable ones. This paper reports research designed to test the same proposition applied to questionnaire cover designs.


In seven mail surveys involving nine comparisons the ‘likeability’ of different questionnaire cover designs was measured and response rates for the different cover designs compared. Overall, the suggestion that likeability might predict the effectiveness of questionnaire cover design in a mail survey was weakly supported by the seven studies described. However, for five of the six comparisons that involved only graphic designs, the most likeable cover produced a higher response rate, with an average increase of around 2 percent-.


The conclusion is that mail survey practitioners should consider using a ‘likeable’ questionnaire cover design, but avoid cover designs involving photographs, and that likeability in questionnaire cover design can be measured by a simple seven-point semantic differential question; "How much you like this cover?’ This question gives good discrimination between cover designs and produces relative likeability scores that have face validity.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: Utilization of a single, web based instrument for administration in multiple modes (both self-administered and interviewer-administered) presents many challenges for both design and implementation. This paper describes those design challenges, and focuses on mode differences between self-administered and interviewer-administered responses.

The data for our analysis comes from a field test of a follow-up interview of college graduates ten years after graduation (Baccalaureate and Beyond Longitudinal Study conducted for the National Center for Education Statistics, US Department of Education). Our analysis looks at mode differences in terms of demographics and item-level responses including use of don’t know and refuse options.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: One of the factors to be considered in questionnaire design is format. This factor is particularly important in self-administered interviews. Two types of self-administered questionnaires are used in Canada’s Census of Population: a short form (80 percent of the population) and a long form (20 percent of the population). In the 2006 Census, optical scanning and character recognition will be used to capture data. However, such techniques cannot be used on the current long form, mainly because the response spaces for open questions cannot be readily scanned. A number of qualitative surveys have been conducted in the last few years with the aim of improving the long form so that the data can be scanned in. Another aim was to find out from small samples of respondents what would be the best questionnaire format. Those surveys were followed in April 2002 by a quantitative survey whose aim was to recommend the best questionnaire format based on certain criteria. The format had to be such that the data could be scanned and would be of acceptable quality. The survey involved testing four different questionnaire formats on four independent samples of households. The survey included matrix formats and sequential formats. Following the survey, a detailed analysis was carried out to determine which format was best on the basis of criteria such as data quality, collection costs and response burden. The presentation will focus on the criteria used to determine the best format and the key findings of the analysis.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: The basic parameters of cognitive interviewing for improving survey questionnaires (e.g., prompting participants to think out loud, probing about interpretations of questions, and so on) have been well established for a number of years. However, these basic parameters are general enough to allow considerable latitude in practices—thus, activities that share the label of “cognitive interviewing” could in reality be quite different.

Current cognitive interviewing practices reflect two distinctive paradigms, with different conceptualizations of the role of the interviewer. The first is closely tied to psychological methods of protocol analysis and considers the interviewer to be primarily a data collector. Such interviewers are expected to intervene in the data collection process as little as possible, thereby allowing systematic analysis across interviews. The second is a more active form of in-depth interviewing in which the interviewer is charged with exploring meanings of responses in a less structured manner. Such interviewers would make active decisions about issues to explore based on particulars emerging from each interview.

As background, this paper will show how these somewhat divergent practices came to be incorporated under the same label. Its main emphasis will be on exploring the practical implications of these paradigms in terms of what interviewers should actually do in cognitive interviewers, what sort of background would be ideal for them to meet their objectives, and how interviewer behavior shapes the evidence that is used to make questionnaire design decisions. It will also explore some of the tradeoffs involved in scripted vs. unscripted probing, propose guidelines for choosing probes, and criteria for determining whether or not probes are working effectively. The theoretical arguments raised in this paper will be complemented by examples from cognitive interview studies conducted at NCHS.