Course 1 - Questionnaire Design

August 13-24, 2012

Instructors:   Marek Fuchs, Tanja Kunz, Matthias Emde

Course content

In traditional text books, questionnaire design is typically treated as an "art". However, the rules given in such books are either too specifically concerned with particular substantive questions (and generalisation of such rules to other questions is not trivial), or the advice given is too general and it is left to the reader to apply the rules to specific questionnaires. This course aims to avoid this dilemma by means of two complementary strategies: On the one hand, basic concepts relevant to survey measurement will be discussed in order to make participants aware of the mechanisms underlying survey responses. On the other hand, the course will discuss results of field experiments testing various aspects of survey questions in order to demonstrate the impact of design decisions on the answers provided by respondents.

The course starts with an introduction into the total survey error concept as a theoretical framework for the integrated assessment of survey quality. A special focus will be on measurement error and its components and particularly on the impact of the questionnaire on data quality. The course then provides a description of the question-answer process. Based on this concept we will discuss problems in response behaviour like satisficing and social desirability and how to address these problems during questionnaire design.

Using this theoretical framework, we will then assess various types of questions (e.g. closed-ended vs. open-ended questions; multiple response questions) and focus in greater detail on requirements for questions regarding factual information, attitudes and behaviours. Based on field experiments concerning question wording, question order, response order, and other aspects of survey questions, the consequences of design decisions for the answers provided by respondents will be discussed. In addition, advanced measurement tools like event history calendars, diaries and visual analogue scales will be covered.

Since survey research makes use of various survey modes (face-to-face, telephone, mail, and web), the implications of these survey modes for questionnaire design will be highlighted. Furthermore, the course will address the visual design language of questionnaires and we will demonstrate that visual aspects of questionnaires have an effect on survey responses similar to the wording of questions and response categories. The course concludes with an overview of pre-testing methods for survey questionnaires.

The course will be composed of lectures and group work. During group work, participants will work on a questionnaire that will be tested and fielded using an online-access panel in the final days of the course.

Course and learning objectives

The course provides scientific background knowledge for questionnaire design and educates participants in their professional reasoning when designing survey questions and a questionnaire as a whole. At the end of the course, participants should be able to anticipate potential data quality problems during the design stage of a questionnaire, and make use of the acquired knowledge in order to develop high-quality questionnaires.

Course prerequisites

Basic understanding of survey research (introductory course in social science research methods or similar). Note: Participants will need to bring a laptop computer in order to perform the practical exercises in this course.

Recommended reading

  1. Dillman, D. A., J. D. Smyth & L. M. Christian (2008). Internet, mail, and mixed-mode surveys: The tailored design method. New York.
  2. Fuchs, M. (2010). Improving research Governance through use of the total survey error framework. In German Data Forum (Ed.). Building on progress. Expanding the research infrastructure for the social, economic, and behavioral sciences. Vol. 1, Pp. 471-486). Leverkusen Opladen.
  3. Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer & R. Tourangeau (2010). Survey Methodology. 2nd edition. Hoboken, N.J.
  4. Presser, S., J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin et al. (2004). Methods for testing and evaluating survey questionnaires. Hoboken, N.J.
  5. Sudman, S., N. Bradburn & N. Schwarz (1996). Thinking about answers. The application of cognitive Processes to survey methodology. San Francisco.
  6. Tourangeau, R., L. Rips & K. Rasinski (2010). The Psychology of Survey Response. Cambridge.

About the instructor

Prof. Dr. Marek Fuchs is full professor for social science research methods at Darmstadt University of Technology, Germany. He obtained his PhD from Kassel University in 1993 and conducted post-doctoral work at the University of Michigan, Ann Arbor (USA). Since then, he has been the principal investigator of several large scales surveys. His methodological research is particularly concerned with methodological aspects of survey measurement. Over the course of the past 20 years, he has published on laboratory and field-experimental studies concerning questionnaire design for face-to-face surveys, telephone surveys and self-administered surveys (paper & pencil as well as web surveys). He has a long standing experience in teaching courses on survey methodology at the PhD and Master levels to an international audience.