Question Format, Response Effort, and Response Quality: A Methodological Comparison of Agree/Disagree and Item-Specific Questions

November 13, 2017, 1pm

GESIS, Mannheim, B2,8

Jan Karem Höhne

Abstract

In empirical social research, using questions with an agreement scale, also known as agree/disagree (A/D) questions, is a very popular technique for measuring attitudes and opinions. Major national and international surveys, such as the American National Election Study (ANES), the Eurobarometer, and the International Social Survey Program (ISSP) make use of the A/D question format. Methodological considerations, however, suggest that such questions require effortful cognitive processing and are prone to response bias. Therefore, many survey scientists recommend the use of item-specific (IS) questions, which usually consist of an interrogative request for an answer and tailored response categories so that they seem to imply less response effort. In this talk, I propose an explanatory approach to explain differences between A/D and IS questions in terms of response effort and response quality. Furthermore, I provide supporting evidence for this approach from experimental studies employing different methods, such as eye tracking. The overall results indicate that IS questions are associated with higher response effort but are more robust against response bias, such as scale direction effects, and, thus, produce better response quality. All in all, the findings suggest substantial differences between the presumed cognitive complexity of question formats and the response effort expended in responding.

About the speaker

Jan Karem Höhne is doctoral candidate and research associate at the Center of Methods in Social Sciences at the University of Göttingen (Germany). In his PhD thesis he investigates question formats in terms of response effort and response quality. He is expected to receive his PhD in the end of 2017. He is also research fellow of the American-German Fulbright Commission and conducted a research stay at Stanford University (USA) in 2016/17. His main research interests focus on survey methodology, web-survey design, paradata, and eye tracking.