The vagaries of the vignette world revisited: Method effects in factorial surveys

July 07,13:45 h
Mannheim, Conference room B2,8

Frau Prof. Dr. Katrin Auspurg

Abstract:

The factorial survey is a method that combines the advantages of experiments (randomized treatments) and surveys (e.g. comparative ease in regard to including participants in different geographical locations or in general population surveys). Respondents are asked to evaluate hypothetical descriptions of situations or objects (vignettes) that are composed of various attributes (dimensions). In factorial surveys, the values (levels) of dimensions are experimentally varied, so that their impact on respondents’ judgments can be estimated. The method is more and more used in many research fields of the social sciences to study judgement principles, social norms and attitudes, definitions and behavioral intentions. However, as the method is more complex than item-based questions, it is possible that method effects occur that jeopardize the validity of results. The talk will give an overview on recent research on method effects that might occur, including effects of the experimental design (random or deliberate selections of vignettes to be used in a study), range-effects (span of values of a dimension), effects of presentation style (text or tabular), and answering-scale effects. We set up experiments in which we tested these method effects in a between-subjects design. The data consist of 2,500 students from 25 universities all over Germany. The findings suggest that factorial surveys are robust in regard to changes of the presentation style, but several recommendations can be provided that allow more valid and efficient estimates of the causal impact of experimental factors.

Presentation (622 kB)

About the speaker:

Katrin Auspurg holds a full professorship in sociology (specializing in quantitative empirical research) at the Department of Social Sciences at the Goethe-University Frankfurt/Main, Germany. Her research interests are in survey methodology (e.g. multifactorial survey experiments, nonresponse bias), labor market research (e.g. how inequalities on the labor market and in families intersect), and discrimination (e.g. ethnic discrimination on housing markets). She is a co-author of Factorial Survey Experiments (Sage Publications, Series Quantitative Applications in Social Sciences; coauthored by Thomas Hinz).