Complexity, Learning Effects and Plausibility of Vignettes in the Factorial Survey Design

Abstract

The factorial survey is a method of data collection that combines the advantages of survey research and the advantages of experimental designs. Respondents react to hypothetical descriptions of objects or situations (vignettes) instead of answering single-item questions. By varying each dimension of the vignettes in an experimental design, the dimensions’ impact on respondents’ judgments or decisions can be estimated accurately. Thus, the method is able to identify the effect of single factors which are often confounded in reality. So far, only few methodological studies address questions of measurement validity when a factorial survey design is used. The article provides a brief overview of the use of the factorial design in the social sciences and points our still unresolved methodological questions. Using experimental data specifically designed for this purpose our analyses consider the stability of respondents’ judgments with respect to the number of dimensions presented in the vignettes, possible learning effects and ‚implausible’ or ‚illogical’ cases (vignettes describing objects or situations which are rare or even impossible). We test several hypotheses regarding the complexity of vignettes and the consistency of judgments. According to our results, a high complexity of vignettes and implausible cases cause respondents to consider fewer dimensions in their judgments; we find smaller influences of vignette variables while the consistency of the judgments remains the same. Finally, we discuss the practical consequences of these results.