Improving Understanding of Survey Questions with Multimodal Clarification

Abstract

If survey respondents do not interpret a question as it was intended, they may, in effect, answer the wrong question, increasing the chances of inaccurate data. Researchers can bring respondents’ interpretations into alignment with what is intended by defining the terms that respondents are at risk of misunderstanding. This article explores strategies to increase alignment between researchers’ intentions and respondents’ answers by taking advantage of the unique affordances of online surveys compared to paper or other analog formats. Web surveys are often text-based, but allow for the seamless integration of embedded audio material so that users may read, listen to, or both read and listen to survey instructions. Unimodal definitions are either spoken or textual, while multimodal definitions are both spoken and textual. Further, definitions can be designed to take advantage of the affordances of each mode. While mode-invariant definitions contain the same words irrespective of whether they are textual or spoken, mode-optimized definitions are designed to take advantage of the affordances of written and spoken communication. For example, definitions optimized for textual presentation use fewer words than corresponding mode-invariant definitions and are designed so the key information is visually salient, while definitions optimized for spoken presentation are shorter and more colloquial than corresponding mode-invariant definitions. In this study, both mode-optimized and mode-invariant formats improved alignment. Multimodal, mode-optimized definitions produced improved alignment over both types of unimodal definitions. This study suggests that multimodal definitions, when thoughtfully designed, can improve data quality in online surveys without negatively impacting respondents.