Measuring the Response Quality of Open-Ended Questions in a Demographic Web Survey Using Linguistic Complexity Features

Abstract

Open-ended questions (OEQs) are important survey tools for social scientists, but their response quality is often disputed due to the additional load they impose on the respondent. To find ideally worded questions and survey strategies that encourage high-quality responses to OEQs, the quality of textual responses is often assessed. Response length and response latency are often used as measures of response quality, but they do not provide enough information on interpretability and richness of the responses. In this study, we propose a novel way of evaluating the data quality of open-ended responses by leveraging approaches from natural language processing, measuring different linguistic complexity features of responses. Using various automatically generated linguistic features, we compared the quality of responses to distinctly worded sets of questions related to people’s uncertainty about their intention of having children. Overall, we found that the different wording of questions may affect responses on different aspects of linguistic complexity, which canonical indicators fail to reveal. In addition, we found that the variance in response quality could be attributed to both respondent characteristics and different versions of questions. These findings offer practical strategies for incorporating OEQs into a large-scale demographic survey, as well as providing a new perspective in evaluating responses to OEQs in future surveys.