Towards Credible Vote Validation

Abstract

Vote validation (VV) is becoming standard practice in academic election surveys as a solution to frequent over-reporting of voter turnout by survey participants, and VV results are usually presumed accurate. Yet evidence for VV’s accuracy is limited. This article investigates these limits to help calibrate our confidence in the process by which VV data are generated and to improve that process. Concerns with VV are serious: voter files and survey data contain errors; state-of-the-art matching procedures make errors; resulting estimated match probabilities are misleading; part of the process (clerical review) has lacked standards; and evidence purported to show VV’s accuracy is consistent with overall accuracy similar to self-reports. Considering these concerns, I present a transparent and reliable clerical review procedure to evaluate pairs of matched records, I present an interpretive framework for evaluating accuracy, and I argue for skepticism about VV’s accuracy so that researchers make better-informed choices about when and how to use VV data. Self-reports have better sensitivity (counting nearly all voters) while validation has better specificity (excluding more non-voters), and the two measures are subject to different kinds of error, so rather than preferring one automatically, researchers should consider which is more accurate for a given research question and, where possible, compare results from both measures.