Creating Design Weights for a Panel Survey With Multiple Refreshment Samples: A General Discussion With an Application to a Probability-Based Mixed-Mode Panel

Abstract

Panel surveys suffer from attrition, where participants drop out over time. To maintain generalizability, refreshment samples are frequently employed, bringing in new individuals, increasing the number of panelists, and balancing sample composition. Although refreshment samples offer numerous advantages, the inclusion of new panel members may introduce bias into the analysis if the design weights are not appropriately tailored to these new members and adjusted to align with existing panel members. If not correctly accounted for, their inclusion may bias results. This paper addresses the issue of designing proper weights by applying the multiple-frame weighting approach proposed by Kalton and Anderson, which is generally used for cross-sectional surveys, to ongoing panel studies with refreshment samples. We demonstrate its application to a synthetic data set and a probability-based mixed-mode panel with an initial sample and two refreshment samples. We compare estimates obtained using multiple-frame weighting with those obtained using unweighted and naively weighted methods (where design weights are used as calculated for the respective samples without adjusting for the fact that some members of the population have a chance of being sampled more than once due to the refreshments). These comparisons showcase the potential for bias introduced by neglecting proper weighting and underscore the importance of both a multiple-frame weighting approach and meticulous sample documentation.