Predicting panel attrition in longitudinal HRQoL surveys during the COVID-19 pandemic in the US

Abstract

Background

Online longitudinal surveys may be subject to potential biases due to sample attrition. This study was designed to identify potential predictors of attrition using a longitudinal panel survey collected during the COVID-19 pandemic.

Methods

Three waves of data were collected using Amazon Mechanical Turk (MTurk), an online crowd-sourced platform. For each wave, the study sample was collected by referencing a US national representative sample distribution of age, gender, and race, based on US census data. Variables included respondents’ demographics, medical history, socioeconomic status, COVID-19 experience, changes of health behavior, productivity, and health-related quality of life (HRQoL). Results were compared to pre-pandemic US norms. Measures that predicted attrition at different times of the pandemic were identified via logistic regression with stepwise selection.

Results

1467 of 2734 wave 1 respondents participated in wave 2 and, 964 of 2454 wave 2 respondents participated in wave 3. Younger age group, Hispanic origin (p ≤ 0.001) and higher self-rated survey difficulty (p ≤ 0.002) consistently predicted attrition in the following wave. COVID-19 experience, employment, productivity, and limited physical activities were commonly observed variables correlated with attrition with specific measures varying by time periods. From wave 1, mental health conditions, average daily hours worked (p = 0.004), and COVID-19 impact on work productivity (p < 0.001) were associated with a higher attrition rate at wave 2, additional to the aforementioned factors. From wave 2, support of social distancing (p = 0.032), being Republican (p < 0.001), and having just enough money to make ends meet (p = 0.003) were associated with predicted attrition at wave 3.

Conclusions

Attrition in this longitudinal panel survey was not random. Besides commonly identified demographic factors that contribute to panel attrition, COVID-19 presented novel opportunities to address sample biases by correlating attrition with additional behavioral and HRQoL factors in a constantly evolving environment. While age, ethnicity, and survey difficulty consistently predicted attrition, other factors, such as COVID-19 experience, changes of employment, productivity, physical health, mental health, and financial situation impacted panel attrition during the pandemic at various degrees.

Background

The coronavirus disease 2019 (COVID-19) has had significant impacts on various aspects of public health. In addition to the clinical consequences resulted from contracting the virus, uninfected individuals are also susceptible to non-clinical consequences resulted from the pandemic such as health care delays and reduced health-related quality of life (HRQoL) [1]. The pressing need for time-sensitive data collection has led to an explosion of research conducted via online platforms, which provide an effective and efficient strategy for researchers to collect data on COVID-19 within a short time period [2,3,4]. While this methodology may be effective for data collection during the pandemic, it is not without drawbacks [5].

Given the constantly evolving circumstances of the pandemic, longitudinal panels can capture the dynamics of outcomes of interest over time. At the same time, a longitudinal panel may be subject to various types of biases that limit generalizability of the results, especially when conducted online [6]. One such bias is non-response bias, i.e. respondents within the panel not completing follow-up surveys and/or dropping out of the study, which can occur for various, unknown reasons, ranging from loss of interest in the study to an inability to participate due to personal circumstances. Regardless of the reasons, whenever attrition is not random, panel data are subject to potential biases. Consequently, the findings may not be capturing the panel’s true longitudinal changes, and thus leads to threats to sample validity.

Our research team conducted a three-wave longitudinal panel survey online from April 1st 2020 to March 15th 2021 to monitor changes in HRQoL, among other characteristics, throughout the first year in pandemic [1]. This provided a unique opportunity to assess determinants of panel attrition in a longitudinal survey during the pandemic in the US. The objective of the study was to assess the association between an extensive range of survey respondents’ demographic, health behavioral, employment status, HRQoL measures and their participation status in the three waves of the data collection, with the hope of aiding researchers to better interpret future results generated from similar types of panel data, and providing insights on panel data attrition for studies that collect data online during crises such as the COVID-19 pandemic.

Methods

Survey overview

We collected a total of three waves of survey data from April 2020 to March 2021 to assess changes in HRQoL over time in the US. Wave 1 data were collected from April 1st to May 6th, 2020 (n = 2734). Wave 2 data was collected from July 4th to September 4th, 2020 (n = 2454). Wave 3 data was collected from January 10th to March 15th, 2021 (n = 2252). We used a panel data structure designed for data collection, i.e., for each wave, while sample attrition occurred, new and additional participants were recruited to ensure that we recruited a comparable sample size for each wave.

Participant recruitment

We used Amazon’s Mechanical Turk (MTurk) platform to field the survey. Amazon MTurk is an online crowd-sourced platform that allows large-scale surveys to be deployed [7]. Respondents aged 18 years or older and resided in the United States registered as “workers” in MTurk were eligible to participate in our survey. Because the platform is online, all tasks require an active internet connection. There were no other exclusion criteria. The sampling strategy was non-probabilistic as we did not restrict who completed the survey on the platform. Age, gender and race were stratified to be similar to the general US population. Participants were compensated €1.50 (approximately $2 USD) for their time to complete each survey. Informed consent was obtained at the beginning of the survey.

Measures

We collected information on various sample characteristics including respondents’ demographics, COVID-19 status, HRQoL, health behavior, employment status, and productivity changes related to COVID-19. HRQoL was measured using the EuroQol EQ-5D-5L, the Veterans Rand 12-Item Health Survey (VR-12), Patient Health Questionnaire-4 (PHQ-4), the 2-item Connor-Davidson Resilience Scale (CD-RISC 2), Primary Care PTSD Screen for DSM-5 (PC-PTSD-5), as well as questions selected from the Patient-Reported Outcomes Measurement Information System (PROMIS), and Behavioral Risk Factor Surveillance System (BRFSS) questionnaire. Additional measures were calculated using information collected from the survey (Additional file 1: Appendix 1).

The primary analysis focused on identifying predictors of attrition at waves 2 and 3, referencing the sample characteristics in the previous wave. We also assessed any changes in identified predictors of attrition between different periods of the pandemic.

Statistical analysis

All study variables were pre-processed by removing outliers and imputing missing values. Specifically, outliers were removed for variables that resulted from open-ended questions such as “How many more/less hours do you sleep than before COVID-19?” or questions that were autogenerated by the survey platform to indicate the time used to complete survey. In addition, unrealistic values (e.g., weight of 5 pounds) were also removed. Missing values were imputed based on appropriate distributions with parameters calculated using sample mean and standard deviation. Differences before and after imputation were checked using Kolmogorov-Smirnoff test (Additional file 1: Appendix 1).

Descriptive analyses were performed on all variables of interest from wave 1 and wave 2. Because only participation status from wave 3 was used in the analysis, sample characteristics of wave 3 were not analyzed. We compared the sample characteristics at wave 1 and wave 2 to US population norms [8,9,10,11,12,13]. We also compared sample characteristics at wave 1 by wave 2 attrition and sample characteristics at wave 2 by wave 3 attrition, using statistical tests appropriate for the distribution of the measure (e.g., t-test, chi-square test). We then employed standard logistic regression with stepwise selection to identify the most parsimonious model that predicted the attrition at wave 2 and wave 3 using characteristics in the previous wave, respectively. Age, gender, race, and ethnicity were fixed in the regression model based on background knowledge. The a priori significance level for variable entry and removal by the stepwise selection was 5%. The stepwise procedure combines forward selection and backward elimination to produce a list of plausible explanatory variables [14]. While stepwise regression has been critiqued with overfitting the model by including nuisance variables, this issue was of a lesser concern since we were primarily interested in themes represented by the selected variables that were associated with attrition. All analyses were conducted using SAS software version 9.4 (SAS Institute, Cary, NC).

Results

Sample characteristics

Wave 1 contained 2734 respondents, amongst whom 1467 (53.7%) participated in wave 2. Wave 2 included a total of 2454 respondents, and 964 (39.3%) of them participated in wave 3. In addition, 261 respondents participated in wave 1 and wave 3 but skipped wave 2. A total of 940 respondents participated in all 3 waves of the survey.

Respondents in Wave 1 ranged from age 18 to 82 years, with a mean (SD) age of 42.6 (± 14.3) years, 1365 (49.9%) were female, 1879 (68.7%) were white, 2446 (89.5%) were non-Hispanic, 1683 (61.6%) had a bachelor or higher degree, and 1276 (46.7%) were married. Wave 2 consisted of 2454 respondents ranging in age also from 18 to 82 years, with a mean (SD) age of 40.6 (± 13.3) years, 1073 (43.7%) were female, 1878 (76.5%) were white, 1987 (81.0%) were non-Hispanic, 1756 (71.5%) had a Bachelor or higher degree, and 1424 (58.0%) were married. Compared to the general US population, our wave 1 sample was slightly older, more likely to be single, and had higher education level. Less individuals identified as Hispanic or Black but more identified as multi-race. Income was more bell-shaped than the general US population. Wave 2 participants were younger, less likely to be female, and more likely to be married compared to the general US population; on the other hand, the race and ethnicity composition was comparable (Table 1).

Table 1 Sample demographic characteristics vs. US population

Before any adjustment for confounding, age and ethnicity in the wave 1 sample were significantly different by wave 2 attrition. Those who were younger, of Hispanic origin were more likely to drop out in wave 2. Age, race, ethnicity, education, marital status, region, income, and political affiliation in the wave 2 sample were all significantly different by wave 3 attrition. Wave 2 participants who were younger, black, Hispanic, married, college-educated, Republican, lived in the west, and had an income between $35,000–$74,999 were more likely to drop out at wave 3 (Table 2). Other variables that were significantly different by wave 2 and wave 3 attrition are shown in the appendices (Additional file 1: Appendices 2 and 3).

Table 2 Demographic characteristics by attrition

Wave 1 predictors of attrition in wave 2

Table 3 presents odds ratios calculated based on results from the logistic regression after stepwise selection. After controlling for all other measures collected in the wave 1 survey, age, race, ethnicity, experiencing COVID-19-like symptoms, change of normal diet, average hours of sleep per day, hours missed from work due to COVID-19, COVID-19 impact on productivity, self-rated survey difficulty, and specific HRQoL questions from the VR-12 (Q2a, “Does your health now limit you in moderate activities, such as moving a table, pushing a vacuum cleaner, bowling or playing golf? If so, how much?”; Q6b, “How much of the time during the past 4 weeks did you have a lot of energy?”; Q9, “Compared to one year ago, how would you rate your emotional problems now?”) were found to be associated with attrition at wave 2.

Table 3 Factors associated with panel attrition

Those aged 18–24 were significantly more likely to drop out of the study at wave 2 compared to other age groups. Respondents belonging to multiple race groups (OR 1.630, 95% CI 1.297–2.048) and of Hispanic origin (OR 1.698, 95% CI 1.266, 2.279) were more likely to drop out. Experiencing COVID-19-like symptoms but not requiring hospitalization (OR 1.384, 95% CI 1.052–1.821), change of normal diet (OR 1.246, 95% CI 1.042–1.490), and being essential workers (OR 1.223, 95% CI 1.020–1.466) were significantly associated with attrition. Average hours of sleep per day (OR 1.103, 95% CI 1.046–1.162), self-reported COVID-19 impact on productivity (OR 1.061, 95% CI 1.028–1.095), and self-reported survey difficulty (OR 1.075, 95% CI 1.026–1.126) were also positively correlated with attrition, but the association with average hours worked per day (OR 0.966, 95% CI 0.944–0.989) was negative.

As for HRQoL measures, those who were limited in moderate activities were 1.355 (95% CI 1.088, 1.688) times more likely to drop out than those who were not limited at all. Compared to those who reported never having a lot of energy, those who reported having energy “a little of the time” (OR 1.436, 95% CI 1.036–1.989) and “some of the time” (OR 1.778, 95% CI 1.289–2.450) were more likely to drop out, but those who answered “A good bit of the time”, “Most of the time”, or “All of the time” were not. Compared to those who reported no change to their emotional problems, those who felt much better than one year ago were 1.974 (95% CI 1.424, 2.736) times were more likely to drop out.

Wave 2 predictors of attrition in wave 3

After controlling for all other measures at wave 2, age, race, ethnicity, marital status, insurance type, political affiliation, medical history (arthritis, diabetes, stroke, and bronchitis), smoking history, BMI category, having medical care deferred, diagnosed of COVID-19, supporting social distancing policy, employment change, finances by the end of the month, self-rated survey difficulty, and two HRQoL measures, EQ-5D-5L visual analog scale (VAS) score and VR-12 Q3b (“During the past 4 weeks, were you limited in the kind of work or activities as a result of your physical health?”) were associated with attrition in wave 3.

Those who were younger than 35 years of age were significantly more likely to drop out at wave 3. Asians (OR 0.573, 95% CI 0.381–0.861) were less likely to drop out compared to Whites. Hispanics were 2.097 (95% CI 1.344, 3.273) times as likely to drop out than non-Hispanics. Compared to married respondents, those who were single (OR 0.774, 95% CI 0.600–0.999) and divorced (OR 0.509, 95% CI 0.338–0.767) were less likely to drop out. Those who had insurance from Medicare (OR 2.298, 95% CI 1.601–3.299) and had no insurance (OR 1.683, 95% CI 1.216–2.329) were more likely to drop out than those who had commercial insurance. Compared to Democrats, Republicans were 1.654 times as likely to drop out (95% CI 1.262–2.167). Having a history of arthritis (OR 0.621, 95% CI 0.413–0.932) or bronchitis (OR 0.386, 95% CI 0.217–0.686) was negatively associated with attrition while the association was positive with history of diabetes (OR 1.599, 95% CI 1.078–2.372) and stroke (OR 3.682, 95% CI 1.101–12.308). Those who smoked in the past and who were current smokers were both more likely to drop out than non-smokers. Those who were underweight were also more likely to drop out than people with normal weight (OR 1.651, 95% CI 1.076–2.534).

Diagnosis of COVID-19 (OR 5.026, 95% CI 2.026–12.473), having medical care deferred due to COVID-19 (OR 1.612, 95% CI 1.238–2.101), and supporting the social distancing policy (OR 1.047, 95% CI 1.004–1.092) were all positively correlated with attrition at wave 3. Compared to those who experienced no change to their employment, those who could work from home (OR 1.327, 95% CI 1.067–1.651) and those who lost their jobs (OR 2.028, 95% CI 1.035–3.975) were more likely to drop out, but those who were laid off temporarily were less likely to drop out (OR 0.577, 95% CI 0.337–0.988). Self-rated survey difficulty (OR 1.125, 95% CI 1.061–1.193) was also positively correlated with attrition.

Financial situation by the end of the month in wave 2 was found to be a significant predicter of attrition at wave 3. Specifically, those who had just enough money to make ends meet (OR 1.435, 95% CI 1.132–1.821) were more likely to discontinue participation compared to those who ended the month with some money left over. Overall HRQoL measured by EQ-5D-5L visual analog scale (VAS) score was positively correlated with attrition (OR 1.011, 95% CI 1.005–1.017). Additionally, compared to those who were not at all limited in the kind of work or activities due to physical health, those who were limited to some extents were all more likely to drop out, except for those who were always limited.

Discussion

In this study, we aimed to compare factors associated with attrition at different times during the pandemic. To our knowledge, this is also the first study that assesses longitudinal panel attrition regarding COVID-19 related health behavioral changes and HRQoL measures during the pandemic in the US. Our response rate from wave 1 to wave 2 was comparable to those reported in another panel attrition study during a similar period of the COVID-19 pandemic [15]. While we enrolled new participants at wave 2, the response rate from wave 2 to wave 3 was lower. This may result from the longer gap between wave 2 and wave 3 but may also suggest a decreased interest in COVID-19 survey participation as the initial shock wore off.

Despite the constantly changing COVID-19 circumstances during the study period, three characteristics consistently predicted attrition. In both wave 1 and wave 2, young adults were more likely to drop out of the study. This is consistent with reports of higher attrition rates with the younger population in the literature [15,16,17,18,19]. Hispanic participants were also more likely to drop out in both of our samples. Furthermore, self-rated survey difficulty was positively associated with attrition. One study reported that experience with past surveys could be predictive of panel attrition [20]. Participants with an initial negative survey experience are less likely to continue participation. Therefore, to reduce attrition, longitudinal survey designers should balance questionnaire granularity with respondent burden.

While race has also consistently been a predictor of attrition in our data, the specific association remains elusive. The literature is also inconsistent on the effect of race groups on attrition. Studies have found higher dropout rates for Blacks, Asians, American Indians or Alaska Natives, and multiple race groups in various settings but another study reported no such relationship [15, 16, 18, 21, 22]. Although unclear from our data, it is possible that the change in the specific association between race group and attrition during different time periods results from heterogeneous COVID-19 shock on different race groups. This would also explain why evidence from literature is not consistent as each study was reporting an effect specific to the survey topic and study setting.

We found that self-reported concerns for COVID-19 were associated with attrition in both time periods but was expressed by experience of COVID-19 symptoms from wave 1 to wave 2 and confirmed COVID-19 diagnosis from wave 2 to wave 3, suggesting a shift of attention that coincided with the spiking number of new COVID-19 cases in the US in the latter time period [23]. Similarly, employment measures were also predictive of attrition in both time periods but was represented by essential worker status and employment change, respectively, which indicates a shift of job-related concerns. Health behavioral and work productivity changes were only significant in the first half of the study when people were still adjusting to the new lifestyle with capricious lockdown and social distancing policies. Repeatedly, we observe the pattern that while several factors are persistently associated with attrition, the selected measures adapt to reflect the most concerning matter at the time. This dynamic nature calls for researchers’ attention to the circumstance of the data collection, in addition to widely recognized drivers of attrition in the literature.

We found that higher EQ-5D-5L VAS score was predictive of the attrition at wave 3. This result differs from the literature in which one study reported higher participation at multiple time points among cancer survivors with higher HRQoL scores measured by the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30) and another study reported no significant association between survey participation and subjective well-being rated on a 0–10 scale [24, 25]. One possible explanation for this counter-intuitive relationship is that the EQ-5D-5L VAS score may be capturing some unidentified aspect of HRQoL that negatively affected the survey participation. For example, respondents who had higher hopes and were more optimistic about the pandemic might be more dismissive of the survey. Despite the elusive mechanism, this presents a net effect of HRQoL on survey participation, incorporating not only identified but also unidentified aspects of HRQoL such as resilience. One suggestive piece of evidence is the significant difference between those who lost their jobs and those who were laid off temporarily. While financial stress exists in both circumstances, those who were laid off temporarily expected a re-employment in the future and were more resilient about the situation. Another piece of evidence is that measures of ideology such as political affiliation and support of social distancing policy became predictive of attrition in wave 3 but not in wave 2.

Remaining results from the HRQoL surveys suggest that limited physical activity is predictive of attrition in both time periods, despite being represented by different HRQoL questions. Mental health only predicts panel attrition between wave 1 and wave 2. This is consistent with the observation that health behavioral and productivity changes was predictive in and only in that same time period. However, it is inconclusive from these findings how mental health status specifically impacts panel attrition. A study focused on mental health found that those who completed only one or two surveys had higher baseline prevalence of anxiety/depression symptoms than those who completed all four surveys [15]. In our study, while there were significant differences in anxiety/depression measured by EQ-5D-5L and by a few VR-12 questions before controlling for other variables at wave 1 (Additional file 1: Appendix 2), we did not see a similar trend after adjustment. Overall, the impact of HRQoL on panel attrition is found to be bell-shaped. This may result from both low numbers of observations for participants at the extreme (Additional file 1: Appendices 2 and 3) and lower stability for participants in the middle to remain unchanged. The latter explanation is supported by the observation that improvement of emotional health compared to one year ago was significantly associated with attrition. This suggests that change in HRQoL, rather than the specific level of HRQoL, affects attrition.

Longitudinal HRQoL assessment is vital to healthcare research during the current pandemic and online surveys will continue to be a major data source for such research. However, restricting analyses to those who always respond may lead to biased results. Our data reveal some patterns of panel attrition during the COVID-19 pandemic based on demographic, behavioral, and HRQoL characteristics of the survey participants. Furthermore, we identify three categories for predictors of panel attrition, including 1) predictors that affect attrition consistently regardless of survey topic and circumstances of the study (e.g., age and survey difficulty), 2) common themes that are consistently associated with attrition but the specific predictor and association may vary given the circumstances (e.g., race and employment), and 3) predictors or themes that are context-specific (e.g., behavioral and productivity changes). Because both common themes and context-specific predictors can lead to heterogeneous attrition, longitudinal panel studies should always attempt to assess attrition bias before drawing any conclusions.

The current pandemic created an unnatural and constantly evolving situation that is substantially different from pre-pandemic life (e.g., work-from-home orders, movement restrictions, online learning, etc.), thus making it challenging to compare pre-pandemic and pandemic outcomes without violations of the “Ceteris paribus” principle. Our findings may be, to some extent, specific to the COVID-19 pandemic. For example, most non-essential workers were spending much more time at home/online than before the pandemic. However, given that MTurk is a platform for which respondents voluntarily participate, we believe that our conclusions are still relevant to the post-pandemic era. For example, future studies may want to consider more variables when trying to minimize attrition rates in the design stage, or to understand the effects of attrition on results interpretation. While attrition is often not random, most patterns may be predictable based on prior knowledge. Our analysis produced a partially context-specific explanation as to how attrition manifested. Future research can examine our findings and use our categorization of attrition predictors to infer factors that would contribute to attrition in a different context. Additional measures may also be taken to improve follow-up rates and ensure sufficient data collection in the targeted population (e.g., providing compensation in gift cards from Amazon or coffee shop instead of grocery stores to attract younger population), allowing for better control of attrition in future longitudinal studies, although this must be balanced against ensuring sample representativeness. Ultimately, researchers should be aware of and at minimum, acknowledge non-random attrition when interpreting their results.

Limitations

This study has several limitations. First, we recruited participants from the MTurk platform. MTurk has been shown to have mixed external validity to the general US population [26,27,28,29,30,31,32,33,34,35]. Second, our sample was stratified on age, gender and race. Additional measures not listed could improve the generalizability of our sample. Third, our wave 2 sample included participants from wave 1. These participants were more likely to respond to additional surveys, given that they already completed one round of follow-up. Fourth, the gap between wave 2 and wave 3 was longer than that between wave 1 and wave 2, which could lead to increased attrition rates. Finally, the survey questions remained virtually unchanged in all 3 waves, perhaps contributing to respondent burden and leading to higher rates of attrition.

Conclusions

Previous research on panel attrition focused on demographic measures and personality traits. However, the societal disruption caused by the COVID-19 pandemic presents new challenges for follow-up in longitudinal panels. We identified multiple demographics, behavioral, and HRQoL measures that predicted attrition in our panel. These results suggest the need to refresh existing considerations when conducting longitudinal panel surveys during the COVID-19 pandemic. Future research may use our findings to improve study design and data interpretation.