Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England

Background

Patient-reported outcome measures are increasingly being used to compare providers. We studied whether non-response rates to post-operative questionnaires are associated with patients' characteristics and organisational features of providers.

Methods

131 447 patients who underwent a hip or knee replacement, hernia repair or varicose vein surgery in 2009-10 in England. Multivariable logistic regression to calculate adjusted odds ratios of non-response for characteristics of the patients and organisational characteristics of providers. Multiple imputation was used for missing patient characteristics. Providers were included as random effects.

Results

Response rates to the post-operative questionnaire were 85.1% for hip replacement (n = 37 961), 85.3% for knee replacements (n = 44 422), 72.9% for hernia repair (n = 34 964), and 64.8% for varicose vein surgery (n = 14 100). Across the four procedures, there were higher levels of non-response in men (odds ratios 1.03 [95% CI 0.95-1.11] - 1.35 [1.25-1.46]), younger patients (those under 55 years 3.01 [2.72-3.32] - 6.05 [5.49-6.67]), non-white patients (1.24 [1.11-1.38] - 2.08 [1.89-2.31]), patients in the most deprived quintile of socio-economic status (1.47 [1.34-1,62] - 1.86 [1.71-2.03]), those who lived alone (1.11 [0.99-1.23] - 1.27 [1.18-1.36]) and those who had been assisted when completing their pre-operative questionnaire (1.26 [1.10-1.46] -1.67 [1.56-1.79]). Non-response rates were also higher in patients who had poorer pre-operative health (three or more comorbidities: 1.14 [0.96-1.35] - 1.45 [1.30-1.63]). Providers' patient recruitment rates before surgery and the timing of pre-operative questionnaire administration did not affect the rates of response to post-operative questionnaires.

Conclusion

If non-response can be shown to be associated with outcome, then rates of non-response to post-operative questionnaires would need to be taken into account when these measures are being used to compare the performance of providers or to evaluate surgical procedures.

In April 2009 the National Health Service (NHS) in England embarked on a national programme to collect pre- and post-operative patient reported outcome measures (PROMs) in all patients undergoing hip replacement, knee replacement, groin hernia repair, and varicose vein (VV) surgery [1]. Participation in the programme is mandatory for all National Health Service (NHS) and independent sector providers (hospitals and treatment centres) of NHS-funded care, although individual patients can choose not to take part. Potential uses of the data include comparisons of the performance of providers, audit of the equity of care, evaluation of the cost-effectiveness of different procedures, and estimation of the amount of health gain for populations. There are plans to extend the programme to additional surgical procedures. The use of PROMs in long-term conditions is being explored. The only previous use of PROMs on a national scale is for hip surgery in Sweden [2].

Pilot studies for the PROMs Programme suggested that response rates to mailed post-operative questionnaires of between 75% and 90% were achievable, dependent on the operation performed [3, 4]. Whilst these are high, rates vary between health care providers from 30% to 100% [5] which might give rise to biased comparisons if systematic differences exist between responders and non-responders.

Although there is an extensive literature on factors influencing response in household and longitudinal surveys, [6–8] there is less evidence on surveys of hospital patients. In England, the response to the NHS inpatient survey tends to be lower amongst men, younger patients and non-white patients [9]. However, the evidence is more mixed for surveys of specific patient populations: some studies have reported older patients are less likely to respond [10, 11]; some have found the opposite [12] and others report either no association with age [9] or those at both extremes (youngest and oldest) are less likely to respond [13]. As regards the sex of patients, several studies have found no association [11–14]. Most studies have found that patients in worse health at the time of their hospital admission are less likely to respond to later questionnaires [10, 11, 15, 16] although one study did not find such an association [17]. The earlier small study we conducted of PROMs in surgery found that response was higher among older and less deprived patients [4].

The importance of responder bias is evident in the many studies that have reported that non-responders' have poorer outcomes [10, 11, 13–20]. Non-response therefore carries the risk of over-estimating the outcomes of providers with lower response rates. The aim of this study was to establish whether non-response is associated with patients' socio-demographic and clinical characteristics or with organisational factors for four surgical procedures. It was observational rather than testing a hypothesis.

Data sources

The analyses are based on data collected on patients who underwent surgery during the first year of the programme (1 April 2009 to 31 March 2010). Patients were invited to complete a pre-operative questionnaire either when attending the pre-operative assessment clinic (which might be several weeks before their operation) or on admission on the day of surgery. The completed questionnaires were sent by the provider to the NHS Information Centre (IC) for processing. Providers were made up of NHS hospitals, treatment centres (ambulatory surgical centres) and independent (private) hospitals.

The NHS IC links data from patients' pre-operative questionnaires to data on their episode of care recorded in Hospital Episode Statistics (HES) to identify each patient's date of surgery. Post-operative questionnaires are then mailed to patients' homes, from a national source, three (hernia repair and VV surgery) or six (hip and knee replacements) months following surgery.

Linkage to HES was successful for 75% of patients. For those patients whose pre-operative questionnaire could not be linked to HES and for whom, therefore, the date of surgery was unknown, the post-operative questionnaires were sent out six (hernia repair and VV surgery) and nine (hip and knee replacements) months after the date of completion of the pre-operative questionnaire.

Patients were excluded from the analysis if their operation was cancelled or they died before the post-operative questionnaire was due to be sent. Patients who had not returned a completed post-operative questionnaire by the end of August 2011 were classified as non-responders.

Explanatory variables

Pre-operative questionnaires obtained data on the socio-demographic characteristics of patients comprising age, sex, living arrangements (alone or otherwise) and if the patient required assistance completing the pre-operative questionnaire. Data on ethnicity and socio-economic deprivation were not included and had to be obtained from the linked HES record. For patients without a linked HES record, or with missing information in the linked record, we searched for data on ethnicity and post code (from which socio-economic status was derived) in HES records of previous hospital admissions. Deprivation was measured using the Index of Multiple Deprivation (IMD) [21] and was categorised in five groups based on quintiles derived from the national distribution.

Data on several aspects of patients' clinical status were obtained from their pre-operative questionnaires: duration of symptoms; previous surgery for the same condition; comorbidities [4]; the EuroQol EQ-5D index score [22]; and for three procedures, a condition-specific PROM - Oxford Hip Score, [23] Oxford Knee Score, [24] and Aberdeen Varicose Vein Questionnaire [25].

Organisational factors comprised the type of provider (NHS or independent sector), when the pre-operative questionnaire was administered (pre-operative assessment clinic or day of admission), and the patient recruitment rate to the PROMs Programme. Data on questionnaire administration was missing for patients without a linked HES record because the date of surgery was derived from HES. Data on provider recruitment rates (proportion of eligible patients recruited to the survey), available from the NHS IC, [5] was calculated as the number of completed pre-operative questionnaires divided by the number of eligible patients, according to HES, for each procedure. Recruitment rates were not estimated for providers with fewer than 50 eligible patients.

Multiple imputation by chained equations was used to impute values for all missing pre-operative variables (except for organisational factors) for multivariable analysis under the assumption that data were missing at random [26, 27]. Ten data sets were imputed using the ICE command in Stata. Missing data were imputed from the pre-operative variables, provider level mean deprivation, provider level proportion reporting non-white ethnicity, and response to the post -operative questionnaire.

Data analysis

For each operation, the response rate to the post-operative questionnaire was estimated by each patient characteristic and organisational factor. Multivariable logistic regression models were used to estimate odds ratios of non-response with 95% confidence intervals adjusted for all patient characteristics and organisational factors, except provider recruitment rate. Estimates for timing of pre-operative questionnaire administration were based on an analysis of those patients for whom date of surgery was available through linkage with HES. The impact of differences in provider recruitment on non-response was examined separately in a logistic regression analysis, adjusted for other characteristics, with provider included in the model as a random effect to allow for the non-independence of observations within providers. No attempt was made to impute outcomes for non-responders.

Sample size and response rates

Between 1 April 2009 and 31 May 2010, 135 474 patients completed a pre-operative questionnaire. After excluding patients who had died (927) or whose surgery had been cancelled (3100), there were 131 447 patients for analysis: 37 961 hip replacements, 44 422 knee replacements, 34 964 hernia repair, and 14 100 VV surgery. Response rates were higher for hip (85.1%) and knee replacements (85.3%) than for hernia repair (72.9%) and VV surgery (64.8%).

Patients' socio-demographic characteristics

Most hernia surgery patients were men (89%) whereas women formed the majority of patients having the other three procedures (Table 1). Patients having hip and knee replacements were, on average, older than patients having hernia and VV surgery (mean ages 67.7 and 68.7 versus 57.4 and 49.7 years, respectively.) The majority of patients were white. The percentage of non-white patients was largest (6.2%) for VV surgery and smallest (1.8%) for hip replacements. For all procedures, except VV surgery, patients living in the most deprived areas were under-represented given that around 20% of the patients would be expected in each national IMD quintile. Around a quarter of orthopaedic patients lived alone compared with 15% of hernia and VV surgery patients. About a fifth of orthopaedic patients had assistance in completing their pre-operative questionnaire compared with 12% for groin hernia and 8% for VV surgery.

Patients' clinical characteristics

Patients undergoing VV surgery were more likely to have undergone previous surgery for the same condition (39%) than patients undergoing the other three procedures (8-13%) (Table 2). The duration of symptoms also varied by procedure: 63.3% of hernia surgery patients had symptoms for less than one year compared with 73.4% of VV surgery patients having had symptoms for more than five years. Patients undergoing a joint replacement had more comorbidities, which corresponds to their older age. This was also reflected in the distribution of health-related quality of life (QoL) scores, measured by the EQ-5D. Mean EQ-5D scores were 0.34 for hip and 0.40 for knee replacements, 0.79 for hernia repair, and 0.76 for VV surgery.

Organisational factors

Completion of the pre-operative questionnaire on day of admission was more common for hernia and VV surgery patients (Table 3). For VV surgery, this reflects the lack of need for pre-operative assessment as many will not be undergoing a general anaesthetic. Around 16% to 20% of orthopaedic procedures and hernia repair were carried out by independent providers, whilst this was true for only 8% of VV procedures. Rates of recruitment to the PROMs Programme of 70% or more were commoner for hip and knee replacement (almost 60% of providers) than for the other procedures (20-30% of providers).

Association between patient/organisational characteristics and non-response

The adjusted odds ratios of the pre-operative patient characteristics and organisational factors for non-response to a post-operative questionnaire are shown in Table 4. Odds ratios of patients' socio-demographic characteristics show consistent patterns across the four procedures. Non-response rates were higher in men (odds ratio 1.03 [95% CI 0.95-21.11] - 1.35 [1.25-1.46]), those under 55 years (3.01 [2.72-3.32] - 6.05 [5.49-6.67]), non-white patients (1.24 [1.11-1.38] - 2.08 [1.89-2.31]), those in the most deprived IMD quintile (1.47 [1.34-1.62] - 1.86 [1.71-2.03]), those who lived alone (1.11 [0.99-1.23] - 1.27 [1.18-1.36]), and those who had assistance in completing the pre-operative questionnaire (1.26 [1.10-1.46] - 1.67 [1.56-1.79]).

Adjusted odds ratios of the clinical characteristics demonstrate that non-response was higher among patients who had previously undergone surgery for their condition (odds ratio 1.05 [0.97-1.14] - 1.42 [1.30-1.55]). Duration of symptoms was less consistent in its impact on non-response. A longer duration was associated with non-response in hip replacement and hernia repair but for VV surgery a shorter duration increased non-response. There was no significant association between symptom duration and non-response for knee replacement. There was a pattern of patients with worse health (comorbidity, EQ-5D, and condition-specific PROMs) having higher levels of non-response.

There was no evidence that the time of administration of the pre-operative questionnaire was associated with post-operative non-response except for VV surgery patients who had a slightly higher non-response rate if the pre-operative questionnaire had been administered on admission (OR 1.11 [95% CI 1.02-1.21]). For orthopaedic surgery there was evidence of a higher non-response rate in patients treated by NHS than independent providers (1.08 [0.94-1.23] - 1.24 [1.15-1.34]). According to the adjusted odds ratio of the providers' recruitment rate (Table 4), there is no evidence that providers who are successful in recruiting a high proportion of patients will have higher levels of non-response because they recruit more 'reluctant participants'. The odds ratio of a 20% increase in recruitment rate only ranged from 0.99 [0.94-1.04] to 1.03 [0.99-1.06] for the four procedures.

Main findings

Patients who did not respond to the post-operative questionnaire tended to have more severe primary conditions and a poorer pre-operative quality of life. In addition, response was lower in patients who were men, younger, non-white, deprived, living alone, and having undergone previous similar surgery. Orthopaedic patients were less likely to respond if they had been treated in an NHS rather than independent provider. The provider's pre-operative recruitment rate and the timing of pre-operative questionnaire administration were not associated with the post-operative response rate.

Comparison with previous research on non-response to post-operative questionnaires

We have studied many more determinants of the response rate to PROM questionnaires mailed to patients after a hospital episode than have been considered previously in published studies. Only the impact of age, sex and health status have been studied before (apart from in our own small study [4] in which we found more deprived people were less likely to respond). Our finding in this study that younger patients (under 55 years in particular) are less likely to respond is consistent with our previous study [4] and a study in Norway of spinal surgery patients, [12] whereas two other studies have found the opposite [10, 11]. The reason for this lack of consistency is unclear. It may reflect the influence of the medical condition and the clinical treatment: studies finding older people less likely to respond were based on investigations of men undergoing prostate surgery [10] and patients with acute coronary syndrome [11].

Our finding that men were less likely to respond is at odds with the literature, which has reported no association with sex for knee replacement [14] or for a mixed population of hospital patients [13]. Again, the reason for this difference in observed association is unclear.

In contrast, the association we observed of non-response with poorer health has been consistently reported in a wide variety of patients: prostate surgery [10], acute coronary syndrome [11], shoulder surgery [15], all hospital admissions [16], and our previous study of elective surgery [4].

Limitations of the study

The first limitation is that date of surgery was not available for 25% of the patients because their pre-operative questionnaire could not be linked to their HES record. As a result, these patients were not included in the analysis considering the timing of administration of pre-operative questionnaires. It is reassuring that in those patients whose PROMs could be linked to HES, we did not find evidence that the timing of the administration of the pre-operative questionnaires influenced response rates.

Second, it is important to take into account the volume of the data and the fact that they are observational. The large volume allows effects to be estimated with great precision, i.e. narrow confidence intervals, so even small effects are regarded as highly statistically significant. However, there is scope for selection biases and residual confounding such that even modest confounding could alter the apparent significance of a small effect.

Implications of findings

The implications of non-response for the use of PROMs data will depend on the extent to which non-response is associated with outcomes. Previous research has shown that those who do not respond have worse outcomes [10, 11, 13–20]. If this were to prove true for elective surgery and non-response is not taken into account, outcomes will be over-estimated, particularly among providers with high non-response rates.

A number of approaches have been suggested to adjust for bias resulting from non-response [2, 3, 11, 26]. Both weighting and multiple imputation are options for including the outcomes of patients who do not respond, based on the outcomes of patients for whom complete data are available [26, 27]. If this approach is adopted, this study has identified the patient and organisational factors that should be included in such models. However, the validity of these techniques is dependent on the assumption that the probability that data are missing does not depend on the value of the missing item having adjusted for observed characteristics (ie those that have been measured). This, however, may be problematic for PROMs data if non-response bias associated with the outcome is not predicted by observed characteristics (ie characteristics for which measurements do not exist). In such circumstances, index function models, including the Heckman method, would need to be employed [10, 11, 13–20].

If non-response can be shown to be associated with outcome using PROMs in elective surgery, then rates of non-response to post-operative questionnaires would need to be taken into account when these measures are being used to compare the performance of providers or to evaluate surgical procedures.

This work was funded by the Department of Health for England.

Affiliations

1. Department of Health Services Research & Policy, London School of Hygiene & Tropical Medicine, 15-17 Tavistock Place, London, WC1H 9SH, UK

   • Andrew Hutchings
   • , Jenny Neuburger
   • , Kirstin Grosse Frie
   • , Nick Black
   •  & Jan van der Meulen

Corresponding author

Correspondence to Andrew Hutchings.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

AH and JvdM conceived and designed the analyses; AH, JN and KGF carried out the analyses; AH and NB led the drafting which all authors contributed to. All authors read and approved the final manuscript

Reprints and Permissions