Data sources

The analyses are based on data collected on patients who underwent surgery during the first year of the programme (1 April 2009 to 31 March 2010). Patients were invited to complete a pre-operative questionnaire either when attending the pre-operative assessment clinic (which might be several weeks before their operation) or on admission on the day of surgery. The completed questionnaires were sent by the provider to the NHS Information Centre (IC) for processing. Providers were made up of NHS hospitals, treatment centres (ambulatory surgical centres) and independent (private) hospitals.

The NHS IC links data from patients' pre-operative questionnaires to data on their episode of care recorded in Hospital Episode Statistics (HES) to identify each patient's date of surgery. Post-operative questionnaires are then mailed to patients' homes, from a national source, three (hernia repair and VV surgery) or six (hip and knee replacements) months following surgery.

Linkage to HES was successful for 75% of patients. For those patients whose pre-operative questionnaire could not be linked to HES and for whom, therefore, the date of surgery was unknown, the post-operative questionnaires were sent out six (hernia repair and VV surgery) and nine (hip and knee replacements) months after the date of completion of the pre-operative questionnaire.

Patients were excluded from the analysis if their operation was cancelled or they died before the post-operative questionnaire was due to be sent. Patients who had not returned a completed post-operative questionnaire by the end of August 2011 were classified as non-responders.

Explanatory variables

Pre-operative questionnaires obtained data on the socio-demographic characteristics of patients comprising age, sex, living arrangements (alone or otherwise) and if the patient required assistance completing the pre-operative questionnaire. Data on ethnicity and socio-economic deprivation were not included and had to be obtained from the linked HES record. For patients without a linked HES record, or with missing information in the linked record, we searched for data on ethnicity and post code (from which socio-economic status was derived) in HES records of previous hospital admissions. Deprivation was measured using the Index of Multiple Deprivation (IMD) [21] and was categorised in five groups based on quintiles derived from the national distribution.

Data on several aspects of patients' clinical status were obtained from their pre-operative questionnaires: duration of symptoms; previous surgery for the same condition; comorbidities [4]; the EuroQol EQ-5D index score [22]; and for three procedures, a condition-specific PROM - Oxford Hip Score, [23] Oxford Knee Score, [24] and Aberdeen Varicose Vein Questionnaire [25].

Organisational factors comprised the type of provider (NHS or independent sector), when the pre-operative questionnaire was administered (pre-operative assessment clinic or day of admission), and the patient recruitment rate to the PROMs Programme. Data on questionnaire administration was missing for patients without a linked HES record because the date of surgery was derived from HES. Data on provider recruitment rates (proportion of eligible patients recruited to the survey), available from the NHS IC, [5] was calculated as the number of completed pre-operative questionnaires divided by the number of eligible patients, according to HES, for each procedure. Recruitment rates were not estimated for providers with fewer than 50 eligible patients.

Multiple imputation by chained equations was used to impute values for all missing pre-operative variables (except for organisational factors) for multivariable analysis under the assumption that data were missing at random [26, 27]. Ten data sets were imputed using the ICE command in Stata. Missing data were imputed from the pre-operative variables, provider level mean deprivation, provider level proportion reporting non-white ethnicity, and response to the post -operative questionnaire.

Data analysis

For each operation, the response rate to the post-operative questionnaire was estimated by each patient characteristic and organisational factor. Multivariable logistic regression models were used to estimate odds ratios of non-response with 95% confidence intervals adjusted for all patient characteristics and organisational factors, except provider recruitment rate. Estimates for timing of pre-operative questionnaire administration were based on an analysis of those patients for whom date of surgery was available through linkage with HES. The impact of differences in provider recruitment on non-response was examined separately in a logistic regression analysis, adjusted for other characteristics, with provider included in the model as a random effect to allow for the non-independence of observations within providers. No attempt was made to impute outcomes for non-responders.

Sample size and response rates

Between 1 April 2009 and 31 May 2010, 135 474 patients completed a pre-operative questionnaire. After excluding patients who had died (927) or whose surgery had been cancelled (3100), there were 131 447 patients for analysis: 37 961 hip replacements, 44 422 knee replacements, 34 964 hernia repair, and 14 100 VV surgery. Response rates were higher for hip (85.1%) and knee replacements (85.3%) than for hernia repair (72.9%) and VV surgery (64.8%).

Patients' socio-demographic characteristics

Patients' clinical characteristics

Organisational factors

Association between patient/organisational characteristics and non-response

Adjusted odds ratios of the clinical characteristics demonstrate that non-response was higher among patients who had previously undergone surgery for their condition (odds ratio 1.05 [0.97-1.14] - 1.42 [1.30-1.55]). Duration of symptoms was less consistent in its impact on non-response. A longer duration was associated with non-response in hip replacement and hernia repair but for VV surgery a shorter duration increased non-response. There was no significant association between symptom duration and non-response for knee replacement. There was a pattern of patients with worse health (comorbidity, EQ-5D, and condition-specific PROMs) having higher levels of non-response.

There was no evidence that the time of administration of the pre-operative questionnaire was associated with post-operative non-response except for VV surgery patients who had a slightly higher non-response rate if the pre-operative questionnaire had been administered on admission (OR 1.11 [95% CI 1.02-1.21]). For orthopaedic surgery there was evidence of a higher non-response rate in patients treated by NHS than independent providers (1.08 [0.94-1.23] - 1.24 [1.15-1.34]). According to the adjusted odds ratio of the providers' recruitment rate (Table 4), there is no evidence that providers who are successful in recruiting a high proportion of patients will have higher levels of non-response because they recruit more 'reluctant participants'. The odds ratio of a 20% increase in recruitment rate only ranged from 0.99 [0.94-1.04] to 1.03 [0.99-1.06] for the four procedures.

Main findings

Patients who did not respond to the post-operative questionnaire tended to have more severe primary conditions and a poorer pre-operative quality of life. In addition, response was lower in patients who were men, younger, non-white, deprived, living alone, and having undergone previous similar surgery. Orthopaedic patients were less likely to respond if they had been treated in an NHS rather than independent provider. The provider's pre-operative recruitment rate and the timing of pre-operative questionnaire administration were not associated with the post-operative response rate.

Comparison with previous research on non-response to post-operative questionnaires

We have studied many more determinants of the response rate to PROM questionnaires mailed to patients after a hospital episode than have been considered previously in published studies. Only the impact of age, sex and health status have been studied before (apart from in our own small study [4] in which we found more deprived people were less likely to respond). Our finding in this study that younger patients (under 55 years in particular) are less likely to respond is consistent with our previous study [4] and a study in Norway of spinal surgery patients, [12] whereas two other studies have found the opposite [10, 11]. The reason for this lack of consistency is unclear. It may reflect the influence of the medical condition and the clinical treatment: studies finding older people less likely to respond were based on investigations of men undergoing prostate surgery [10] and patients with acute coronary syndrome [11].

Our finding that men were less likely to respond is at odds with the literature, which has reported no association with sex for knee replacement [14] or for a mixed population of hospital patients [13]. Again, the reason for this difference in observed association is unclear.

In contrast, the association we observed of non-response with poorer health has been consistently reported in a wide variety of patients: prostate surgery [10], acute coronary syndrome [11], shoulder surgery [15], all hospital admissions [16], and our previous study of elective surgery [4].

Limitations of the study

The first limitation is that date of surgery was not available for 25% of the patients because their pre-operative questionnaire could not be linked to their HES record. As a result, these patients were not included in the analysis considering the timing of administration of pre-operative questionnaires. It is reassuring that in those patients whose PROMs could be linked to HES, we did not find evidence that the timing of the administration of the pre-operative questionnaires influenced response rates.

Second, it is important to take into account the volume of the data and the fact that they are observational. The large volume allows effects to be estimated with great precision, i.e. narrow confidence intervals, so even small effects are regarded as highly statistically significant. However, there is scope for selection biases and residual confounding such that even modest confounding could alter the apparent significance of a small effect.

Implications of findings

The implications of non-response for the use of PROMs data will depend on the extent to which non-response is associated with outcomes. Previous research has shown that those who do not respond have worse outcomes [10, 11, 13–20]. If this were to prove true for elective surgery and non-response is not taken into account, outcomes will be over-estimated, particularly among providers with high non-response rates.

A number of approaches have been suggested to adjust for bias resulting from non-response [2, 3, 11, 26]. Both weighting and multiple imputation are options for including the outcomes of patients who do not respond, based on the outcomes of patients for whom complete data are available [26, 27]. If this approach is adopted, this study has identified the patient and organisational factors that should be included in such models. However, the validity of these techniques is dependent on the assumption that the probability that data are missing does not depend on the value of the missing item having adjusted for observed characteristics (ie those that have been measured). This, however, may be problematic for PROMs data if non-response bias associated with the outcome is not predicted by observed characteristics (ie characteristics for which measurements do not exist). In such circumstances, index function models, including the Heckman method, would need to be employed [10, 11, 13–20].