Assessing quality of life in a clinical study on heart rehabilitation patients: how well do value sets based on given or experienced health states reflect patients’ valuations?
© Leidl et al. 2016
Received: 4 September 2015
Accepted: 16 March 2016
Published: 22 March 2016
Quality of life as an endpoint in a clinical study may be sensitive to the value set used to derive a single score. Focusing on patients’ actual valuations in a clinical study, we compare different value sets for the EQ-5D-3L and assess how well they reproduce patients’ reported results.
A clinical study comparing inpatient (n = 98) and outpatient (n = 47) rehabilitation of patients after an acute coronary event is re-analyzed. Value sets include: 1. Given health states and time-trade-off valuation (GHS-TTO) rendering economic utilities; 2. Experienced health states and valuation by visual analog scale (EHS-VAS). Valuations are compared with patient-reported VAS rating. Accuracy is assessed by mean absolute error (MAE) and by Pearson’s correlation ρ. External validity is tested by correlation with established MacNew global scores. Drivers of differences between value sets and VAS are analyzed using repeated measures regression.
EHS-VAS had smaller MAEs and higher ρ in all patients and in the inpatient group, and correlated best with MacNew global score. Quality-adjusted survival was more accurately reflected by EHS-VAS. Younger, better educated patients reported lower VAS at admission than the EHS-based value set.
EHS-based estimates were mostly able to reproduce patient-reported valuation. Economic utility measurement is conceptually different, produced results less strongly related to patients’ reports, and resulted in about 20 % longer quality-adjusted survival.
Decision makers should take into account the impact of choosing value sets on effectiveness results. For transferring the results of heart rehabilitation patients from another country or from another valuation method, the EHS-based value set offers a promising estimation option for those decision makers who prioritize patient-reported valuation. Yet, EHS-based estimates may not fully reflect patient-reported VAS in all situations.
Quality of life is a key endpoint in a number of clinical studies. Its measurement requires collection of data on the dimensions and items by which quality of life is being described. In order to gain an overall result, an aggregation step is needed that can be performed by a researcher—such as defining the average across all items as the aggregate—or by an individual’s valuation expressing subjective summary assessment.
This paper considers alternative options to aggregate results by the response of individuals. The valuation of health states is known to vary widely between countries [1–3]. Clinical studies may include patients from different countries, which may influence quality of life results by varying valuations. In addition, health care decision makers and regulators may question whether valuation is appropriate if it has been derived from another population, or by another method than they require. In these cases, results may incur methodological biases when quantifying quality of life endpoints. If access to original study data is provided, the impact of the required approach for valuation can be analyzed by re-valuing health states reported by estimates of the respective value from a population study, that is by using a so-called value set.
Approaches to valuing quality of life may be crucial for study results. Methodologically, valuation can differ in various aspects. Patients or the general population may be asked to perform the valuation. For direct valuation, the visual analog scale (VAS) may be used, or a choice-based method such as the time-trade-off method (TTO) [4, 5]. The health states being valued could be the individual’s own, just experienced health state (EHS), or they could be hypothetical, given health states (GHS). When referring to quality-adjusted life years (QALYs) in economic evaluation studies, quality of life measures are most often integrated using a choice-based valuation of GHS. Such utilities reflect the ex-ante preferences of individuals with regard to health. Typically, they are elicited at the population level, thought to reflect population preferences, and thus used to inform decisions on allocating health care resources . This procedure is well established in decision practice: for example, the UK National Institute for Health and Care Excellence (NICE) has been using this type of information for more than 15 years.
Critical voices have also been raised concerning the theoretical foundations of using such community preferences for allocating health care funds . In addition, some jurisdictions require measurement of patient benefit as the primary indicator in order to decide upon health care technologies. In Sweden, the Dental and Pharmaceutical Benefits Agency prefers EHS-based valuation over GHS-based valuation . In Germany, the Social Code, Book V § 35 1(b) defines patient relevant benefit, including quality of life, as the key effect criterion; this does not refer to ex-ante preferences of the population.
In recent years, several EHS-based value sets have been developed that estimate the individual’s valuation of his/her own health state [8–10]. EHS-based value sets have been used in a range of epidemiological and clinical studies, including diabetes , stroke , hip replacement [13, 14], inflammatory bowel disease , and chronic diseases . EHS-based value sets predict how an average person experiencing a health state would value this health state. For decision makers who focus on patient benefit, EHS-based value sets may thus provide a substitutive valuation in situations where context-specific, patient-reported valuation is lacking. As a pre-condition to such substitution in quality of life measurement, the value set has to accurately predict the patient’s valuation as well as the valuation of patient subgroups, such as patients in the treatment arms of a trial.
This paper assumes the perspective of a decision maker who requests evidence on patient relevant benefit and thus would prefer patients to directly value their own health states. If this is not available, GHS-based and EHS-based value sets derived from population studies could provide options for a second-best solution. In order to assess the performance of such second-best choices, the paper uses quality of life data from a published, clinical study of heart rehabilitation patients. As reference for the endpoint of quality of life, patient-reported valuations are taken. The paper starts out from the counterfactual assumption that the latter are lacking and thus uses two value sets for valuation. Indeed, the clinical study has collected patient-reported valuations. The paper then investigates how well the value sets reflect patients’ valuations.
The clinical study re-analyzed here compared inpatient and outpatient rehabilitation of patients following an acute cardiac event. The study was conducted in Germany and labeled SARAH (Stationäre versus ambulante Rehabilition nach akutem Herzereignis). Results have been presented elsewhere, showing that, over 3 weeks of intervention and a 12-month follow-up, inpatient and outpatient rehabilitation did not differ significantly with regard to the primary medical endpoint of event-free survival, combining myocardial infarction, stroke, heart failure, life-threatening rhythm events, unstable angina, and death , and also did not differ significantly with regard to generic quality of life and cost-effectiveness . The study was carried out according to the Declaration of Helsinki, and was approved by the Institutional Ethics Board of Ulm University. Written informed consent was obtained from all participants. With inpatient rehabilitation representing standard care in this context, feasibility of randomization had to be clarified first. The study thus used a comprehensive cohort design : Patients who had agreed to participate were offered the option of being randomized and, if they refused, were offered the option to choose a treatment arm. Included were patients below 66 years of age, with myocardial infarction occurring less than 3 months before admission to the rehabilitation hospital. Some 163 patients met the inclusion criteria and were recruited. Only four patients agreed to randomization; of the rest, 112 patients starting in the inpatient rehabilitation arm and 51 patients receiving outpatient rehabilitation were allocated on a preference basis. Patient enrolment started in 2002 and study follow-up ended in 2005. For the methodological re-analysis in the present study, all patients with quality of life measurements included in the cost-effectiveness study were used. To compare valuation measurement, we restricted this study to observed measurements, disregarding imputation.
Quality of life was measured using the EQ-5D-3L, a standardized instrument that is available in more than 170 official language versions . A comprehensive review on the use of the EQ-5D-3L in cardiovascular diseases found 60 application studies and ten studies that analyzed validity or reliability, with the results clearly supporting this use. However, results were not stratified with regard to the use of different types of valuation methods . For German heart rehabilitation patients, the EQ-5D-3L has also been shown to be a valid and reliable tool . In the re-analyzed study, patients were requested to fill in the EQ-5D-3L descriptive system and the VAS at six points in time: admission, discharge, and after 3, 6, 9, and 12 months of follow-up (FU).
Approaches studied to value quality of life
Patients’ VAS (reference)
Population (value set)
What is being valued?
Experienced health state
Hypothetical health state
How is it valued?
Anchoring for death
Endpoint when multiplied by time
A patient’s VAS valuation serves as the reference for patient benefit. Performance of the two value sets was analyzed in six steps (Additional file 1: Table S1): comparison of raw values, deviations from the reference, correlation with the reference and with an accepted medical endpoint, comparison of quality-adjusted survival, and identification of factors influencing differences from the reference.
Mean absolute error (MAE) of value sets compared with VAS values reported by patients are investigated over the six measurement points. For correlation between patients’ VAS and value sets, Pearson correlation coefficients ρ are analyzed for absolute valuations as well as differences in valuations over time—the latter are a key indicator of effectiveness trend. To estimate confidence intervals, Bootstrap methods were applied for the correlation coefficients. Correlations were investigated for all patients, for the two study arms of inpatients and outpatients, and for the subgroups of the lower and upper quartiles of patients with regard to VAS valuation reported at admission.
External validity is analyzed using an acknowledged clinical measure of quality of life in cardiac patients: the MacNew, specifically its global score . For all patients, Pearson correlations with the MacNew global score are calculated for patients’ own valuations as well as for the two value sets, again for both absolute valuations reported as well as differences in valuations over time.
Overall treatment effect in terms of quality-adjusted survival is captured by multiplying value by the duration it applies to. In case of the choice-based GHS-Germany, this produces traditional quality-adjusted life years (QALYs). For patients’ VAS and EHS-Germany, quality-adjusted survival is based upon experienced health and thus differs from the ex-ante concept of utility-based QALYs. Analyses are conducted for all patients, for the inpatient and outpatient rehabilitation arms of the clinical study, and for the upper and lower quartiles of patients in terms of quality of life reported. These stratifications are intended to reflect the increasing relevance of analyzing patients by subgroups, which is found especially in patient benefit assessment in German drug regulation .
Finally, we explain differences between the two value sets and patients’ own reports by repeated measures regression. Explanatory variables include socioeconomic ones such as age and sex, education, and family status as well as health determinants such as smoking and baseline MacNew global score. All effects are further tested on differences with respect to the time point at which average differences were largest.
The sample in this study comprised a total of 145 patients with 98 patients in the inpatient arm and 47 patients in the outpatient arm. The overall share of women was 22.8 % with 21.4 % in inpatient care and 25.5 % in outpatient care. Age ranged from 26 to 76 years, averaging 55.6 years with 54.2 years in the inpatient arm and 56.2 years in the outpatient arm.
Valuations by patients’ VAS and value sets, six observation points
FU 3 months
FU 6 months
FU 9 months
FU 12 months
Pearson correlation of value sets with patients’ VAS valuations and their differences
Lower quartile, patients’ VAS at admission
Upper quartile, patients’ VAS at admission
MAEs between both the EHS-based and the GHS-based value sets and patients’ VAS differed most at “admission”, with an increase of 50 % for the EHS-based value set and 56 % for the GHS-based one. According to the repeated measures regression, these time dependent differences were significantly more pronounced for younger patients. For the EHS-based value set, an additional increase was found for patients with higher level of education and for patients not living alone (Additional file 1: Table S2). For the GHS-based value set, MAEs were larger for patients with lower MacNew values at baseline, over all observation points (Additional file 1: Table S3).
The counterfactual design of this re-analysis enabled comparison with patients’ own valuations that were used as a reference. In most analyses, the population-based estimates of the EHS-based approach were found to closely reproduce the reference of patient-reported valuations. This was especially pronounced for the mean differences between reference and value set, which also reflect potential bias. It also notably existed for mean absolute errors that integrate all individual variation. In addition, differences in correlations for all patients and for subgroups investigated underscored the closer relation of the EHS-based value sets to patient-reported outcome. As the standard used in economic evaluation studies, the GHS-based value set was found to be systematically less strongly related to patients’ reports including both absolute valuations and differences in valuations over time, and also to render systematically higher levels of overall patient benefit in terms of quality-adjusted survival. The latter corresponds to earlier findings that the GHS-based value set tends to underestimate VAS values reported for health states with severe problems, and to overestimate them for states with no or moderate problems [9, 23]. In the present study, the share of patients reporting a severe problem in at least one of the EQ-5D dimensions was 7.6 % at admission, and reduced to 3.0 % at 12 months follow-up. The GHS-based value set describes a larger gain for these patients than they report on the VAS, thus contributing to a higher level of quality-adjusted survival.
EHS-based results were not only more strongly linked to the benefits directly derived from patients, but even showed lower dispersion than these reference values. However, they were found not to reproduce valuation appropriately directly after acute treatment for myocardial infarction: At admission to rehabilitation, patients’ VAS was much lower than the EHS-based approach. Acute live-threatening experience may thus hardly be reflected in the population sample from which the EHS-based value set has been estimated. In particular, younger patients with a higher level of education and not living alone tended to report VAS values lower than the EHS-based value set. The experience of these subgroups was not fully reflected by estimates of population experience.
Patients analyzed had presented with different types of acute cardiac events, although the study design does not allow for extrapolation of findings to all post-acute heart patients. A main methodological limitation of this study is that the two national value sets used are based on quite different concepts: GHS-based valuation is suited to decision makers who intend to allocate money according to ex-ante preferences of the population regarding health. EHS-based valuation aims to derive patient benefit from an average experience of a population sample. It is thus suited to decision makers whose priority is to assess benefit from the patient’s perspective. Decision makers have to make a normative choice about which concept they want to use when appraising the evidence. Given that the EHS-based value set is conceptually more closely related to the reference of patients’ VAS, it could be expected that it may render better estimates for the reference. It is well known that the TTO method tends to produce higher values than VAS valuation, for example when comparing national value sets based on these two methods [1–3]. Quantifying the relative influence of valuation method (VAS, TTO) and type of health state values (GHS, EHS) would have required comparison of four types of value sets which were not available. Yet, for a clinical study on heart rehabilitation patients, three decision relevant points were elaborated here by comparing patient-reported outcome with two value sets: 1. it was shown to what extent traditional, utility-based quality of life measurement and QALYs reflect patient benefit; 2. it was quantified to what extent the normative choice between the two value sets affects effectiveness outcome; and 3. it was shown that the EHS-based value set offers an option to estimate patient-reported outcomes while identifying situations in which estimation is not accurate.
A methodological limitation is that the GHS-based value set has been anchored for the state of being dead, whereas the EHS-based concept could not be anchored for consistency. Aside from methodological discussion about anchoring , studies have shown that the impact of anchoring on results may be minor in general population samples , which is where the EHS-based approach has been derived from.
Another important point is that the original SARAH study only included patients from the German health care system. The present study thus could not investigate the transfer of quality of life results between health systems. In order to fully quantify the transfer problem, comparison between value sets of identical methodology adapted to two health systems would be needed. Results from an EHS-based value set may yet be used to check whether the outcomes of a clinical study are sensitive to the valuation approach. Decision makers can thus be informed about whether or not valuation methods and transfer problems may play a role in the assessment of patient benefit.
A last conceptual limitation is that, with regard to a specific health care system, only the valuation step has been considered. A similar type of problem might eventually occur for the description of quality of life, although this was not included in the scope of this study.
To jurisdictions responsible for market access, the concept of valuing quality of life is an issue of salient relevance. For a clinical intervention study, this analysis is, to the best of our knowledge, the first to quantify the impact on outcomes measured of using an EHS-based value set instead of the traditional GHS-based approach. Decision makers who consider patient relevant benefit should especially take into account possible differences between traditional economic utilities and patient-reported outcomes.
The results provide a new option to those who give priority to patient-reported outcomes and to results derived from their target population: In order to adapt quality of life results from clinical studies that have been derived in other populations or have not been fully based on patients’ reports, the EHS-based value set may be used to estimate patients’ valuations. This appropriately achieved, the resulting clinical endpoint may better reflect patient benefit, and may thus bring closer together clinical and economic evaluations.
The performance of the EHS-based estimation is very promising, but the results also indicated that, in situations close to acute vital events, estimates of general population experience may not fully reproduce the patients’ perspective. Yet, the performance of EHS-based value sets in clinical populations different from the one investigated here needs to be tested before use.
An additional file shows tables giving an overview on performance analysis and on repeated measures regression to explain absolute differences between the value set and patients’ VAS [see Additional file 1.doc].
We gratefully acknowledge the work of further members of the team who evaluated effectiveness and cost-effectiveness in the original SARAH study (all University of Ulm, Germany): Armin Imhof, Wolfgang König, Yuefei Liu, Rainer Muche, Cornelia Kropf, Susanne Brandstetter, and Daniel H Schiefer.
Parts of this study have been supported by joint public grants from the German Federal Ministry for Education and Research (01GD0108) and the Federation of German Pension Insurance Institutes (02706).
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Szende A, Janssen B, Cabases J. Self-reported population health: an international perspective based on EQ-5D. Dordrecht: Springer; 2014Google Scholar
- Knies S, Evers SM, Candel MJ, Severens JL, Ament AJ. Utilities of the EQ-5D: transferable or not? PharmacoEconomics. 2009;27(9):767–79. doi:10.2165/11314120-000000000-00000.View ArticlePubMedGoogle Scholar
- König HH, Bernert S, Angermeyer MC, Matschinger H, Martinez M, Vilagut G, et al. Comparison of population health status in six European countries: results of a representative survey using the EQ-5D questionnaire. Med Care. 2009;47(2):255–61. doi:10.1097/MLR.0b013e318184759e.View ArticlePubMedGoogle Scholar
- Torrance GW. Measurement of health state utilities for economic appraisal. J Health Econ. 1986;5(1):1–30.View ArticlePubMedGoogle Scholar
- Parkin D, Devlin N. Is there a case for using visual analogue scale valuations in cost–utility analysis? Health Econ. 2006;15(7):653–64. doi:10.1002/hec.1086.View ArticlePubMedGoogle Scholar
- Weinstein MC, Torrance G, McGuire A. QALYs: the basics. Value Health. 2009;12 Suppl 1:S5–9. doi:10.1111/j.1524-4733.2009.00515.x.View ArticlePubMedGoogle Scholar
- Gandjour A. Theoretical foundation of patient v. population preferences in calculating QALYs. Med Decis Making. 2010;30(4):E57–63. doi:10.1177/0272989x10370488.View ArticlePubMedGoogle Scholar
- Burström K, Sun S, Gerdtham UG, Henriksson M, Johannesson M, Levin LA, et al. Swedish experience-based value sets for EQ-5D health states. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2014;23(2):431–42. doi:10.1007/s11136-013-0496-4.View ArticleGoogle Scholar
- Leidl R, Reitmeir P. A value set for the EQ-5D based on experienced health states: development and testing for the German population. PharmacoEconomics. 2011;29(6):521–34. doi:10.2165/11538380-000000000-00000.View ArticlePubMedGoogle Scholar
- Sun S, Chen J, Kind P, Xu L, Zhang Y, Burstrom K. Experience-based VAS values for EQ-5D-3L health states in a national general population health survey in China. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2015;24(3):693–703. doi:10.1007/s11136-014-0793-6.View ArticleGoogle Scholar
- Kiadaliri AA, Gerdtham UG, Eliasson B, Gudbjornsdottir S, Svensson AM, Carlsson KS. Health utilities of type 2 diabetes-related complications: a cross-sectional study in Sweden. Int J Environ Res Public Health. 2014;11(5):4939–52. doi:10.3390/ijerph110504939.View ArticlePubMedPubMed CentralGoogle Scholar
- Hunger M, Sabariego C, Stollenwerk B, Cieza A, Leidl R. Validity, reliability and responsiveness of the EQ-5D in German stroke patients undergoing rehabilitation. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2012;21(7):1205–16. doi:10.1007/s11136-011-0024-3.View ArticleGoogle Scholar
- Gordon M, Greene M, Frumento P, Rolfson O, Garellick G, Stark A. Age- and health-related quality of life after total hip replacement: decreasing gains in patients above 70 years of age. Acta Orthop. 2014;85(3):244–9. doi:10.3109/17453674.2014.916492.View ArticlePubMedPubMed CentralGoogle Scholar
- Vogl M, Leidl R, Plotz W, Gutacker N. Comparison of pre- and post-operative health-related quality of life and length of stay after primary total hip replacement in matched English and German patient cohorts. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2015;24(2):513–20. doi:10.1007/s11136-014-0782-9.View ArticleGoogle Scholar
- Leidl R, Reitmeir P, König HH, Stark R. The performance of a value set for the EQ-5D based on experienced health states in patients with inflammatory bowel disease. Value Health. 2012;15(1):151–7. doi:10.1016/j.jval.2011.08.004.View ArticlePubMedGoogle Scholar
- Little M, Reitmeir P, Peters A, Leidl R. Does experience matter when valuing health? A comparison of EQ-5D value tariffs in a German population study. Value Health. 2014;14(4):364–71.View ArticleGoogle Scholar
- Steinacker JM, Liu Y, Muche R, Koenig W, Hahmann H, Imhof A, et al. Long term effects of comprehensive cardiac rehabilitation in an inpatient and outpatient setting. Swiss Med Wkly. 2011;140:w13141. doi:10.4414/smw.2010.13141.PubMedGoogle Scholar
- Schweikert B, Hahmann H, Steinacker JM, Imhof A, Muche R, Koenig W, et al. Intervention study shows outpatient cardiac rehabilitation to be economically at least as attractive as inpatient rehabilitation. Clin Res Cardiol. 2009;98(12):787–95. doi:10.1007/s00392-009-0081-6.View ArticlePubMedGoogle Scholar
- Muche R, Imhof A, Studiengruppe S. Das Comprehensive Cohort Design als Alternative zur randomisierten kontrollierten Studie in der Rehabilitationsforschung: Vor- und Nachteile sowie Anwendung in der SARAH-Studie. Rehabilitation. 2003;42(06):343–9. doi:10.1055/s-2003-45457.View ArticlePubMedGoogle Scholar
- EuroQol Group. EQ-5D-3L. 2016. http://www.euroqol.org/eq-5d-products/eq-5d-3l.html. Accessed March 17, 2016.Google Scholar
- Dyer MT, Goldsmith KA, Sharples LS, Buxton MJ. A review of health utilities using the EQ-5D in studies of cardiovascular disease. Health Qual Life Outcomes. 2010;8:13. doi:10.1186/1477-7525-8-13.View ArticlePubMedPubMed CentralGoogle Scholar
- Schweikert B, Hahmann H, Leidl R. Validation of the EuroQol questionnaire in cardiac rehabilitation. Heart. 2006;92(1):62–7. doi:10.1136/hrt.2004.052787.View ArticlePubMedPubMed CentralGoogle Scholar
- Greiner W, Claes C, Busschbach JJ, von der Schulenburg JM. Validating the EQ-5D with time trade off for the German population. Eur J Health Econ. 2005;6(2):124–30.View ArticlePubMedGoogle Scholar
- Dixon T, Lim LL, Oldridge NB. The MacNew heart disease health-related quality of life instrument: reference data for users. Qual Life Res Int J Qual Life Asp Treat Care Rehab. 2002;11(2):173–83.View ArticleGoogle Scholar
- Ruof J, Schwartz FW, Schulenburg JM, Dintsios CM. Early benefit assessment (EBA) in Germany: analysing decisions 18 months after introducing the new AMNOG legislation. Eur J Health Econ. 2014;15(6):577–89. doi:10.1007/s10198-013-0495-y.View ArticlePubMedPubMed CentralGoogle Scholar
- Devlin NJ, Tsuchiya A, Buckingham K, Tilling C. A uniform time trade off method for states better and worse than dead: feasibility study of the ‘lead time’ approach. Health Econ. 2011;20(3):348–61. doi:10.1002/hec.1596.View ArticlePubMedGoogle Scholar
- Bernert S, Fernandez A, Haro JM, Konig HH, Alonso J, Vilagut G, et al. Comparison of different valuation methods for population health status measured by the EQ-5D in three European countries. Value Health. 2009;12(5):750–8. doi:10.1111/j.1524-4733.2009.00509.x.View ArticlePubMedGoogle Scholar