Retrospective evaluation versus population norms for the measurement of baseline health status
Health and Quality of Life Outcomes volume 10, Article number: 68 (2012)
Patient recall or the application of population norms are commonly used methods to estimate (unobservable) health status prior to acute-onset illness or injury; however, both measures are potentially subject to bias. This article reports tests of the validity of both approaches, and discusses the implications for reporting changes in health-related quality of life following acute-onset illness or injury.
Recalled pre-injury health status and health status at 5- and 12-months post-injury were collected from participants in a prospective cohort study of people injured in New Zealand. Reported post-injury health status was compared with recalled pre-injury status and New Zealand norms for two groups: those who reported having fully recovered, and those who had not.
There was a small but statistically significant difference between pre- and post-injury health state valuations for people who had fully recovered, with recalled pre-injury health status being higher than reported post-injury health. Perceived health status for those who had fully recovered was significantly higher than the population norm.
Retrospective evaluation of health status is more appropriate than the application of population norms to estimate health status prior to acute-onset injury or illness, although there may be a small upward bias in such measurements.
Generic measures of health status are designed to gauge changes in people’s health status over time such as their recovery from illness or injury. Instruments such as the Health Utilities Index, SF-6D and EQ-5D are used for deriving health state preference values for calculating Quality-Adjusted Life Years (QALYs) for use in economic cost-effectiveness analyses . This article uses the EQ-5D. Developed by the EuroQol Group, the EQ-5D represents health in terms of five dimensions: mobility, self-care, ability to participate in usual activities, pain or discomfort, and anxiety or depression; with three possible responses per dimension (no problems, moderate problems, and extreme problems) .
The EQ-5D has been included in national population health surveys in the United Kingdom, Canada, China, Finland, Spain, Denmark, the United States and New Zealand [3, 4]. The National Institute for Health and Clinical Excellence (NICE) has recommended the EQ-5D be used in trials and observational studies of health outcomes to provide QALY information about the effects of new treatments . Since 2009, NHS secondary health providers in England have been asked to collect EQ-5D data for four surgical patient groups, pre- and post-operatively, as part of the Patient Reported Outcome Measures (PROMS) initiative . Data have been collected from hundreds of thousands of patients so far .
To measure change in health status, information about pre- and post-intervention health is required. Similarly, if the focus is determining health burden borne by groups affected by particular conditions, information about health before and after the onset of the condition is required. However, in studies looking at acute-onset conditions (e.g. cancer, stroke, injury), participants are usually recruited only after the health event has occurred. In such cases, researchers tend to adopt one of two approaches. Either they apply population norms to estimate pre-onset health, or they ask participants to ‘recall’ their pre-onset health status. Both approaches have limitations.
Applying population norms as the pre-onset health status may, logically, either under- or over-estimate the true health burden. The former would arise when, in fact, the people affected by a condition such as myocardial infarction were actually in poorer health before the myocardial infarction occurred; the latter when, in fact, the people with, say, traumatic brain injury were in better health than the general population before the injury. Similarly, when applying recalled health status the burden may be over-estimated if participants recall unrealistically high health states.
A study by Watson and colleagues  investigated the health status of patients reporting they had “recovered” from injury vis-à-vis their recalled pre-injury status and population norms, using the SF-36, SF-6D and AQoL instruments. No statistically significant differences, or at most only marginal differences, were found between recalled pre-injury health status and “recovered” status one year later, and recalled pre-injury statuses were consistently higher than general population norms. However, this study was restricted to hospitalised patients, and had a small sample size (n = 186), of whom only 61 reported full recovery.
Using a similar method, but with a larger cohort and wider range of injuries, the Prospective Outcomes of Injury Study (POIS) underway in New Zealand provides an opportunity to investigate the validity of using retrospective evaluation or population norms as proxies for pre-onset health valuation using the EQ-5D. This article reports the results of this analysis and discusses the implications for reporting changes in health-related quality of life (HRQoL) following illness or injury.
The Accident Compensation Corporation (ACC) provides universal no-fault insurance for people injured in New Zealand. This article uses data from POIS, a prospective cohort study of individuals, aged between 18 and 64 years, recruited from the ACC entitlement claims register between December 2007 and June 2009 [9, 10]. Participants included patients with all injury types, except those whose injuries were a result of self-harm or sexual assault, and covered a wide range of injury severities. Participants (n = 2856) completed a first interview 3.2 months (on average) after injury, with follow-up interviews approximately 5 months (average of 4.6) and 12 (12.3) months after injury. The POIS study received ethical approval from the New Zealand Health and Disability Multi-region Ethics Committee (MEC/07/07/093).
Two components of the POIS interview were used to identify “fully recovered” participants. First, participants were asked at the start of the 5-month and 12-month interviews whether they had completely recovered or were still affected by their injury. Second, they were assessed on the 12-item World Health Organization Disability Assessment Schedule (WHODAS 2.0), an instrument developed by the World Health Organization (WHO) to measure disability . The WHODAS rates participants’ difficulty completing a set of 12 activities over the previous 30 days, with five responses options from “None” to “Extreme/Cannot Do.” In all three interviews, participants were asked to complete the WHODAS instrument for their current health, and, in the first (3-month) interview they were asked to complete the WHODAS for the 30 days preceding their injury. We defined participants as having recovered from injury at the 5-month and 12-month interviews if they, in effect, passed both of the tests discussed above – i.e. reported having “completely recovered” and having attained at least their pre-injury functioning on all 12 WHODAS items.
At the first interview, participants were asked to complete the EQ-5D with respect to their pre-injury health status. They also did the same with respect to their (current) health status at 5- and 12-months after injury. The New Zealand EQ-5D valuation set  was used to convert participants’ health profiles on five dimensions to values on a scale from 0 (death) to 1 (perfect health), with negative values for states considered to be worse than dead.
To test the validity of participants’ retrospective evaluations of their health status, we compared their recalled pre-injury health with their reported health at 5- and 12-months after injury. These comparisons were performed for two groups: participants who reported having fully recovered, and others who reported having not. We hypothesised that for fully-recovered participants if their recalled pre-injury health status is unbiased then it would be the same as their post-injury status.
We also compared participants’ health valuations with population norms from the survey of the New Zealand general population undertaken in 1999 from which the above-mentioned EQ-5D valuation set was derived . Respondents described their own EQ-5D health status, and preference values were then applied from the valuation set. Population norms were calculated as the age- and sex-adjusted average of respondents’ valuations. We hypothesised that if population norms are a valid proxy for the pre-injury health of injured patients then population norms would approximate the health status of POIS participants who reported having fully recovered.
As represented on the EQ-5D’s five dimensions, 2842 POIS participants recalled their pre-injury health status at the first interview, and 1475 and 2262 respectively also reported their current health status at the 5- and 12- month post-injury interviews. Fewer participants completed the second (5-month) interview, as many had completed the first interview at the time the second interview was scheduled for, due to unanticipated delays in recruitment and interviewing . Via the survey of the general population, 1250 respondents described their health status on the EQ-5D, of whom 964 were aged 18–64 and identified their sex, allowing for construction of adjusted population norms.
Table 1 describes the age, sex, recovery status, health status, and injury types of POIS participants and respondents to the general population survey. POIS participants were on average younger and more likely to be male than those in the general population survey.
Recovery status was reported by 1248 participants at the 5-month interview, of whom 287 (23%) were fully recovered, and by 1937 participants at the 12-month interview, of whom 706 (36%) were fully recovered. We used two measures of recovery – i.e. participants’ reports of having “completely recovered” and having attained at least the same WHODAS level – as each measure on its own may miss some important aspects of recovery. For example, at the 12-month interview, 970 participants reported that they had “completely recovered” and 1025 reported they had attained at least the same level of functioning as pre-injury on all WHODAS dimensions, but only 706 passed both these tests.
The most common injury types were: spine dislocation, sprain, or strain; upper extremity fracture; upper extremity dislocation, sprain, or strain; lower extremity fracture; and lower extremity dislocation, sprain, or strain. Note that the injury data record more than one injury type for many participants, so the percentages do not add to 100.
The mean (unadjusted) EQ-5D health state value for the general population was 0.82. For the POIS cohort, their mean recalled pre-injury value was 0.94, falling to 0.75 and 0.78 5 and 12 months after injury respectively, where all three estimates are statistically significantly different than the general population mean (p < 0.001).
Pre- and post-injury health status
If recalled pre-injury health valuation is unbiased, we would expect that: (1) pre-injury health state values are statistically the same as post-injury values for fully recovered participants, and (2) pre-injury health state values are significantly higher than post-injury values for non-recovered participants. We found a small but statistically significant positive difference for participants who had fully recovered, and a large positive difference for participants who had not fully recovered (Table 2). These differences were consistent when measuring recovery at both 5-months and 12-months post-injury.
Health status of POIS participants and the general population
To test the validity of using population norms as a proxy for pre-injury health, we compared age- and sex-adjusted population norms with POIS participants’ health status before and after injury, by recovery status. Both the recovered and non-recovered groups had significantly better recalled pre-injury health than the corresponding New Zealand norm (Table 3 – upper panel). Participants who had fully recovered also reported significantly higher post-injury health than the general population, while the non-recovered reported significantly lower health values than the general population (Table 3 – lower panel).
Our results show that both retrospectively measured pre-injury health status and population norms differ from the health status reported by participants who had fully recovered from injury. The difference is greater between population norms and recovered health status than between recalled pre-injury status and recovered status. These findings are consistent with patients’ recall of their health status prior to injury exhibiting a small upward bias, and the general population being unrepresentative of those who are injured.
These results contrast with those of Watson and colleagues , who found that for completely recovered patients their retrospectively measured pre-injury health state values closely matched their values 12-months after injury using the SF-6D, although they found a marginally significant difference using the AQoL. A likely explanation for this difference is the increased statistical power available in our tests due to the larger sample size of the POIS cohort (1937 reporting recovery status at the 12-month interview compared to 186 for Watson et al.). However, there is also a possibility that recall bias may have been more pronounced in our study due to our 3-month delay between the injury event and measurement of recalled pre-injury health status.
Several studies have found that the general population may not be representative of populations of ill or injured individuals in terms of pre-onset health status [13–15]. Some populations, such as people with hip fracture , are in poorer health than the general population prior to their injury, whereas others, for example gunshot victims , are in better health. This article’s findings support the view that those who are injured are generally healthier than the general population.
An alternative explanation for the observed difference between self-reported health and the general population norm is ‘response shift’. This is the theory that individuals’ reference points for health status valuations change as their health changes . Study participants, having had experience with a poorer health state due to injury, may tend to inflate their assessments of both their pre-injury and recovered health states by implicit comparison with their injured state. Without the opportunity to undertake prospective evaluations of pre-injury health, it is not possible to test for the presence of response shift. Schwartz and Sprangers  argue that the existence of response shift implies that the use of recalled health status is more appropriate than prospective measurement for evaluating the impact of health state changes on HRQoL, as recalled and current status are both completed with the same internal standard of measurement (i.e. experience with the new health state). This argument also implies that recalled pre-injury evaluation should be used instead of population norms to assess changes in health status, as the general population has not had the same injury experience as the study population.
Implicit theories of memory may help to explain the small bias found in recalled health status . One important implicit theory in this context focuses on the stability of perceptions of self. Though people generally assume consistency in their personal attributes, significant events – such as injury – can provide a context in which recall is altered. Without a suitable reference point with which to recall their pre-injury health, people may begin by assessing their current health status and then adjusting that status for the expected change due to injury. If people tend to overestimate the change caused by injury, retrospective evaluation will be biased upward compared to actual pre-injury health. Our results provide some evidence in support of this theory, although the estimated effect is small.
Retrospective evaluation of pre-onset health status is likely to be more appropriate than applying population norms to measure the effects of acute-onset illness or injury on HRQoL, although users of this approach should be aware of the potential for a small upward bias in such measurements.
Potential Outcomes of Injury Study
Short Form 6-dimension health status instrument
EuroQol 5-dimension health status instrument
Assessment of Quality of Life health status instrument
Quality-adjusted life year
National Institute for Health and Clinical Excellence
Patient Reported Outcomes
Health-related quality of life
Accident Compensation Corporation
World Health Organization
- WHODAS 2.0:
World Health Organization Disability Assessment Schedule.
Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL: Methods for the Economic Evaluation of Health Care Programmes. 3rd edition. Oxford: Oxford University Press; 2005.
Brooks R: EuroQol: the current state of play. Health Policy 1996, 37: 53–72. 10.1016/0168-8510(96)00822-6
Szende A, Williams A: Measuring Self-Reported Population Health: An International Perspective Based on EQ-5D. Hungary: SpringMed Publishing Ltd; 2004.
Sørensen J, Davidsen M, Gudex C, Pedersen KM, Brønnum-Hansen H: Danish EQ-5D population norms. Scand J Public Health 2009, 37: 467–474. 10.1177/1403494809105286
National Institute for Health and Clinical Excellence: Guide to the methods of technology appraisal. London; 2008.
Department of Health: Guidance on the routine collection of Patient Reported Outcome Measures (PROMs). London; 2008.
NHS – The Information Centre: HESOnline Hospital Episode Statistics: Patient Reported Outcomes Measures (PROMs) Monthly Summary (April 2009 - January 2011). [http://www.hesonline.nhs.uk/Ease/servlet/ContentServer?siteID=1937&categoryID=1488]
Watson W, Ozanne-Smith J, Richardson J: Retrospective baseline measurement of self-reported health status and health-related quality of life versus population norms in the evaluation of post-injury losses. Inj Prev 2007, 13: 45–50. 10.1136/ip.2005.010157
Derrett S, Langley J, Hokowhitu B, Ameratunga S, Hansen P, Davie G, Wyeth E, Lilley R: Prospective outcomes of injury study. Inj Prev 2009, 15: e3. 10.1136/ip.2009.022558
Derrett S, Davie G, Ameratunga S, Wyeth E, Colhoun S, Wilson S, Samaranayaka A, Lilley R, Hokowhitu B, Hansen P, Langley J: Prospective Outcomes of Injury Study: recruitment, and participant characteristics, health and disability status. Inj Prev 2011, 17: 415–418. 10.1136/injuryprev-2011-040044
Üstün TB, Kostanjsek N, Chatterji S, Rehm J (Eds): Measuring Health and Disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0). Geneva, Switzerland: World Health Organization; 2010.
Devlin NJ, Hansen P, Kind P, Williams A: Logical inconsistencies in survey respondents' health state valuations – a methodological challenge for estimating social tariffs. Health Econ 2003, 12: 529–544. 10.1002/hec.741
Cameron C, Purdie D, Kliewer E, McClure R: Differences in prevalence of pre-existing morbidity between injured and non-injured populations. Bull World Health Organ 2005, 83: 345–352.
Patterson DR, Finch CP, Wiechman SA, Bonsack R, Gibran N, Heimbach D: Premorbid mental health status of adult burn patients: comparison with a normative sample. J Burn Care Rehabil 2003, 24: 347–350. 10.1097/01.BCR.0000086070.91033.7F
Gabbe BJ, Cameron PA, Graves SE, Williamson OD, Edwards ER: Preinjury status: are orthopaedic trauma patients different than the general population? J Orthop Trauma 2007, 21: 223–228. 10.1097/BOT.0b013e31803eb13c
Leibson CL, Tosteson ANA, Gabriel SE, Ransom JE, Melton LJ: Mortality, disability, and nursing home use for persons with and without hip fracture: a population-based study. J Am Geriatr Soc 2002, 50: 1644–1650. 10.1046/j.1532-5415.2002.50455.x
Greenspan AI, Kellermann AL: Physical and psychological outcomes 8 months after serious gunshot injury. J Trauma 2002, 53: 709–716. 10.1097/00005373-200210000-00015
Norman G: Hi! How are you? Response shift, implicit theories and differing epistemologies. Qual Life Res 2003, 12: 239–249. 10.1023/A:1023211129926
Schwartz CE, Sprangers MAG: Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Soc Sci Med 1999, 48: 1531–1548. 10.1016/S0277-9536(99)00047-7
Ross M: Relation of Implicit Theories to the Construction of Personal Histories. Psychol Rev 1989, 96: 341–357.
We are grateful to participants in the Potential Outcomes of Injury Study (POIS) for sharing their information with us and to James Black for his helpful comments on an earlier version of the article. Thank you also to two referees and the editors of the journal. The study was funded by the Health Research Council of New Zealand (2007–2013) and co-funded by the Accident Compensation Corporation, New Zealand (2007–2010). The views and conclusions in the article are the authors’ and may not reflect those of the funders.
All authors declare that they have no competing interests that may be relevant to the submitted work.
RW led the preparation of this article with support from SD, and conducted the statistical analysis. SD leads the POIS research team. PH and JL are POIS co-investigators. All authors contributed to the writing and editing of the manuscript. RW is guarantor. All authors read and approved the final manuscript.
About this article
Cite this article
Wilson, R., Derrett, S., Hansen, P. et al. Retrospective evaluation versus population norms for the measurement of baseline health status. Health Qual Life Outcomes 10, 68 (2012). https://doi.org/10.1186/1477-7525-10-68