Skip to main content
Download PDF
Download ePub

The impact of patient-reported outcome (PRO) data from clinical trials: a systematic review and critical analysis

Health and Quality of Life Outcomes volume 17, Article number: 156 (2019) Cite this article

Abstract

Background

Patient-reported outcomes (PROs) are commonly collected in clinical trials and should provide impactful evidence on the effect of interventions on patient symptoms and quality of life. However, it is unclear how PRO impact is currently realised in practice. In addition, the different types of impact associated with PRO trial results, their barriers and facilitators, and appropriate impact metrics are not well defined. Therefore, our objectives were: i) to determine the range of potential impacts from PRO clinical trial data, ii) identify potential PRO impact metrics and iii) identify barriers/facilitators to maximising PRO impact; and iv) to examine real-world evidence of PRO trial data impact based on Research Excellence Framework (REF) impact case studies.

Methods

Two independent investigators searched MEDLINE, EMBASE, CINAHL+, HMIC databases from inception until December 2018. Articles were eligible if they discussed research impact in the context of PRO clinical trial data. In addition, the REF 2014 database was systematically searched. REF impact case studies were included if they incorporated PRO data in a clinical trial.

Results

Thirty-nine publications of eleven thousand four hundred eighty screened met the inclusion criteria. Nine types of PRO trial impact were identified; the most frequent of which centred around PRO data informing clinical decision-making. The included publications identified several barriers and facilitators around PRO trial design, conduct, analysis and report that can hinder or promote the impact of PRO trial data. Sixty-nine out of two hundred nine screened REF 2014 case studies were included. 12 (17%) REF case studies led to demonstrable impact including changes to international guidelines; national guidelines; influencing cost-effectiveness analysis; and influencing drug approvals.

Conclusions

PRO trial data may potentially lead to a range of benefits for patients and society, which can be measured through appropriate impact metrics. However, in practice there is relatively limited evidence demonstrating directly attributable and indirect real world PRO-related research impact. In part, this is due to the wider challenges of measuring the impact of research and PRO-specific issues around design, conduct, analysis and reporting. Adherence to guidelines and multi-stakeholder collaboration is essential to maximise the use of PRO trial data, facilitate impact and minimise research waste.

Trial registration

Systematic Review registration PROSPERO CRD42017067799.

Introduction

Patient-reported outcomes (PROs) are increasingly used in clinical trials to assess the impact of a medical treatment or intervention. PRO assess a range of outcomes including symptoms, functional health, well-being and psychological issues from the patients’ perspective, without interpretation by a clinician [1,2,3]. Between 2007 and 2013, 26,337 (27%) of the clinical trials registered on ClinicalTrials.gov included PROs [4]. However, there is growing evidence that there is substantial research waste in relation to PROs [5, 6]. A recent systematic evaluation of oncology clinical trials determined that PRO protocol items were frequently omitted, non-reporting of PRO trials results was common and PRO publications were considerable delayed and presented suboptimal standards of reporting [5]. Thus, important PRO evidence may not be available to benefit patients and society.

Assessing the impact of research is a complex activity; however, it is important that the impact of PRO data is understood as this may inform funding allocations and demonstrate accountability to government, stakeholders and society [7]. Impact is defined as “any identifiable benefit to, or positive influence on, the economy, society, public policy or services, culture, the environment or quality of life …” (p.26) [8]. A number of reviews describe potential pathways (i.e. “a way of achieving a specified result; a course of action’ [9]) to general research impact [8, 10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33]. However, few studies have investigated the optimal pathway or methods for augmenting or evaluating specific impacts of PRO trial data or the extent to which PRO impact is being realised. It is also not clear which is the most appropriate way to measure PRO impact, or the barriers and facilitators to realising that impact.

One way of assessing real-world impact is via the United Kingdom (UK) Higher Education Funding Council for England Research Excellence Framework (REF) impact case studies. During the REF 2014 exercise, UK higher education institutions submitted impact case studies: narratives that described the impact of research conducted during a specific time-period, including a number of case studies describing clinical trials involving PROs. REF case studies present meaningful, far-reaching and properly articulated impact that is demonstrated through convincing evidence. The impact presented focuses on the benefits of the research rather than on the pathways of research impact, allowing the assessment of real-world impact on society [34]. Examination of these case studies can enhance understanding of the best methods for maximising and measuring PRO research impact, not only in the UK, but also internationally since a number of the studies described in REF are international studies [10, 35].

Therefore, the study had four objectives. First, to conduct a systematic review of the literature to: i) determine the range of potential impact that may arise from clinical trial PRO clinical trial data, ii) identify potential PRO impact metrics iii) identify barriers/facilitators to maximising PRO impact and; iv) to examine REF 2014 impact case studies to explore real-world evidence of PRO trial data impact.

Methods

This systematic review was registered on the PROSPERO database (CRD42017067799) and results are reported in accordance with PRISMA guidelines [36].

Search strategy

Systematic review

Two reviewers (SCR and OLA) systematically and independently searched MEDLINE (Ovid), EMBASE, HMIC and CINAHL+ databases (inception to December 2018) for articles discussing the impact of PRO data collected from clinical trials from inception to December 2018 (see Additional file 1 for the full search strategy). The authors (SCR/MC/DK) designed the search strategy with input from a University of Birmingham Information Specialist. In addition, the keywords ‘patient reported outcome measure*’, ‘PROs’, ‘PRO’, ‘PROM’, ‘PROMS’, ‘HRQOL’, ‘HRQL’, ‘quality of life’, ‘impact’ and ‘clinical trial*’ were searched on Google Scholar, where the initial 100 results were screened. Only the first 100 results (10 pages) were revised, as article relevance diminishes with each page of results [37]. Lastly, additional publications (n = 3) were sought through communication with methodological PRO experts facilitated by MC/DK. Hand-searching of reference lists and citation searches of the included publications was also conducted to identify additional relevant articles.

REF 2014 impact case studies

The keywords “trial*” and “quality of life” or “patient reported outcome*” were introduced in the REF 2014 database. The search strategy was restricted to: i) Unit of assessment: main panel A (see Table 1 for further detail), ii) Summary impact type: ‘health’ and iii) Research subject area: medical and health sciences.

Table 1 REF 2014 – Main panel A
Full size table

Eligibility criteria

Systematic review publications were deemed eligible if they discussed research impact in the context of PRO clinical trial data. In particular, we sought information on the types of impact (and pathways to impact) thought to be associated with PRO findings, proposed methods for measuring such impact and perceived barriers/facilitators to generating PRO-specific research impact. Publications were excluded if: i) solely focused on PROs used in routine clinical practice as the focus of this review was on the proposed PRO impact from trials; ii) trial publications reporting PRO results as the focus was research impact rather than primary results; or iii) conference abstracts. REF 2014 impact case studies were eligible if they included a trial in which PRO data were collected. There were no language restrictions.

Data screening

Systematic review

The screening process was conducted independently by two reviewers (SCR and OLA). Citations were downloaded into Endnote® software (version X7.3.1) and duplicates deleted. Records were screened by title and abstract. Potentially relevant articles were identified for further full-text screening (SCR and OLA). Discrepancies were resolved through discussion with a third reviewer (MC/DK/AS) if required.

REF 2014 impact case studies

The screening process was also conducted independently by SCR and OLA. The case studies were downloaded into a Microsoft Excel spreadsheet. Records were screened by title and summary of the impact. Relevant case studies were selected for further full-text screening (SCR and OLA). Discrepancies were resolved through discussion, with a third reviewer (MC/DK/AS) as necessary.

Data extraction/coding

Systematic review

Data extraction was done after the final selection of the included articles. SCR and OLA independently identified text excerpts that provided information on PRO-specific impact types, pathways, metrics, barriers or facilitators from the systematic review. Both reviewers independently imported text excerpts into a qualitative data analysis software package (QRS NVivo 11). They generated categories independently using descriptive coding under the directed content analysis framework [38]. The ‘pathways to research impact’ framework [10] was deductively applied to the data in order to identify types of impact and impact metrics. Data which did not fit within the existing framework were added to a ‘miscellaneous’ category. ‘Influence on policy-making’ was the only impact category discussed by the articles included in the systematic review. Subsequently, the data coded into this impact category was organised into subgroups. Through deductive coding, the following types of impact were identified: ‘inform clinical practice’, ‘inform clinical guidelines’, ‘inform clinical decision-making’, ‘inform health policy’ and ‘inform shared decision-making’.

Inductive coding was undertaken to describe and interpret more detailed codes within the ‘influence on policy-making’ and miscellaneous categories. The following types of impact were identified through inductive coding: ‘support drug approval’, ‘support pricing decisions’, ‘support reimbursement decisions’ and ‘inform consent for treatment’. In addition, inductive coding was used to identify further impact metrics, and barriers and facilitators to PRO trial impact. Development of overarching themes occurred after the coding process and collation of codes. The following details were also extracted from all the included publications: author, publication year, journal, methodology, study focus and type of PRO data impact.

REF 2014 impact case studies

Deductive and inductive was also undertaken to identify types of impact, impact metrics and barriers and facilitators among the REF case studies. In addition, the following details were extracted from the REF 2014 case studies: name, submitting institution and clinical area; trial name and year of publication, trial design, leading study centre, trial phase, trial primary and secondary outcomes, PRO instrument, significance of primary and secondary trial outcomes and type of impact. Furthermore, type of impact was further classified as either: i) direct PRO impact, where there was evidence of a direct link between PRO trial findings and subsequent impact. ii) Indirect PRO impact, where a trial including PROs subsequently led to impact, but it was not possible to directly attribute this impact to the PRO findings over and above the other trial outcomes; or iii) no evidence of PRO impact, where a trial including PROs failed to lead to impact. SCR and OLA independently piloted the coding frames, following discussion with MC/DK/AS/CM to resolve discrepancies. Finally, systematic review and impact case studies coding frames were validated by the co-authors MC/DK/AS/CM, who possess expertise in PRO clinical trial data, research impact and qualitative data analysis.

Results

Systematic review

Included studies

The search strategy retrieved 11,377 citations from MEDLINE (Ovid), EMBASE, HMIC, and CINAHL+; 100 citations were returned using Google Scholar and 6 through expert communication (PRISMA flow diagram, Additional file 2). Eight thousand eight hundred seventy-seven citations were excluded following review of title and abstract. In total, 32 full-text publications were assessed. Sixteen articles were excluded at this stage, as they assessed PRO data in routine care as an intervention. An additional 23 articles were included following hand-searching of reference lists and citation searches. In total, 39 eligible publications were included in the synthesis.

Study characteristics

The characteristics of the 39 included publications are summarised in Appendix 1. Fifteen (38%) publications were classified as systematic reviews, 11 as literature reviews (28%), eight (20%) as commentaries, three as qualitative studies and two as guidance papers. Non-English publications were identified through the different bibliographic methods used.

PRO impact types and pathways to impact

The included publications identified nine types of impact that authors proposed could be associated with PRO trial findings. These included; ‘informing clinical practice’, ‘informing clinical guidelines’, ‘informing health policy’, ‘supporting drug approval’, ‘supporting pricing decisions’ and ‘supporting reimbursement decisions’, ‘informing clinical decision-making’ and ‘informing shared decision-making’ and ‘informing consent for treatment’ (Fig. 1).

Fig. 1
figure 1

Proposed PRO impact types

Full size image

The majority of publications (69%) focused on the potential impact of PRO trial results on clinical decision-making [7, 39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67]. Clinical decision-making refers to the clinical evidence-based decisions made regarding individual patient care by the clinicians whilst considering clinician’s knowledge, skills and attitudes, resources available and the patient’s own concerns, values and preferences [68]. 15% of the publications considered the benefits of using PRO trial results to inform clinical practice [53, 56, 62,63,64]. Clinical practice refers to the healthcare services provided to patients at an organisational level, which are normally adopted following clinical practice guidelines [69].

One paper focused on the impact of PRO trial results on clinical guideline development [46] and one paper focused on PRO data use in the development of healthcare policy [7]. Several publications recognised the impact of PRO trial findings on drug approval (29%) [49, 66, 70,71,72,73,74,75,76], pricing (7%) [61, 72, 77] and reimbursement decisions (20%) [7, 65, 66, 72, 75, 77, 78]. Eight publications (20%) discussed the influence of PRO data on pharmaceutical labelling claims by the Food and Drug Administration (FDA) and European Medicines Agency (EMA) [49, 66, 70,71,72,73,74,75,76]. An example given by one author was ruxolitinib (Jakafi™): an oral inhibitor to treat intermediate or high risk patients with myelofibrosis. This was the first FDA approved oncology drug that used PROs as an endpoint and followed FDA guidance to support a PRO-based labelling claim [78]. The oncology drug was approved based on reduction in spleen volume and improvement in symptom severity (e.g. weight loss, night sweats, itching, abdominal pain/discomfort, bone pain, cough, inactivity, early satiety and fever), as measured by the Myelofibrosis Symptom Assessment Form (MFSAF) v2.0 Total Symptom Score.

Lastly, one paper discussed the impact of PRO data on shared decision-making [79] and two papers (2%) on informing consent for treatment [44, 54]. The types of potential PRO impact proposed in each publication are summarised in Fig. 1 and described in Appendix 2. Shared decision-making is also important and four publications outlined the potential benefits of including PRO findings alongside other outcomes such as survival. This allowed patients and their clinicians to make an informed joint decision about treatment preferences and symptom management based on mutual understanding of treatment objectives and expectations [1,2,3,4].

PRO impact metrics

Based on the review of the 39 included publications, two impact metrics were identified. Number of pharmaceutical labelling claims approved, mentioned by eight authors [70,71,72,73,74,75,76, 80] and number of promotional labelling claims, discussed by one author [70].

Barriers to impact

Authors identified different perceived methodological barriers to PRO trial impact, which fell within the following categories: ‘trial design’, ‘conduct and analysis’, ‘reporting’ and the ‘use of PRO data in practice’.

Trial design

Suboptimal PRO-specific trial design was cited by a large number (n = 22, 56%) of publications as a major barrier to realising PRO trial impact [40,41,42,43, 45,46,47, 49, 51, 52, 55, 62,63,64, 70, 72, 73, 76, 78, 81,82,83]. Particular areas of concern were selection of inappropriate or invalid PRO measures (n = 10, 52%). Lack of development of PRO cross-cultural items (n = 2, 9%). Broader methodological issues leading to potential bias and which may hinder PRO data use include allocation concealment, randomisation, blinding of participants and personnel and blinding of outcomes assessment (n = 7, 18%).

Conduct and analysis

Authors also highlighted that the way the trial was conducted and type of analysis carried out could act as an impact barrier. Over a third of publications (n = 12, 31%) identified PRO-specific barriers associated with trial conduct and analysis [40, 41, 56, 57, 62, 63, 70, 72, 73, 75, 78, 83]. The most frequent barriers mentioned by authors were low PRO compliance rates (n = 10, 83%), lack of personnel training on administration of PRO instruments (n = 3, 25%), incomplete follow-up of HRQL assessment (n = 2, 16%), selection of inappropriate statistical methods to handle missing data (n = 2, 16%).

Reporting

Incomplete or suboptimal reporting of PRO trial data was cited by 23 (65%) publications as a barrier to research impact if PRO trial findings and generalisability is not clearly presented [7, 38,39,40,41,42,43, 45,46,47,48,49,50,51,52,53, 56, 57, 63, 64, 72, 82, 83]. The most common barriers to impact were failure to report: the rationale for the chosen PRO instrument (n = 10, 43%), mode of administration (n = 10, 43%).

Use of PRO data in practice

Adoption of PRO trial findings into clinical practice was identified as somewhat problematic by 17 (43%) of the publications [39,40,41, 43, 45, 46, 52, 54,55,56, 61, 63, 64, 75, 78, 81, 82]. Key issues included lack of training/practice for clinicians on interpreting PRO data (n = 11, 64%) and lack of familiarity with PRO measures (n = 7, 41%).

Facilitators to impact

The review of the included articles identified a number of suggested facilitators purported to enhance the probability of realising PRO specific impact. Two (5%) authors suggested strict adherence to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) initiative to improve the completeness of trial protocols and reduce risk of bias [51, 62]. Eight (18%) authors proposed adherence to the Consolidated Standards of Reporting Trials (CONSORT) PRO Extension statement [7, 51,52,53, 56, 62, 64, 83] and two (5%) authors to the CONSORT statement [7, 51], in order to enhance transparency and complete reporting of PRO clinical trials. A detailed summary of the barriers and facilitators highlighted by the included publication are presented in Table 2, which includes additional resources identified by communication with methodological PRO experts (MC/DK).

Table 2 Barriers and facilitators to maximising PRO trial data
Full size table

REF 2014 impact case studies

Examples of clinical trial PRO impact were explored using REF2014 case studies. These identified a range of impact metrics.

Included studies

The search strategy yielded 209 REF 2014 impact case studies (PRISMA Flow Diagram, Additional file 3). Case studies were excluded if they did not include a clinical trial or the clinical trials did not incorporate a PRO element, meaning 69 relevant case studies were subsequently included in the analysis.

PRO clinical trials characteristics

The characteristics of the PRO clinical trials included across the eligible REF 2014 case studies are detailed in Table 3.

Table 3 PRO clinical trials characteristics
Full size table

Full details of the included case studies are available in Additional file 4. The assessment of the PRO trial metrics was considered using the ‘pathways to research impact’ framework [10]. Following this, two new additional impact metrics were identified, cost-effectiveness and drug/device approval. The summary of the PRO impact metrics is depicted in Fig. 2.

Fig. 2
figure 2

PRO trial impact metrics. *Additional PRO impact metrics identified. **There was not direct mention of this impact metric within the REF 2014 case studies; however, ‘drug approval’ could embrace the concept of patents granted/licenses awarded and brought to the market. Impact metrics were drew upon the ‘pathways to research impact’ framework [6]

Full size image

Real-world evidence of PRO impact

Assessment of the 69 eligible case studies determined that (n = 12, 17%) appeared to lead to direct demonstrable PRO impact, (n = 12, 17%) showed evidence of indirect PRO impact and (n = 45, 66%) provided no evidence of PRO impact (Fig. 2). Trials that included PROs as primary outcome (50%) reported a larger number of trials leading to direct impact than those trials that had PROs as secondary outcome (83%).

Direct PRO impact

The most common types of direct PRO impact presented across the case studies included: number of publications (n = 12, 17%), citation rates (n = 12, 17%), changes to international guidelines (n = 5, 7%), contribution to national guidelines (n = 4, 6%), contribution to evidence of cost-effectiveness (n = 3, 4%) and informing drug approval (n = 2, 3%). In addition, several case studies demonstrated more than one type of impact.

Indirect PRO impact

The most common types of indirect PRO impact included: number of publications (n = 12, 17%), citation rates (n = 12, 17%), changes to national guidelines (n = 10, 14%), contribution to international guidelines (n = 9, 13%) and national practice (n = 9, 13%) and contribution to evidence presented in conferences, seminars and workshops (n = 5, 7%).

Absence of evidence around PRO impact

The assessment of the included case studies demonstrated that the impact of PRO trial data is not usually captured in the long-term, specifically under the impact categories health and health system impacts, health related and social impact and economic impact.

Discussion

This manuscript is the first to present a systematic review aimed at identifying the potential types of PRO trial impact alongside real-world evidence of impact. It will allow researchers, clinicians, funders and policy-makers to consider pathways to research impact before conducting PRO trial research and to identify metrics to assess impact prospectively. In the same way, it will allow PRO stakeholders to consider facilitators at the design, conduct, analysis and reporting stages whilst avoiding recurrent barriers to generating PRO-specific impact and minimising research waste. High quality clinical trials involving PROs may lead to benefits for patients and society.

Nine types of potential PRO trial impact were identified (Fig. 1): informing clinical practice, informing clinical guidelines, informing health policy, supporting drug approval, supporting pricing and supporting reimbursement decisions, informing clinical and shared decision-making and informing consent for treatment. Only four impact metrics were proposed to measure the impact of PRO data, number of pharmaceutical claims and promotional labelling claims and inform drug/device approval and cost-effectiveness. Further research to formalise PRO-specific impact metrics is required.

Authors suggested that potential barriers to the use of PRO trial findings to inform healthcare decision-making and patient care included poor quality trial design, conduct, analysis, reporting and uptake in practice [55]. Several of the barriers comprised within ‘uptake of PRO trial results in practice’ (Table 2) are not unique to PRO clinical trials. These challenges are also encountered in the implementation of PRO data collected in routine clinical practice to inform patient care or for audit/benchmarking purposes. For instance, ‘high levels of missing data’ ‘overburden of staff, clinicians, participants and resources’, ‘lack of training/guidance for clinicians on interpreting PRO data’ are challenges commonly faced in routine practice [84, 85]. Furthermore, it is important to note that many of these challenges are not unique to PRO data/trials. Greater efforts are required to improve outcome selection, collection and reporting in both in trials and routine care [86, 87].

Suboptimal reporting of PRO trial data was the most discussed barrier (65%), which might hinder the maximisation of PRO trial findings. Therefore, addressing poor and incomplete reporting is essential, as it is unethical to waste research funding, resources and patients’ efforts and time invested during the collection PRO trial data [5, 6]. In recent years, a number of methodological guidelines have been developed to address the different barriers highlighted by this systematic review. These include: the SPIRIT-PRO Extension to improve the completeness of trial protocols [3]; The ongoing work of the SISAQOL Consortium to standardise the analysis and interpretation of PRO and quality of life from oncology clinical trials [88]; CONSORT-PRO Extension to facilitate optimal reporting guidance of trials that include PROs as primary or secondary outcome [46] and; the work carried out by Snyder et al. (2017) to present PRO trial findings [89]. The adoption of these guidelines has the potential to improve the design, conduct, analysis and report of PRO trials thus ensuring that high-quality data that may benefit patients and society are obtained from trials. The uptake of these guidelines is currently being promoted through PROTEUS (Patient-Reported Outcomes Tools: Engaging Users & Stakeholders) Consortium, which is funded by the US Patient Centred Outcomes Research Institute (PCORI) [90].

The literature suggests that adherence to guidelines should be endorsed/mandated by journals/editors in order to ensure high quality PRO data through the trial design, implementation, analysis and reporting stages. In addition, the FDA [2] and EMA [91] provide guidance to sponsors on reporting of PRO instrument development, measurement properties, implementation, analysis, and interpretation used to support drug approval and pharmaceutical labelling claims in the United States and Europe, respectively. Additional facilitators identified to maximise the realisation of impact in practice were reporting of PRO results adequately within the main trial publication, whilst considering journals word restrictions [43, 52, 53, 56, 64] and clinicians receiving training/guidance on interpretation of PRO data [41, 43, 52, 63]. Thus, it is essential that funders, ethics committees, journal editors and trial researchers proactively work together to ensure that PRO studies follow optimal design, conduct and analysis and reporting.

Although the systematic review publications may not have included information on PRO trial data impact, we explored whether evidence of impact could be identified from REF 2014 impact case studies as by their nature, they present an opportunity for researchers to highlight the impacts of their research. Sixty-nine REF 2014 case studies included a trial where PRO data were collected. Of these, 24 (34%) presented evidence of PRO trial impact that was classified as direct or indirect impact. Direct attribution of impact to PRO trial data was possible in 12 trials, most commonly informing national and international clinical guidelines. A number of potential impact categories are currently unrealised or under reported. This could be attributed to the fact that some of the PRO trials associated to the case studies have been published in the last years, which limits demonstration of PRO trial impact in the mid and long-term [10]. Furthermore, it was often difficult to unpick the exact contribution of PRO data to this impact as they were commonly combined with ‘clinical’ outcome data.

The REF 2014 case study ‘Heart failure: Improving the quality of life and survival of heart failure patients through Cardiac Resynchronisation Therapy’, submitted independently by the University of Birmingham [92] and Hull [93] is described below (Table 4) to illustrate the different facilitators that might help translating PRO findings into clinical practice. This example was chosen, as it is one of the case studies that have led to most varied impact and will provide researchers a useful guide about how to maximise PRO trial data and reduce research waste (Fig. 3).

Table 4 Practical guide for researchers
Full size table
Fig. 3
figure 3

CARE-HF Trial Pathways to PRO Trial Impact. Logos reproduced with permission of ESC and ACC

Full size image

However, reported barriers and facilitators in the literature focused predominantly on the PRO clinical trial design, conduct and analysis stages. There was a dearth of information on how to address barriers to generating PRO trial impact in practice. Additionally, identified impact was mainly focused on primary research (e.g. publications, citations and conference). There was little attention on policy-making, health & health systems, health related & societal and economic impact, which is generally realised in the mid and long term. Thus, further research in this area is required to identify facilitators to maximise PRO trial impact in the longer term. This will be achieved through interviews with international stakeholders in order to explore in-depth perceived barriers and facilitators to effective dissemination and impact on healthcare decisions and patient care. Work will be conducted to refine the ‘pathways to research impact’ framework in the context of PRO trial impact. In addition, it is important to mention that it is well established that certain impact categories (e.g. primary research impact via publications) are easier to measure than others (e.g. societal impact), which can limit the number of available impact metrics to measure the impact of PRO trial data.

Limitations

This systematic review summarised the different types of impact thought to be associated with PRO trial findings and proposes metrics to measure impact in practice. The main limitation was that due to poor indexing, over half of the included publications were identified through hand-searching of references lists and citation searches methods rather than databases searching. Therefore, some relevant publications might not be included in this article if they failed to mention a type of impact in the title/abstract. However, we made efforts to identify all the relevant publications. The search strategy adopted did not include the search term ‘self-rated health’, which could have led to the exclusion of relevant articles. Nonetheless, our search strategy was informed by the Oxford PROM Group Construct & Instrument Type Filter [94], which was modified according to the objectives of this systematic review. Although there were no language restrictions, we did not systematically search non-English databases. In addition, a formal quality appraisal was not undertaken to assess the quality of the studies included. We acknowledge that a significant amount of the evidence we found was based on expert opinion, which does not rank highly in the evidence hierarchy. Furthermore, a small number of the included studies were discussed by different authors, which may have influenced the frequency counts. However, this does not affect the conclusions of this systematic review.

The ‘pathways to research impact’ framework was used to measure the impact of PRO trial data. This framework was selected as it synthetises all the existing types of healthcare research impact and metrics (Fig. 2). However, not all the types of impact outlined by the framework are relevant to PRO trial data (e.g. human rights and United Nation Millennium Development Goals). The REF is an expert review process solely focused on the UK HEIs, which may limit the generalisability of the impact of the PRO data, although 30% of the trials were categorised as international trials. In some instances, it was not possible to confirm the impact described by the REF 2014 impact case studies, as there was no access to some sources provided. It is important to consider that the case studies had word count restrictions, which could have led to under reporting of impact. In addition, the majority of the articles included in the first section of the systematic review focused on oncology. Therefore, the findings presented in this study can only be generalised to oncology PRO clinical trials.

Conclusion

This review provides a summary of the different types of potential PRO impact identified in the literature, supported by real-world examples. The impact of PRO clinical trials can be attributed to PRO results and measured through different impact metrics. It is essential that researchers and authors design, conduct and analyse and report high quality PRO trial results and; proactively tackle barriers to PRO impact in order to maximise the impact of PRO clinical trials in the short, mid and long term to fully realise benefits for society. Adherence to guidance and multi-stakeholder collaboration is essential to maximise the utilisation of PRO trial data, while minimising research waste and maximising future patient care.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Abbreviations

CINALH+:

Cumulative Index to Nursing and Allied Health Literature

CONSORT:

Consolidated Standards of Reporting Trials

CONSORT-PRO Extension:

Consolidated Standards of Reporting Trials-Patient Reported Outcomes Extension

EMA:

European Medicine Agency

EMBASE:

Excerpta Medica Database

EORTC QLQ-C30:

European Organization for Research and Treatment - Core Quality of Life Questionnaire

EQ-5D:

European Quality of Life Instrument - 5 Dimension

FDA:

Food and Drug Administration

HADS:

Hospital Anxiety and Depression Scale

HEIs:

Higher Education Institutions

HMIC:

Health Management Information Consortium

MEDLINE:

Medical Literature Analysis and Retrieval System Online

MFSAF:

Myelofibrosis Symptom Assessment Form

PCORI:

Patient Centred Outcomes Research Institute

PRISMA:

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

PROs:

Patient-reported outcomes

PROSPERO:

The International Prospective Register of Systematic Reviews

PROTEUS:

Patient-Reported Outcomes Tools: Engaging Users & Stakeholders

REF:

Research Excellence Framework

SF-36:

Short Form Health Survey 36-item

SISAQOL:

The Setting International Standards in Analyzing Patient-Reported Outcomes and Quality of Life Endpoints Data

SPIRIT:

Standard Protocol Items: Recommendations for Interventional Trial

SPIRIT-PRO Extension:

Standard Protocol Items: Recommendations for Interventional Trial-Patient Reported Outcomes Extensions

UK:

United Kingdom

VAS:

Visual Analogue Scale

References

  1. Mitchell K. How do Patient-Reported measures contribute to value in health care? vol. 2017: Institute of Healthcare Improvement; 2014.

  2. FDA: Guidance for industry: Patient-Reported outcome measures: Use in medical product development to support labeling claims. 2009. https://www.fda.gov/media/77832/download. Accessed Feb 2018.

  3. Calvert M, Kyte D, Mercieca-Bebber R, et al. Guidelines for inclusion of patient-reported outcomes in clinical trial protocols: the spirit-pro extension. JAMA. 2018;319:483–94.

    Article  PubMed  Google Scholar 

  4. Vodicka E, Kim K, Devine E, Gnanasakthy A, Scoggins J, Patrick D. Inclusion of patient-reported outcome measures in registered clinical trials: evidence from ClinicalTrials.gov (2007–2013). Contemp Clin Trials. 2015;43:1–9.

    Article  CAS  PubMed  Google Scholar 

  5. Kyte D, Retzer A, Ahmed K, Keeley T, Armes J, Brown JM, Calman L, Gavin A, Glaser AW, Greenfield DM, et al. Systematic evaluation of Patient-Reported outcome protocol content and reporting in Cancer trials. J Natl Cancer Inst. 2019;111(11):1–9..

  6. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas. 2018;9:353–67.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Calvert M, Blazeby J, Revicki D, Moher D, Brundage M. Reporting quality of life in clinical trials: a CONSORT extension. Lancet. 2011;378:1684–5.

    Article  PubMed  Google Scholar 

  8. REF 2014: Assessment framework and guidance on submissions [https://www.ref.ac.uk/2014/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf]. Accessed Feb 2018.

  9. Oxford Dictionaries - pathway [http://www.oxforddictionaries.com/definition/english/pathway]. Accessed July 2019.

  10. Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ. Assessing the impact of healthcare research: a systematic review of methodological frameworks. PLoS Med. 2017;14:e1002370.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Royal Netherlands Academy of arts and sciences. The societal impact of applied health research - towards a quality assessment system [https://www.knaw.nl/en/news/publications/the-societal-impact-of-applied-health-research/@@download/pdf_file/20021098.pdf]. Accessed Feb 2018.

  12. RAE 2008 - Guidance in submissions [http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf]. Accessed Feb 2018.

  13. Morton S. Progressing research impact assessment: A ‘contributions’ approach. Res Eval. 2015;24:405–19.

    Article  Google Scholar 

  14. Primary Health Care Research Impact Project: Final Report Stage 1 [http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3338.pdf]. Accessed Feb 2018.

  15. Brueton VC, Vale CL, Choodari-Oskooei B, Jinks R, Tierney JF. Measuring the impact of methodological research: a framework and methods to identify evidence of impact. Trials. 2014;15:464 461p.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Weiss AP. Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatr. 2007;164:206–14.

    Article  PubMed  Google Scholar 

  17. Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8:165–70.

    Article  PubMed  Google Scholar 

  18. Canavan J, Gillen A, Shaw A. Measuring research impact : developing practical and cost-effective approaches. Evid Policy. 2009;5:167–77.

    Article  Google Scholar 

  19. Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, Kinsman L. ‘Making evidence count’: a framework to monitor the impact of health services research. Aust J Rural Health. 2012;20:51–8.

    Article  PubMed  Google Scholar 

  20. Spaapen J, van Drooge L. Introducing ‘productive interactions’ in social impact assessment. Res Eval. 2011;20:211–8.

    Article  Google Scholar 

  21. Guinea J, Sela E, Gómez-Núñez AJ, Mangwende T, Ambali A, Ngum N, Jaramillo H, Gallego JM, Patiño A, Latorre C, et al. Impact oriented monitoring: a new methodology for monitoring and evaluation of international public health research projects. Res Eval. 2015;24:131–45.

    Article  Google Scholar 

  22. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1:35–43.

    Article  CAS  PubMed  Google Scholar 

  23. Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Res Eval. 2008;17:163–73.

    Article  Google Scholar 

  24. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: development and implementation of the Alberta innovates – health solutions impact framework. Res Eval. 2012;21:354–67.

    Article  Google Scholar 

  25. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, Bauman AE, Redman S, Chapman S. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2014;13:3–3 1p.

    Article  Google Scholar 

  26. Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a research impact framework. BMC Health Serv Res. 2006;6:134.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10.

  28. Landry R, Amara N, Lamari M. Climbing the ladder of research utilization - evidence from social science research. Sci Commun. 2001;22:396–422.

    Article  Google Scholar 

  29. Canadian Institutes of Health Research. Developing a CIHR framework to measure the impact of health research [http://publications.gc.ca/collections/Collection/MR21-65-2005E.pdf]. Accessed Feb 2018.

  30. Canadian Academy of Health Sciences. Making an impact - A preferred framework and indicators to measure returns on investment in health research [http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf]. Accessed Feb 2018.

  31. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc. 2010;98:17–23.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Donovan C. The Australian research quality framework: a live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. N Dir Eval. 2008;118:47–60.

    Article  Google Scholar 

  33. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997;13:68–80.

    Article  CAS  PubMed  Google Scholar 

  34. Reed M. What makes a 4* research impact case study for REF2021? In: Fast Track Impact, vol. 2018; 2017.

    Google Scholar 

  35. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: a review. Res Eval. 2014;23:21–32.

    Article  Google Scholar 

  36. Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  37. iProspect. iProspect Search Engine User Behavior Study: iProspectcom, Inc; 2006.

  38. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88.

    Article  PubMed  Google Scholar 

  39. Efficace F, Bottomley A, Osoba D, Gotay C, Flechtner H, D'Haese S, Zurlo A. Beyond the development of health-related quality-of-life (HRQOL) measures: a checklist for evaluating HRQOL outcomes in cancer clinical trials -- does HRQOL evaluation in prostate cancer research inform clinical decision making? J Clin Oncol. 2003;21:3502–11.

    Article  PubMed  Google Scholar 

  40. Bjordal K. Impact of quality of life measurement in daily clinical practice. Ann Oncol. 2004;15:iv279–82.

    Article  PubMed  Google Scholar 

  41. Avery K, Blazeby JM. Quality of life assessment in surgical oncology trials. World J Surg. 2006;30:1163–72.

    Article  PubMed  Google Scholar 

  42. Efficace F, Kemmler G, Vignetti M, Mandelli F, Molica S, Holzner B. Health-related quality of life assessment and reported outcomes in leukaemia randomised controlled trials - a systematic review to evaluate the added value in supporting clinical decision making. Eur J Cancer. 2008;44:1497–506.

    Article  PubMed  Google Scholar 

  43. Gujral S, Avery KNL, Blazeby JM. Quality of life after surgery for colorectal cancer: clinical implications of results from randomised trials. Support Care Cancer. 2008;16:127–32.

    Article  PubMed  Google Scholar 

  44. Parameswaran R, McNair A, Avery KN, Berrisford RG, Wajed SA, Sprangers MA, Blazeby JM. The role of health-related quality of life outcomes in clinical decision making in surgery for esophageal cancer: a systematic review. Ann Surg Oncol. 2008;15:2372–9.

    Article  CAS  PubMed  Google Scholar 

  45. Lemieux J, Goodwin PJ, Bordeleau LJ, Lauzier S, Theberge V. Quality-of-life measurement in randomized clinical trials in breast cancer: an updated systematic review (2001-2009). J Natl Cancer Inst. 2011;103:178–231.

    Article  PubMed  Google Scholar 

  46. Calvert M, Brundage M, Jacobsen PB, Schunemann HJ, Efficace F. The CONSORT Patient-Reported outcome (PRO) extension: implications for clinical trials and practice. Health Qual Life Outcomes. 2013;11:184.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Jacobs M, Macefield RC, Blazeby JM, Korfage IJ, Henegouwen MIV, de Haes H, Smets EM, Sprangers MAG. Systematic review reveals limitations of studies evaluating health-related quality of life after potentially curative treatment for esophageal cancer. Qual Life Res. 2013;22:1787–803.

    Article  PubMed  Google Scholar 

  48. Dirven L, Taphoorn MJB, Reijneveld JC, Blazeby J, Jacobs M, Pusic A, La Sala E, Stupp R, Fayers P, Efficace F, Patient Reported O. The level of patient-reported outcome reporting in randomised controlled trials of brain tumour patients: a systematic review. Eur J Cancer. 2014;50:2432–48.

    Article  PubMed  Google Scholar 

  49. Efficace F, Feuerstein M, Fayers P, Cafaro V, Eastham J, Pusic A, Blazeby J. Patient-reported outcomes in randomised controlled trials of prostate cancer: methodological quality and impact on clinical decision making. Eur Urol. 2014;66:416–27.

    Article  PubMed  Google Scholar 

  50. Efficace F, Jacobs M, Pusic A, Greimel E, Piciocchi A, Kieffer JM, Gilbert A, Fayers P, Blazeby J, Registry EQoLGPROMOTIO-P. Patient-reported outcomes in randomised controlled trials of gynaecological cancers: investigating methodological quality and impact on clinical decision-making. Eur J Cancer. 2014;50:1925–41.

    Article  PubMed  Google Scholar 

  51. Rees JR, Whale K, Fish D, Fayers P, Cafaro V, Pusic A, Blazeby JM, Efficace F. Patient-reported outcomes in randomised controlled trials of colorectal cancer: an analysis determining the availability of robust data to inform clinical decision-making. J Cancer Res Clin Oncol. 2015;141:2181–92.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Rouette J, Blazeby J, King M, Calvert M, Peng Y, Meyer RM, Ringash J, Walker M, Brundage MD. Integrating health-related quality of life findings from randomized clinical trials into practice: an international study of oncologists' perspectives. Qual Life Res. 2015;24:1317–25.

    Article  PubMed  Google Scholar 

  53. McNair AGK, Macefield RC, Blencowe NS, Brookes ST, Blazeby JM. ‘Trial exegesis’: methods for synthesizing clinical and Patient Reported outcome (PRO) data in trials to inform clinical practice. A systematic review. PLoS One. 2016;11:e0160998.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Blazeby JM, Avery K, Sprangers M, Pikhart H, Fayers P, Donovan J. Health-related quality of life measurement in randomized clinical trials in surgical oncology. J Clin Oncol. 2006;24:3178–86.

    Article  PubMed  Google Scholar 

  55. Snyder C, Brundage M. Integrating patient-reported outcomes in healthcare policy, research and practice. Expert Rev Pharmacoecon Outcomes Res. 2010;10:351–3.

    Article  PubMed  Google Scholar 

  56. Calvert M, Brundage M, Jacobsen PB, Schünemann HJ, Efficace F. The CONSORT Patient-Reported outcome (PRO) extension: implications for clinical trials and practice. Health Qual Life Outcomes. 2013;11:184.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Bottomley A, Efficace F, Thomas R, Vanvoorden V, Ahmedzai SH. Health-related quality of life in non–small-cell lung Cancer: methodologic issues in randomized controlled trials. J Clin Oncol. 2003;21:2982–92.

    Article  PubMed  Google Scholar 

  58. Efficace F, Kemmler G, Vignetti M, Mandelli F, Molica S, Holzner B. Health-related quality of life assessment and reported outcomes in leukaemia randomised controlled trials – a systematic review to evaluate the added value in supporting clinical decision making. Eur J Cancer. 2008;44:1497–506.

    Article  PubMed  Google Scholar 

  59. Ganz PA. Assessing the quality and value of quality-of-life measurement in breast Cancer clinical trials. J Natl Cancer Inst. 2011;103:196–9.

    Article  PubMed  Google Scholar 

  60. Goodwin PJ, Black JT, Bordeleau LJ, Ganz PA. Health-related quality-of-life measurement in randomized clinical trials in breast Cancer—taking stock. J Natl Cancer Inst. 2003;95:263–81.

    Article  PubMed  Google Scholar 

  61. Hao Y, Krohe M, Yaworsky A, Shields AL, Mazar I, Foley C, Globe D. Clinical trial Patient-reported outcomes data: going beyond the label in oncology. Clin Ther. 2016;38:811–20.

    Article  PubMed  Google Scholar 

  62. Jacobs M, Macefield RC, Blazeby JM, Korfage IJ, van Berge Henegouwen MI, de Haes HCJM, Smets EM, Sprangers MAG. Systematic review reveals limitations of studies evaluating health-related quality of life after potentially curative treatment for esophageal cancer. Qual Life Res. 2013;22:1787–803.

    Article  PubMed  Google Scholar 

  63. McNair AG, Blazeby JM. Health-related quality-of-life assessment in GI cancer randomized trials: improving the impact on clinical practice. Expert Rev Pharmacoecon Outcomes Res. 2009;9:559–67.

    Article  PubMed  Google Scholar 

  64. Mercieca-Bebber RL, Perreca A, King M, Macann A, Whale K, Soldati S, Jacobs M, Efficace F. Patient-reported outcomes in head and neck and thyroid cancer randomised controlled trials: a systematic review of completeness of reporting and impact on interpretation. Eur J Cancer. 2016;56:144–61.

    Article  PubMed  Google Scholar 

  65. Mott FE. Patient Reported outcomes (PROs) as part of value-based care can shape therapy guidelines: impact on emerging targeted agents and immunotherapy protocols in resource-limited regions. Oncol Ther. 2017;5:69–74.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Basch E, Geoghegan C, Coons S, et al. Patient-reported outcomes in cancer drug development and us regulatory review: perspectives from industry, the food and drug administration, and the patient. JAMA Oncol. 2015;1:375–9.

    Article  PubMed  Google Scholar 

  67. Au H-J, Ringash J, Brundage M, Palmer M, Richardson H, Meyer RM. Added value of health-related quality of life measurement in cancer clinical trials: the experience of the NCIC CTG. Expert Rev Pharmacoecon Outcomes Res. 2010;10:119–28.

    Article  PubMed  Google Scholar 

  68. Glasziou PP, Mar CD, Salisbury J. Evidence-Based Practice Workbook: Wiley; 2009.

  69. Kristensen N, Nymann C, Konradsen H. Implementing research results in clinical practice- the experiences of healthcare professionals. BMC Health Serv Res. 2016;16:1–10.

    Google Scholar 

  70. Revicki DA, Osoba D, Fairclough D, Barofsky I, Berzon R, Leidy NK, Rothman M. Recommendations on health-related quality of life research to support labeling and promotional claims in the United States. Qual Life Res. 2000;9:887–900.

    Article  CAS  PubMed  Google Scholar 

  71. Patrick DL, Burke LB, Powers JH, Scott JA, Rock EP, Dawisha S, O'Neill R, Kennedy DL. Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health. 2007;10(Suppl 2):S125–37.

    Article  PubMed  Google Scholar 

  72. Doward LC, Gnanasakthy A, Baker MG. Patient reported outcomes: looking beyond the label claim. Health Qual Life Outcomes. 2010;8:89.

    Article  PubMed  PubMed Central  Google Scholar 

  73. DeMuro C, Clark M, Mordin M, Fehnel S, Copley-Merriman C, Gnanasakthy A. Reasons for rejection of Patient-Reported outcome label claims: a compilation based on a review of Patient-Reported outcome Use among new molecular entities and biologic license applications, 2006–2010. Value Health. 2012;15:443–8.

    Article  PubMed  Google Scholar 

  74. Coon CD. The Use of Patient-reported outcomes in demonstrating safety and efficacy in oncology. Clin Ther. 2016;38:756–8.

    Article  PubMed  Google Scholar 

  75. Anker SD, Agewall S, Borggrefe M, Calvert M, Jaime Caro J, Cowie MR, Ford I, Paty JA, Riley JP, Swedberg K, et al. The importance of patient-reported outcomes: a call for their comprehensive integration in cardiovascular clinical trials. Eur Heart J. 2014;35:2001–9.

    Article  PubMed  Google Scholar 

  76. Gnanasakthy A, DeMuro C, Clark M, Haydysch E, Ma E, Bonthapally V. Patient-Reported outcomes labeling for products approved by the Office of Hematology and Oncology Products of the US Food and Drug Administration (2010-2014). J Clin Oncol. 2016;34:1928–34.

    Article  CAS  PubMed  Google Scholar 

  77. Arpinelli F, Bamfi F. The FDA guidance for industry on PROs: the point of view of a pharmaceutical company. Health Qual Life Outcomes. 2006;4:85.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Zagadailov E, Fine M, Shields A. Patient-Reported outcomes are changing the landscape in oncology care: challenges and opportunities for payers. Am Health Drug Benefits. 2013;6:264–74.

    PubMed  PubMed Central  Google Scholar 

  79. Sztankay M, Giesinger JM, Zabernigg A, Krempler E, Pall G, Hilbe W, Burghuber O, Hochmair M, Rumpold G, Doering S, Holzner B. Clinical decision-making and health-related quality of life during first-line and maintenance therapy in patients with advanced non-small cell lung cancer (NSCLC): findings from a real-world setting. BMC Cancer. 2017;17:565.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Nixon AWD, Muehlhausen W. Patient reported outcomes to supoort drug development decision making. Farmeconomia Health Econ Ther Pathways. 2015;16:35–7.

    Article  Google Scholar 

  81. Rees J, Hurt CN, Gollins S, Mukherjee S, Maughan T, Falk SJ, Staffurth J, Ray R, Bashir N, Geh JI, et al. Patient-reported outcomes during and after definitive chemoradiotherapy for oesophageal cancer. Br J Cancer. 2015;113:603–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  82. Brundage M, Bass B, Davidson J, Queenan J, Bezjak A, Ringash J, Wilkinson A, Feldman-Stewart D. Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers. Qual Life Res. 2011;20:653–64.

    Article  PubMed  Google Scholar 

  83. Dirven L, Taphoorn MJB, Reijneveld JC, Blazeby J, Jacobs M, Pusic A, La Sala E, Stupp R, Fayers P, Efficace F. The level of patient-reported outcome reporting in randomised controlled trials of brain tumour patients: a systematic review. Eur J Cancer. 2014;50:2432–48.

    Article  PubMed  Google Scholar 

  84. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measure in routine clinical practice: lack of impact or lack of theory? Soc Sci Med. 2005;60:833–43.

    Article  PubMed  Google Scholar 

  85. Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, Revicki DA, Symonds T, Parada A, Alonso J. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17:179–93.

    Article  CAS  PubMed  Google Scholar 

  86. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, Clarke M, Gargon E, Gorst S, Harman N, et al. The COMET handbook: version 1.0. Trials. 2017;18:280.

    Article  PubMed  PubMed Central  Google Scholar 

  87. Calvert M, Kyte D, Price G, Valderas JM, Hjollund NH. Maximising the impact of patient reported outcome assessment for patients and society. BMJ. 2019;364:k5267.

    Article  PubMed  Google Scholar 

  88. Bottomley A, Pe M, Sloan J, Basch E, Bonnetain F, Calvert M, Campbell A, Cleeland C, Cocks K, Collette L, et al. Analysing data from patient-reported outcome and quality of life endpoints for cancer clinical trials: a start in setting international standards. Lancet Oncol. 2016;17:e510–4.

    Article  PubMed  Google Scholar 

  89. Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD, the PRODPSAB. What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123:1848–59.

    Article  PubMed  PubMed Central  Google Scholar 

  90. "PROTEUS" Patient-Reported outcomes tools: engaging users & stakeholders [https://www.pcori.org/research-results/2018/proteus-patient-reported-outcomes-tools-engaging-users-stakeholders]. Accessed June 2019.

  91. Use CfMPfH. Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products. London: European Medicines Agency; 2005.

    Google Scholar 

  92. REF 2014 impact case studies. 'Heart failure: improving the quality of life and survival of heart failure patients through cardiac resynchronisation therapy' [https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=38798]. Accessed Nov 2017.

  93. REF 2014 impact case studies, 'Improving well-being and outcome for patients with heart failure using cardiac resynchronisation therapy (CRT)' [https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=36388]. Accessed Nov 2017.

  94. University of Oxford - PROM Group Construct & Instrument Type Filters [https://cosmin.nl/wp-content/uploads/prom-search-filter-oxford-2010.pdf]. Accessed April 2017

Download references

Acknowledgements

We would like to thank Dr. Thomas Keeley, Scientist, Patient Centred Outcomes (Value Evidence and Outcomes) at GSK for his input whilst working at the University of Birmingham. Prof Richard Lehman, Professor of the Shared Understanding of Medicine, University of Birmingham, for providing guidance and feedback on the manuscript. We would also like to thank Mrs. Susan Bayliss, Information Specialist, University of Birmingham.

Funding

SCR is funded by Consejo Nacional de Ciencia y Tecnología (CONACyT).

Author information

Affiliations

  1. Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK

    Samantha Cruz Rivera, Derek G. Kyte, Olalekan Lee Aiyegbusi, Anita L. Slade, Christel McMullan & Melanie J. Calvert

  2. NIHR Birmingham Biomedical Research Centre, NIHR Surgical Reconstruction and Microbiology Research Centre University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, Birmingham, UK

    Derek G. Kyte, Anita L. Slade & Melanie J. Calvert

Authors
  1. Samantha Cruz Rivera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Derek G. Kyte
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olalekan Lee Aiyegbusi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anita L. Slade
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christel McMullan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Melanie J. Calvert
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SCR, MC and DK designed the study, SCR and OLA reviewed and interpreted the data, SCR drafted the manuscript. MC, DK, AS and CM reviewed analysis and substantively revised the manuscript. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Melanie J. Calvert.

Ethics declarations

Ethics approval and consent to participate

Not Applicable

Consent for publication

Not Applicable

Competing interests

MC, DK and AS are funded by the NIHR Birmingham Biomedical Research Centre and the NIHR Surgical Reconstruction and Microbiology Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. DK reports grants from Macmillan Cancer Support, Innovate UK, the National Institute for Health Research (NIHR) PDF-2016-09-009, NIHR Birmingham Biomedical Research Centre and NIHR Surgical Reconstruction and Microbiology Research Centre (SRMRC) at the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust and personal fees from Merck outside the submitted work. OLA is funded by the NIHR Birmingham Biomedical Research Centre and the Health Foundation’s Improvement Science PhD Awards Programme. The Health Foundation is an independent charity working to improve the quality of healthcare in the UK. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, the Department of Health or CONACyT. MC is a co-investigator within Health Data Research UK Midlands.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Search strategies

Additional file 2.

Systematic Review PRISMA Flow Diagram

Additional file 3.

REF 2014 Impact case studies PRISMA flow diagram

Additional file 4.

REF impact case studies

Appendices

Appendix 1

Table 5 Study characteristics of the literature review
Full size table

Appendix 2

Potential types of PRO impact proposed

Clinical practice, clinical guidelines and health policy

A number of authors discussed the potential influence of PRO data on clinical practice, clinical guidelines and health policy. Several authors felt inclusion of PRO data on clinical guidelines may influence clinical practice, by fulfilling unmet clinical needs and leading to improved patient centre care by helping patients make more informed decisions on their care [44, 48, 51, 58,59,60, 76]. Three studies suggested that the inclusion of PRO data in clinical guidelines might ensure wider acceptance of guideline recommendations among patients, while enhancing implementation through health policy [5, 51, 60].

Drug and device approval

Authors reported that PRO data is increasingly used to provide evidence for drug and device approval, especially in oncology clinical trials [67, 68]. Eight publications discussed the influence of PRO data on pharmaceutical labelling claims by the Food and Drug Administration (FDA) and European Medicines Agency (EMA) [44, 62, 64,65,66,67,68,69,70]. One publication suggested that PRO data may inform drug and device approval through communication of the benefits and harms of the intervention has to clinicians, patients and other consumers [64].

Pricing and reimbursement decisions

Three publications discussed the influence of PRO data on pricing decisions [57, 66, 71] and seven on drug reimbursement decisions [5, 57, 61, 66, 69, 71, 74]. Authors suggested that inclusion of PRO data provided valuable information regarding the risk-benefit of new interventions. Several authors postulated that within the context of oncology treatment, PRO data have the potential to identify costly events reported by the patients in the trials. Hence, interventions with reduced toxicity and added PRO benefit may influence payer decision-makers [72]. Additionally, the inclusion of patient advocacy groups may influence the availability of interventions, enhancing the patient healthcare experience by incorporating the ‘patient’s voice’ throughout payer decision-making [57, 66].

One such example illustrated in the literature, is the decision by NICE (2015) to recommend the use of nintedanib plus docetaxel (Vargatef®) as a treatment option for locally advanced, metastatic or locally recurrent non-small-cell lung cancer of adenocarcinoma histology. The drug approval was primarily based on improved survival, minimal adverse drug effects and fewer detrimental effects on health-related quality of life (HRQL) compared to chemotherapy treatment. PRO data suggested that HRQL, as measured by the EuroQol-5 Dimension (EQ-5D), the European Organization for Research and Treatment core quality of life questionnaire (EORTC QLC-C30) and the lung cancer–specific Quality of Life Questionnaire (EORTC QLQ-LC13) was similar in both groups [57, 96]. However, the PRO measures also demonstrated better pain management in patients randomised to nintedanib plus docetaxel based on information from the pain items [57, 96]. Therefore, additional drug benefits (symptom improvement and tolerability) were demonstrated through incorporation of PRO measurements into the trial, which similarly informed the reimbursement decision. The additional HRQL benefits may have been missed without the supporting PRO data within this trial.

Clinical decision-making

While reducing the impact of intervention toxicity and the impact on HRQL is important for drug labelling claims, it is also an important consideration for choices in clinical decision-making. According to Goodwin et al. (2003), when there is medical treatment equivalence, HRQL data has the potential to inform clinical decision-making by prioritising quality of life outcomes and reduction in toxicity when making clinical decisions [56]. The authors conducted a systematic review of breast cancer clinical trials including PROs and determined that HRQL contributed to clinical decision-making within primary management (surgery, radiation and hormone therapies) and symptom control/supportive care setting of breast cancer.

In total, 26 publications presented evidence that PRO data may help in the selection of optimal treatment, patient’s symptom experience and management, satisfaction with care and might predict prognosis, which has the potential to inform clinical decision-making based on the clinicians’ critical appraisal and interpretation of the available information [5, 34,35,36, 38,39,40, 44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61, 74].

Dirven et al. (2014) conducted a systematic review of PRO clinical trials in patient with brain tumours and demonstrated that HRQL can be used alongside overall and progression-free survival to inform clinical decision-making. One of the clinical trials included determined that the combination of concomitant and adjuvant temozolomide and radiotherapy has become standard care for newly diagnosed patients with glioblastoma [53]. This combination treatment led to significantly prolonged overall and progression-free survival, without negatively impacting HRQL in the long-term as measured with the EORTC QLQ-C30 questionnaire and Brain Cancer Module (BN-20) [53].

Shared decision-making

Shared decision-making is also important and four publications outlined the potential benefits of including PRO findings alongside other outcomes such as survival. This allowed patients and their clinicians to make an informed joint decision about treatment preferences and symptom management based on mutual understanding of treatment objectives and expectations [44, 58, 59, 73].

Sztankay et al. (2017) assessed HRQL during first-line chemotherapy with pametrexed and maintenance therapy (MT) among patients with advanced non-small cell lung cancer [73]. First-line chemotherapy for patients with advanced non-small cell lung cancer was shown to improve overall progression-free survival. However, as measured with the EORTC QLQ-C30 and EORTC QLQ-LC13, MT compared to first-line chemotherapy was associated with lower HRQL and improvements in nausea, vomiting, appetite loss, constipation and pain. This information presented alongside survival data, allowed patients and clinicians to make real-world informed joint decisions regarding treatment options.

Informed treatment consent

Consent for treatment refers to the authorisation given by a patient to receive a treatment, once the clinician presents a diagnosis, relevant treatment options and respective risks and benefits to the patient [97]. Two publications discussed the influence of PRO trial data on treatment consent [39, 49]. Parameswaran et al. (2008) presented a systematic review of two randomised controlled trials, 9 longitudinal studies and 11 cross-sectional studies. The authors determined that only 11 studies presented data that was capable of effectively informing patient consent. This statement was based on the assessment of the HRQL methodology of the studies, through the HRQL checklist by Efficace et al. (2003) [34]. For instance, as measured with the EORTC-QLQ-C30 and MOS-SF36, one of the eleven studies determined that surgery for oesophageal cancer patients has a detrimental impact on quality of life in the postoperative stage (e.g. anastomic leaks, sepsis and cardiac and pulmonary complications) and in some cases; quality of life among survival patients does not improve in the long term [39]. Therefore, communicating HRQL and clinical data to patients after the intervention could help inform patients about relevant information regarding recovery and outcomes before undergoing an intervention and inform the consent process.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rivera, S.C., Kyte, D.G., Aiyegbusi, O.L. et al. The impact of patient-reported outcome (PRO) data from clinical trials: a systematic review and critical analysis. Health Qual Life Outcomes 17, 156 (2019). https://doi.org/10.1186/s12955-019-1220-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12955-019-1220-z

Keywords

  • Patient-reported outcomes
  • Quality of life
  • Impact
  • REF case studies
  • Clinical trials

Health and Quality of Life Outcomes

ISSN: 1477-7525

Contact us