- Open Access
Incorporating development of a patient-reported outcome instrument in a clinical drug development program: examples from a heart failure program
© The Author(s). 2016
Received: 6 January 2016
Accepted: 6 September 2016
Published: 15 September 2016
Patient-reported outcome (PRO) measures can be used to support label claims if they adhere to US Food & Drug Administration guidance. The process of developing a new PRO measure is expensive and time-consuming. We report the results of qualitative studies to develop new PRO measures for use in clinical trials of omecamtiv mecarbil (a selective, small molecule activator of cardiac myosin) for patients with heart failure (HF), as well as the lessons learned from the development process.
Concept elicitation focus groups and individual interviews were conducted with patients with HF to identify concepts for the instrument. Cognitive interviews with HF patients were used to confirm that no essential concepts were missing and to assess patient comprehension of the instrument and items.
During concept elicitation, the most frequently reported HF symptoms were shortness of breath, tiredness, fluid retention, fatigue, dizziness/light-headedness, swelling, weight fluctuation, and trouble sleeping. Two measures were developed based on the concepts: the Heart Failure Symptom Diary (HF-SD) and the Heart Failure Impact Scale (HFIS). Findings from cognitive interviews suggested that the items in the HF-SD and HFIS were relevant and well understood by patients. Multiple iterations of concept elicitation and cognitive interviews were needed based on FDA request for a broader patient population in the qualitative study. Lessons learned from the omecamtiv mecarbil PRO/clinical development program are discussed, including challenges of qualitative studies, patient recruitment, expected and actual timelines, cost, and engagement with various stakeholders.
Development of a new PRO measure to support a label claim requires significant investment and early planning, as demonstrated by the omecamtiv mecarbil program.
In a patient-centered healthcare system, incorporating the patient’s perspective in research has become a priority [1, 2]. Patient-reported outcome (PRO) data may be used to support label claims for promotional purposes and/or scientific dissemination under specific guidance from the US Food and Drug Administration (FDA) [3, 4] and in general by the European Medicines Agency (EMA) . The specific guidance from the FDA was published in 2009 , and the FDA’s Patient-Focused Drug Development Initiative was developed in 2013 . Pharmaceutical companies may use PRO information to distinguish a product from a competing product, and to contribute to discussions with Health Technology Assessment (HTA) agencies and payers to support reimbursement decisions [6, 7]. An important aspect of a clinical program is to use tools that enable patients’ symptoms and symptom impacts on their lives to be captured and quantified. PROs are therefore essential in providing a metric by which treatment success can be measured.
Current regulatory requirements for PRO measures to be used in clinical trials are not consistent between regulatory agencies. The FDA has issued the most specific and stringent guidance to be considered in the development of PRO measures intended for use in label claims , while the EMA has published a reflection paper on the use of health-related quality of life measures in drug evaluations . To support a labeling claim in the US, a PRO endpoint must be at least a pre-specified key secondary endpoint , whereas the EMA suggests that the PRO endpoint be driven by an a priori formulated hypothesis and detailed in the statistical analysis plan of a clinical trial that is suitably powered for the PRO analysis . Because of these differences, sponsors must take differing regional agency requirements into consideration in global development programs.
An important component of the FDA guidelines is that a PRO measure must demonstrate content validity, which is the extent to which the instrument measures the concept of interest. In addition, the FDA specifies that a PRO instrument needs to be well defined, reliable, used in accordance with the instrument’s documented measurement capability, and to have demonstrated validity in the target patient population . Several descriptions of the process recommended in the development of a good quality PRO measure have been published [8–11], but reports of the practical considerations of integrating development of a PRO measure into a clinical program are rare.
The importance of integrating a PRO measure in randomized clinical trials (RCTs) of drugs for the treatment of cardiovascular disease (CVD) has recently been highlighted . It is important that patients are able to describe the impact of a novel treatment on their symptoms, functions, and health-related quality of life using scientifically sound PRO measures in CVD RCTs. Since the release of the FDA PRO guidance, no sponsor has achieved a PRO label claim for the treatment of chronic HF (CHF), even though old label claims for carvedilol and other products exist. PROs developed prior to the guidance may have been acceptable as label claims in the past. However, a recent literature review of PROs for chronic HF patients concluded that none of the existing disease-specific PRO instruments for HF satisfy all criteria outlined in the current FDA guidance , and therefore may no longer be considered adequate to support label claims.
The goal of this paper is to present the steps for integrating the development of PRO measures into a clinical drug development program with the aim of achieving a label claim with the FDA on the severity of symptoms and their impact on functioning for patients with HF. We will provide real-world examples of the processes, challenges, solutions, and lessons learned as part of the development of a symptom and impact measure integrated into a HF clinical trial program for omecamtiv mecarbil. Omecamtiv mecarbil is a selective, small molecule activator of cardiac myosin  that has been demonstrated to increase cardiac performance in animal models , healthy human volunteers , and patients with CHF .
Process required for pro development: a real-world example
Omecamtiv mecarbil experience: A literature review and gap analysis  was conducted after completion of phase I and phase IIa studies of omecamtiv mecarbil [15, 16]. The results of the review identified multiple PRO measures for HF symptoms and impacts, but none was considered to meet fully the criteria put forward by the FDA in the PRO guidance document for use in a clinical trial to support a label claim, particularly due to gaps in documentation of content validity . The FDA is most favorable of direct measures of symptoms and physical limitations as opposed to more abstract concepts of social limitations and patients’ perceptions of their quality of life . Therefore, a decision was made to develop two new measures: one to assess the symptoms of CHF and one to assess the impact of symptoms on functioning. A PRO development committee of clinical and PRO experts was formed.
Lessons learned: The number of existing PRO measures and the extensive use of existing PRO measures do not guarantee that any existing measure meets current FDA PRO guidance criteria .
Omecamtiv mecarbil experience: An institutional review board (IRB)-approved qualitative study was conducted; patients provided written informed consent before initiation of study-related activities. One-on-one concept elicitation interviews and focus groups were conducted by trained interviewers and moderators using semi-structured interview guides. Each session was recorded and subsequently transcribed for analysis using qualitative analysis software. An open coding approach was used in the analysis. Additional concept elicitation interviews were conducted until saturation was achieved.
Three concept elicitation focus groups and six individual interviews were conducted with 19 US adults with CHF (mean age 73.8 [standard deviation (SD) = 13] years; 21 % female; 79 % white; 11 % New York Heart Association [NYHA] class IV disease). The most frequently reported symptoms due to HF in concept elicitation focus groups and interviews were shortness of breath (reported by 74 % of patients), tiredness (58 %), fluid retention (58 %), fatigue (47 %), dizziness/light-headedness (32 %) swelling (32 %), weight fluctuation (26 %), and trouble sleeping (21 %). Other symptoms mentioned by less than 20 % of participants included difficulty concentrating, muscle problems (pain, cramps, numbness), loss of appetite, and bloating. Concept saturation of symptoms was reached by the end of the interviews.
HF had the greatest negative impact on physical (difficulty walking [58 %], needing frequent rests [54 %], reduced ability to do household chores [37 %]), social (reduced social/family interactions [37 %], participation in recreational activities/hobbies [26 %]), and emotional (frustration stemming from diminished ability or inability to complete activities [26 %]) domains. Other impacts on functioning mentioned by less than 20 % of participants included difficulty carrying/lifting objects, running, feeling dependent on others, dressing, impacts on intimate relationships, depression, and mood swings. Concept saturation of impacts was reached by the end of the interviews.
Initial item pools of two HF measures, the Heart Failure Symptom Diary (HF-SD) and Heart Failure Impact Scale (HFIS), were created based on the results of the concept elicitation focus groups and interviews, along with experts, and input from PRO, translation, and clinical experts, and a review of available literature. All items were programmed in an electronic device for further testing.
Lessons learned: A recruitment strategy to ensure inclusion of patient groups that are difficult to recruit (e.g., female patients with CHF and patients with severe [NYHA class IV] disease) is an important consideration at the early stages of development, but can present practical challenges by slowing the development process. Use of a recruitment tracker and continuous contact with study sites to understand which patients are available are helpful strategies, as is securing back-up sites. Close relationships with hospital settings may enhance recruitment of patients with acute and severe disease.
Omecamtiv mecarbil experience: Cognitive interviews using the HF-SD and HFIS electronic PRO (ePRO) measures were conducted with 18 adult US patients with CHF (mean age 68.8 [SD = 12] years, 33 % female, 83 % white, 22 % NYHA class IV disease). Findings from the cognitive interviews suggested that items of the HF-SD and HFIS were relevant and well understood by patients with CHF. Response options were also well understood and appropriate for the items. The recall periods for the HF-SD and HFIS were adequate for recollection of symptom experience and impacts due to HF. Results from cognitive interviews also suggested the need for two additional items of impacted activities: indoor and outdoor activities were changed to separate items, and lifting and carrying were changed to separate items. Some items were reworded for clarity.
Lessons learned: Achieving saturation of concepts in the concept elicitation focus groups and interviews ensured that no new concepts were found in the cognitive interviews, and only the separation of related items was needed for the HF-SD and HFIS following the cognitive interviews.
Omecamtiv mecarbil experience: Results of concept elicitation interviews and cognitive interviews were presented to the FDA with request for feedback on the presented evidence in support of the content validity of the two measures. A second round of concept elicitation interviews and cognitive interviews was suggested by the FDA to solicit input from patients with specific demographic (women, African Americans) and clinical (NYHA class IV disease) characteristics. Additional concept elicitation interviews were therefore conducted with 15 US adults with CHF (mean age 59.7 [SD = 10.8] years; 73 % female; 47 % white; 13 % NYHA class IV disease). No participants were included in both rounds of qualitative interviews. The second round of concept elicitation interviews supported the concepts endorsed in the first round of interviews, no new concepts were identified, and no changes were made to the instruments. Additional cognitive interviews were conducted with ten additional US adults with CHF (mean age 68.6 [SD = 9] years; 30 % female; 40 % white; 20 % NYHA class IV disease). The findings from the additional cognitive interviews demonstrated that items of the HF-SD and the HFIS were well understood and relevant to CHF patients, supporting the findings of the initial cognitive interviews.
Lessons learned: A challenge was the recruitment of patients with NYHA class IV disease outside of large, specialized academic centers, as current treatment regimens are effective and prevent development of HF to this stage in many patients. While theoretically important, the successful saturation of concepts in the initial round of qualitative studies meant that no new concepts were identified in the second round.
Omecamtiv mecarbil experience: The HF-SD and HFIS measures were piloted in a phase 2 trial to assess item performance in order to identify and select the final instrument items. This trial also provided the opportunity for an initial assessment of the psychometric properties of the final measures. The measures were administered as ePRO assessments, which were initially tested as part of the cognitive interviews, and enabled an assessment of the usability of electronic data capture in HF patients in advance of proceeding with the electronic mode of administration in the phase 3 clinical trial. In addition to providing initial evidence of the measurement properties of the HF-SD and the HFIS, this step provides additional evidence for the measure’s content validity.
Lessons learned: Including ePRO administration in the cognitive interview phase provided information on usability and avoided a study to assess concordance between paper and ePRO modes of administration, which is time- and cost-consuming. Testing for feasibility and ease of use of ePRO administration early in development is useful before embarking on its use in clinical trials.
Practical and logistical issues: key learnings from the omecamtiv mecarbil clinical program
Omecamtiv mecarbil experience: The expected and actual timelines of HF PRO development stages are presented in Fig. 2 . Early planning in our program led to successful completion of the first round of qualitative interviews and drafting of instruments before the start of phase 2. Work on the PRO strategy for the omecamtiv mecarbil clinical development program started in 2012 and was expected to be completed by 2013. Several factors caused delays in the original timeline, including the need for a second round of concept elicitation interviews and cognitive interviews, difficulties with recruitment of patients with NYHA class IV disease for qualitative interviews, and the urgency and priority status of the drug. Another factor was related to potential difficulty in identifying clinical sites that could help recruit the population of interest within the desired timeframe. Sites using local IRBs usually require lengthier contracting and review processes than sites with central IRBs.
Lessons learned: An early start is critical as delays in the original timeline are likely. Sponsors should anticipate and plan for a second round of qualitative studies, including strategies for recruitment of specific types of patients.
Factors that affect the cost of PRO development
Patient and site recruitment
Use of vendors to identify and recruit patients, payment for participation in qualitative studies, ease of access to target patient groups
Multiple stakeholder involvement
PRO experts, IRBs, clinicians, patients, researchers, PRO and ePRO vendors
Number of iterations
Multiple rounds of concept elicitation interviews and cognitive interviews to achieve saturation of concepts and satisfactory evidence of content validity
Delays in clinical program unrelated to PRO development
Need to update reviews prior to regulatory approval, increased risk of staff changes reducing efficiency of process
Approach selected for quantitative validation
Stand-alone vs regular clinical trials
Mode of instrument administration
ePRO vs pencil and paper vs mixed mode; need to demonstrate instrument equivalence
Clinical trial design
Number of languages and cultural adaptations needed
Dossier preparation, conference presentations, manuscripts, etc.
Omecamtiv mecarbil experience: The planning of the PRO strategy for the omecamtiv mecarbil clinical development program started in 2011 with a literature review and the initial budget for the PRO development was within the cost estimate published in 2011. The need to update the literature review prior to regulatory submission, and to conduct a second round of concept elicitation interviews and cognitive interviews and secondary collection of clinical data from the first round of interviews based on FDA feedback adversely affected the timeline and increased the cost. With careful planning we have managed to save costs by including the ePRO measure in the phase 2 clinical trial, thus eliminating the need for a stand-alone study.
Lessons learned: The iterative nature of qualitative studies can easily increase costs, but this can be offset by initiating ePRO development early in the planning process.
Engagement with various stakeholders
Omecamtiv mecarbil experience: A PRO committee was formed to oversee development of the new instruments. Patients with CHF were involved in concept elicitation and cognitive interview studies. Within the sponsor organization, representatives from Health Economics and Outcomes Research, Biostatistics, Commercial, Clinical Development, and Regulatory Affairs departments contributed to the PRO development program. Qualitative studies were designed and implemented by a vendor. FDA review and feedback was solicited. Clinician and payer feedback were solicited through market/payer research advisory boards and 1:1 interviews.
Lessons learned: Involvement of multiple stakeholders in PRO development ensures and improves the quality of the final measure and contributes to better understanding of the role a PRO can play in clinical trials for involved parties. Careful planning and management of the practical aspects of stakeholder involvement is critical for the success of the program.
Discussion and conclusions
A recent paper from Anker et al.  highlights the importance of integration of PRO measures in clinical trials and notes that currently PROs are not routinely included in CVD trials with the intent to support label claims. The authors advocate for an increased focus on the inclusion of scientifically sound PRO measures in CVD trials. While the Anker paper provides some broad recommendations and considerations in the design of clinical trials with the integration of a PRO, it does not provide specific details on the practical challenges and solutions for including PROs that can support labeling claims. This also applies to the recent publications by Thompson et al. and Garin, et al., which provide reviews of the use of HF PROs in HF trials [2, 19]. While existing publications describing the development and psychometric evaluation of PRO measures provide detailed theoretical steps, they rarely address the practical challenges and decision-making required in the process. Our report builds on the broad consideration from the Anker paper and provides a specific practical example of the integration process and an insight into the lessons learned.
Several governmental, nongovernmental and professional organizations have recognized the value of PROs and are actively developing or acknowledging the need to develop disease-specific PRO instruments for both regulatory approval and quality assessment. In 2005, the National Institutes of Health (NIH) established the Patient Reported Outcomes Measurement Information System (PROMIS), comprising multiple item banks built on existing instruments measuring a range of outcomes. PROMIS is currently used by various US governmental agencies (including the FDA) to assist with development and quality assessment of PRO measures . In 2012, the Centers for Medicare and Medicaid Services (CMS) funded the National Quality Forum (NQF) to develop guidance on PRO measures based on the National Quality Strategy (NQS) to improve medical care for individuals . The CMS now considers the patient perspective in the development of all PRO measures . The Critical Path Institute (C-Path) is a nonprofit, public-private organization that partners with the FDA, PRO researchers, pharmaceutical companies, and the NIH to develop data standards, measurement standards, and methods standards to evaluate new therapies . In 2013, the American Heart Association (AHA) published a scientific statement emphasizing the importance of measuring patients’ health status using PROs , followed by a similar endorsement by the European Society of Cardiology (ESC) a year later . Other professional societies have been involved in activities to further scientific development of cardiovascular assessment measures [24, 25]. Notably, the Patient-Centered Outcomes Research Institute (PCORI) has funded an effort to develop PROMIS Condition-Specific Impact Assessment (PROMIS-CSIA) for patients with HF and osteoarthritis . All PCORI-funded studies require engagement of patients as collaborators as a condition for funding.
Developing a new PRO measure in accordance with the FDA PRO guidance to support a label claim requires significant investment and early planning in the development of a clinical trial program. In this paper, a real-world example in the area of HF was provided to illustrate some of the successes and challenges experienced and to report on the methods of PRO development for HF symptom and impact measures.
We thank Julia R. Gage (on behalf of Amgen Inc.) for assistance with writing the manuscript.
This study was funded by Amgen Inc.
Availability of data and materials
The datasets analyzed for this article are not publicly available because the objective of the study was to present a PRO development process and not to report new data per se, and the instruments are still in development.
IW contributed to the conception and design of the study, acquisition of data, and analysis and interpretation of the data. MA contributed to the conception and design of the study and analysis and interpretation of the data. HO contributed to the conception and design of the study, acquisition of data, and analysis and interpretation of the data. RvM contributed to the conception and design of the study, acquisition of data, and analysis and interpretation of the data. DC contributed to the conception and design of the study and analysis and interpretation of the data. FIM contributed to the conception and design of the study. DLP contributed to the conception and design of the study, acquisition of the data, and the analysis and interpretation of the data. JS contributed to the conception and design of the study and the analysis and interpretation of the data. JRT contributed to the conception and design of the study and the analysis and interpretation of the data. All authors contributed to the drafting of the manuscript, critical revision of the manuscript for important intellectual content, and approved the draft for submission.
I. Wiklund, M. Anatchkova, H. Oko-Osi, and R. von Maltzahn are employees of Evidera, which received funding from Amgen Inc. for this study. D. Chau is an employee and shareholder of Amgen Inc. F.I. Malik is an employee and shareholder of Cytokinetics, Inc., which developed omecamtiv mecarbil in collaboration with Amgen Inc. D.L. Patrick has served as a consultant to Amgen Inc. J. Spertus has served as a consultant to Amgen Inc. for this study, serves as a consultant to Novartis Pharmaceuticals Corporation, Bayer Healthcare, and Regeneron Pharmaceuticals, Inc., has received grant support from Gilead Sciences, Inc., Eli Lilly & Co., the National Institutes of Health, American Heart Association, the Patient-Centered Outcomes Research Institute, and the Aetna Foundation, and has an equity interest in Health Outcomes Sciences. He also owns the copyright to the Seattle Angina Questionnaire, Peripheral Artery Questionnaire, and Kansas City Cardiomyopathy Questionnaire. J.R. Teerlink has received grant/research support and is a consultant to Amgen Inc., Cytokinetics, Inc., Novartis Pharmaceuticals Corp., and Trevena Inc.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Wu AW, Snyder C, Clancy CM, Steinwachs DM. Adding the patient perspective to comparative effectiveness research. Health Aff (Millwood). 2010;29:1863–71.Google Scholar
- Thompson LE, Bekelman DB, Allen LA, Peterson PN. Patient-reported outcomes in heart failure: existing measures and future uses. Curr Heart Fail Rep. 2015;12:236–46.View ArticlePubMedGoogle Scholar
- Guidance for Industry. Patient-reported outcome measures: use in medical product development to support labeling claims. [http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282]. Accessed 8 Sep 2016.
- The voice of the patient: a series of reports from FDA’s Patient-Focused Drug Development Initiative. [http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm368342.htm]. Accessed 8 Sep 2016.
- Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products (EMEA/CHMP/EWP/139391/2004). [http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003637.pdf]. Accessed 8 Sep 2016.
- Anker SD, Agewall S, Borggrefe M, Calvert M, Jaime Caro J, Cowie MR, Ford I, Paty JA, Riley JP, Swedberg K, et al. The importance of patient-reported outcomes: a call for their comprehensive integration in cardiovascular clinical trials. Eur Heart J. 2014;35:2001–9.View ArticlePubMedGoogle Scholar
- Gnanasakthy A, Lewis S, Clark M, Mordin M, DeMuro C. Potential of patient-reported outcomes as nonprimary endpoints in clinical trials. Health Qual Life Outcomes. 2013;11:83.View ArticlePubMedPubMed CentralGoogle Scholar
- Bredart A, Marrel A, Abetz-Webb L, Lasch K, Acquadro C. Interviewing to develop Patient-Reported Outcome (PRO) measures for clinical research: eliciting patients’ experience. Health Qual Life Outcomes. 2014;12:15.View ArticlePubMedPubMed CentralGoogle Scholar
- Rothrock NE, Kaiser KA, Cella D. Developing a valid patient-reported outcome measure. Clin Pharmacol Ther. 2011;90:737–42.View ArticlePubMedPubMed CentralGoogle Scholar
- Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L. Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1--eliciting concepts for a new PRO instrument. Value Health. 2011;14:967–77.View ArticlePubMedGoogle Scholar
- Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L. Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding. Value Health. 2011;14:978–88.View ArticlePubMedGoogle Scholar
- Psotka M, Von Maltzahn R, Anatchkova M, Agodoa I, Chau D, Malik FI, Patrick D, Spertus J, Wiklund I, Teerlink JR. Assessment of patient-reported outcome instruments for patients with heart failure. JACC Heart Fail. 2016. doi:10.1016/j.jchf.2016.04.010. Epub ahead of print.
- Malik FI, Hartman JJ, Elias KA, Morgan BP, Rodriguez H, Brejc K, Anderson RL, Sueoka SH, Lee KH, Finer JT, et al. Cardiac myosin activation: a potential therapeutic approach for systolic heart failure. Science. 2011;331:1439–43.View ArticlePubMedPubMed CentralGoogle Scholar
- Shen YT, Malik FI, Zhao X, Depre C, Dhar SK, Abarzua P, Morgans DJ, Vatner SF. Improvement of cardiac function by a cardiac Myosin activator in conscious dogs with systolic heart failure. Circ Heart Fail. 2010;3:522–7.View ArticlePubMedGoogle Scholar
- Teerlink JR, Clarke CP, Saikali KG, Lee JH, Chen MM, Escandon RD, Elliott L, Bee R, Habibzadeh MR, Goldman JH, et al. Dose-dependent augmentation of cardiac systolic function with the selective cardiac myosin activator, omecamtiv mecarbil: a first-in-man study. Lancet. 2011;378:667–75.View ArticlePubMedGoogle Scholar
- Cleland JG, Teerlink JR, Senior R, Nifontov EM, Mc Murray JJ, Lang CC, Tsyrlin VA, Greenberg BH, Mayet J, Francis DP, et al. The effects of the cardiac myosin activator, omecamtiv mecarbil, on cardiac function in systolic heart failure: a double-blind, placebo-controlled, crossover, dose-ranging phase 2 trial. Lancet. 2011;378:676–83.View ArticlePubMedGoogle Scholar
- Leidy NK, Vernon M. Perspectives on patient-reported outcomes: content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics. 2008;26:363–70.View ArticlePubMedGoogle Scholar
- Patient-reported outcome (PRO) instruments as drug development tools. [http://c-path.org/wp-content/uploads/2013/09/consensussciencepatientreportedoutcomesinstrumentsastools.pdf]. Accessed 8 Sep 2016.
- Garin O, Ferrer M, Pont A, Wiklund I, Van Ganse E, Vilagut G, Almansa J, Ribera A, Alonso J. Evidence on the global measurement model of the Minnesota Living with Heart Failure Questionnaire. Qual Life Res. 2013;22:2675–84.View ArticlePubMedGoogle Scholar
- National Strategy for Quality Improvement in Health Care, 2014 Agency Specific Plans. [http://www.ahrq.gov/workingforquality/agencyplans/nih-specific-plan-nqs2014.pdf]. Accessed 8 Sep 2016.
- Using measurement for quality improvement. [http://www.ahrq.gov/workingforquality/nqs/webinar091714/webinar5.pdf]. Accessed 8 Sep 2016.
- Patient-Reported Outcome Consortium. [http://c-path.org/programs/pro/]. Accessed 8 Sep 2016.
- Rumsfeld JS, Alexander KP, Goff Jr DC, Graham MM, Ho PM, Masoudi FA, Moser DK, Roger VL, Slaughter MS, Smolderen KG, et al. Cardiovascular health: the importance of measuring patient-reported health status: a scientific statement from the American Heart Association. Circulation. 2013;127:2233–49.View ArticlePubMedGoogle Scholar
- Bonow RO, Ganiats TG, Beam CT, Blake K, Casey Jr DE, Goodlin SJ, Grady KL, Hundley RF, Jessup M, Lynn TE, et al. ACCF/AHA/AMA-PCPI 2011 performance measures for adults with heart failure: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures and the American Medical Association-Physician Consortium for Performance Improvement. Circulation. 2012;125:2382–401.View ArticlePubMedGoogle Scholar
- Yancy CW, Jessup M, Bozkurt B, Butler J, Casey Jr DE, Drazner MH, Fonarow GC, Geraci SA, Horwich T, Januzzi JL, et al. ACCF/AHA guideline for the management of heart failure: a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. Circulation. 2013;2013(128):e240–327.Google Scholar
- Facilitating patient reported outcome measurement for key conditions. [http://www.pcori.org/research-results/2013/facilitating-patient-reported-outcome-measurement-key-conditions]. Accessed 8 Sep 2016.