- Study protocol
- Open Access
Implementation analysis of patient reported outcomes (PROs) in oncological routine care: an observational study protocol
Health and Quality of Life Outcomes volume 18, Article number: 3 (2020)
The successful implementation of patient-reported outcomes (PROs) in clinical routine faces many challenges, first and foremost the lack of consideration thereof in the patient care process. The aim of this study will be to first identify relevant barriers and facilitators and then design suitable implementation strategies which will be evaluated to improve the effectiveness of a PRO measure assessment in inpatient and outpatient cancer routine care.
During the preparation phase, interviews with oncological patients (N = 28) and medical staff (N = 4) as well as focus groups with medical staff (N = 18) across five different departments caring for cancer patients were conducted. On the basis of these, qualitative content analysis revealed relevant barriers and facilitators for implementation of PROs in cancer care. Subsequently, implementation strategies and a model of implementation were developed. In the study phase, implementation strategies will be evaluated based on nine different implementation outcomes in five different oncological clinics. Evaluation of the implementation process will take place during three months in each clinic and data will be conducted pre, while and post implementation of the PRO measure. Therefore a sample size of 60 participants of whom 30 staff members and 30 participants will be questioned using existing and newly developed implementation outcome evaluation instruments.
Key to improving the effectiveness of PRO assessment in the time-critical clinical environment is the utilization of easy-to-use, electronic PRO questionnaires directly linked to patients’ records thereby improving consideration of PROs in patient care. In order to validate the effectiveness of this implementation process further, an evaluation parallel to implementation following an observational study design with a mixed-methods approach will be conducted. This study could contribute to the development of adequate evaluation processes of implementation of PROs to foster sustainable integration of PRO measures into routine cancer care.
This study was registered at Open Science Framework (https://osf.io/y7xce/).
Contributions to the literature
Research has shown that the successful implementation of PROs in cancer routine care faces many challenges, first and foremost the lack of consideration thereof in the patient care process.
According to research and practice findings, inhibiting factors differ between oncological in- and outpatient clinics. Therefore, implementation strategies and process tailored to the individual needs of cancer units, patients and staff members have to be developed in order to facilitate implementation.
This study contributes to the efforts of sustainably implementing PROs into cancer care through precise evaluation considering various implementation outcomes and measurements.
Today, patient reported outcomes (PROs) maintain an important role in patient centered care. PROs are self-assessment measures to collect information on health-related quality of life (HrQoL), physical discomfort or patient perceived health status . Especially in chronic diseases PROs play an important role to generate data on the patient experience . Therefore, research strongly suggests to implement PROs as a tool to improve the quality of patient care . On the other hand it is surprising that the implementation of this information lags behind and that research is required in this respect . HrQoL is one major dimension assessed through PRO measures in oncological care . As cancer patients often experience physical and psychosocial consequences of their disease and its treatment, evaluation of HrQoL is important to get a full understanding of patient’s needs . Interest in the use of HrQoL ratings in daily clinical practice has increased substantially . However, successful implementation of PROs in clinical routine faces many challenges. Therefore, integration and use of results of PRO measures in oncological care is lacking .
Implementation is characterized as the use of strategies that serve to integrate and adapt an intervention into a specific setting . Therefore, implementation research focuses on methods and strategies to understand and enhance successful integration of health care interventions . In this context, Proctor et al. (2010) define implementation outcomes as “the effects of deliberate and purposive actions to implement new treatments, practices, and services”. Concluding, it is the aim of implementation research to provide sustainable and accepted implementation strategies for interventions to promote long-term use in routine care.
The evaluation of implementing health-related interventions in complex health care settings e.g. in the multidisciplinary setting of cancer care in a University Medical Center, is often poorly reported . Implementation studies use diverse approaches and terminology to measure the success of implementation of interventions [11, 12]. However, to correctly interpret success or need for improvement of an intervention, it is key to be able to distinguish between poor efficacy of the intervention itself as a result of failure and unsuitable implementation strategies of the intervention . Without detailed analysis of the implementation process, these sources of error are difficult to assign . Therefore, it is important to also define and evaluate next to the intervention itself, the implementation process into clinical routine practice .
Many studies indicate that the use of PROs has been found to be useful, but there is often a lack of clear interpretation and structure for the application of the instrument in clinical routine . From a clinician’s point of view, frequent barriers for implementation of PROs are lack of time, lack of training and support, and low personal confidence . On the patient’s side, too burdensome interventions, e.g. too long or critical questions, can hinder effective implementation of PROs . From an organizational level, resources and strategies for successful implementation are often missing . Another problem arises when there is no adequate response by physicians e.g. to address critical PRO results . Therefore, comprehensive research on inhibitory and beneficial factors for the use of PROs in clinical routine is important to facilitate the implementation process and to maintain the sustainability of PRO interventions in oncological care .
Proctor et al. (2011) propose eight dimensions following e.g. the RE-AIM Framework promoted by Glasgow (2007) to evaluate implementation of interventions in health care: Acceptability, Adoption, Appropriateness, Cost, Feasibility, Fidelity, Penetration and Sustainability [11, 18]. Implementation of interventions should be evaluated on these eight dimensions in order to gain precise information on the implementation process and to identify possible barriers. However, instruments evaluating implementation outcomes are lacking which leads to struggle in evaluating implementation processes satisfyingly . Hence, important changes of implementations strategies cannot be made to further improve the implementation process.
Concluding, implementation science advises to consider certain factors that can substantially influence implementation efforts . The purpose of our study is to identify beneficial and inhibiting conditions for clinicians and patients to use a PRO measure assessing health-related quality of life in cancer patients in clinical routine. On the basis of these findings, the PRO measure will be implemented into oncological routine care in a University Hospital in Germany. To ensure sustainable use of the PRO measure and its outcomes, the implementation will be evaluated following the dimensions proposed by Proctor et al. (2011) as well as the Consolidated Framework for Implementation Research (CFIR) [11, 12].
To evaluate the implementation of the PRO measure, an observational studyFootnote 1 with a mixed method design will be conducted. The study contains of two phases and it is planned to combine qualitative and quantitative data in an exploratory mixed methods study design. Study participants will be recruited at five inpatient and outpatient clinics of the University Medical Centre Hamburg Eppendorf (II. Medical Clinic and Polyclinic, the Department of Stem Cell Transplantation, the Department of Gynecology, the Department of Radiotherapy and Radiation Oncology and the Department of Otolaryngology, where the PRO measure will be implemented and evaluated. Inclusion criteria for patients are inpatient or outpatient cancer treatment in one of the five clinics, sufficient language skills in German and no severe cognitive or verbal impairments in providing information and giving informed consent. The study received approval by the ethics committee of the medical association Hamburg (PV5636).
(For detailed outline of the study design see the enclosed Additional file 1 StaRI Checklist.)
Aim of the preparation phase is to assess relevant barriers and facilitators for implementation of a PRO measure to assess HrQoL of cancer patients to prepare implementation in the study phase. For this purpose, interviews with oncological patients (N = 28) and oncologists (N = 4) were undertaken. To facilitate further discussion and exchange, five focus groups with oncologists, oncological nurses and psychologists were conducted. Patients as well as clinicians were asked to name possible barriers and facilitators for implementation of the PRO measure. The results were presented to a group of eight experts for discussion. Psychooncologists, oncologists, quality of life scientists, staff nurses, representatives of the quality management and a representative of a health insurance were present. On the grounds of the findings and of current state of research, the implementation process and implementation strategies were determined. Implementation strategies are: 1) a concise PRO measure (development of the questionnaire will be described elsewhere), 2) electronic input into a software with direct interface to patients’ records, 3) software and process training for medical staff and 4) guidance on pathways for further care. This preparation phase took place from December 2017 until April 2018.
In this study phase a PRO measure will be implemented into clinical routine practice at the University Medical Center Hamburg-Eppendorf at five oncological clinics. Gradually, one clinic after another will be included in the implementation. While implementation will take place, the implementation process will be evaluated in every clinic for three months: pre, while and post to first implementation of the PRO measure (see Fig. 1). Therefore, purposeful sampling will be used to ask medical staff member (i.e. nurses and doctors) to complete questionnaires assessing relevant implementation outcomes. For patients, purposeful sampling will be used by medical staff members to point out patients who are well enough to take part in semi structured interviews. Questionnaires will be presented in paper pencil format to medical staff members, interviews will be conducted by trained research assistants following semi structured interview guidelines. Additionally, one nurse and one physician of each clinic will be asked further questions concerning implementation outcomes in a semi structured interview. Furthermore, statistics on usage and response to the PRO measure will be retrieved from the electronic patient documentation system of the clinics. A pilot run will be conducted. Staff members of the University Medical Centre will be asked to give their impressions by using the thinking out loud technique, in order to assess the comprehensibility and feasibility of the evaluation questionnaires. The statistical survey will start at the end of July 2019 and end in December 2019.
Recruitment of patients in study phase I will be carried out in cooperation with the II. Medical Clinic and Polyclinic, the Department of Stem Cell Transplantation, the Department of Gynecology, the Department of Radiotherapy and Radiation Oncology and the Department of Otolaryngology.
Recruitment and procedure
Potential patients to be questioned were pointed out by staff. The appointed patients were asked to participate and to give a written consent and were interviewed by scientific staff. Interviews with oncological patients (N = 28) and oncologists (N = 4) were undertaken. To facilitate further discussion and exchange, five focus groups with oncologists, oncological nurses and psychologists were conducted. Patients as well as clinicians were asked to name possible barriers and facilitators for implementation of the PRO measure. The results were presented to a group of eight experts for discussion.
A random sample of eligible patients to be questioned will be pointed out by staff. Cross-sectional samples of patients as well as longitudinal samples of medical staff members will be questioned at three different times: pre, while and post implementation process. Unlike patients, same staff members will be questioned pre, while and post implementation. Regarding fluctuation of patients during evaluation of the implementation, different patients will be questioned while and post implementation process.
All three phases of the evaluation of the implementation will take into account appraisal of patients. Patients nor clinicians will be involved in the conception of the study.
Measurements and outcomes
A semi structured interview guide was developed based on Helfferich (2009) asking one main question concerning possible barriers and facilitators of implementation of a PRO measure in routine care . Focus groups were carried out following a focus group guide referring to Barbour (2014) including the same main question as the interview guide .
Implementation of the PRO measure will be evaluated based on the dimensions proposed by Proctor et al. (2011) . Additionally, next to sociodemographic data the perceived benefit of the implementation of the PRO measure will be assessed. Different implementation outcome dimensions will be assessed at three stages of implementation: pre, while and post (see Table 1). “Acceptability” will be assessed using a German translation of the Acceptability E-Scale ) while as well as post implementation of the PRO measure. The 6-item questionnaire will be translated into German following the TRAPD protocol . “Adoption” will be assessed pre and post implementation using the German translation of the Organizational Readiness for Implementing Change (ORIC) questionnaire . “Appropriateness” will be assessed while and post implementation using the relevance scale of a German translation Workshop Evaluation Form (TCU Weval) questionnaire . The items will be translated into German following the TRAPD protocol . “Feasibility” will be assessed pre and post implementation using the program support scale of the Weval questionnaire . The items will be translated into German following the TRAPD protocol . “Cost” will be assessed pre implementation by one question on the expected time taken to record information and address problems according to the PRO measure and post implementation by one question about the time it actually takes for staff members to record information and address possible problems. “Fidelity” and “Penetration” will be assessed while and post implementation using one question for each dimension in a short survey as well as field notes taken by scientific staff members as well as statistical reports of clinical records. “Sustainability” will be assessed while and post implementation using two questions on the use of the PRO measure as well as through statistical reports of clinical records and field notes. “Benefit” will be assessed asking patients while and post implementation with one question about the perceived benefit of the PRO measure for the treatment of the patient. Staff members will be questioned on the “Benefit” by one question pre implementation on the expected benefit of the PRO measure and post implementation on the actual benefit for the treatment of patients of the PRO measure.
Interviews, focus groups and expert discussion were carried out by scientific staff, recorded and afterwards transcribed by staff members. The qualitative data was structured via MAXQDA 10 and analyzed using qualitative content analysis based on Mayring . Within the procedure of analyzing the data, deductive-inductive category application was used: deductive main-categories (generated through literature research) and inductive sub-categories (derived from text analysis). Quality criteria to be examined for the qualitative content analysis were e.g. interrater reliability and communicative validation.
For quantitative data originated in the study phase, analyses of variance will be computed to compare the three different stages during implementation in the five clinics (SPSS Vers. 25). Missing data will be imputed using the expectation–maximization algorithm . Transformations of data will only be applied, if data structure requires so (i.e. non normality of residuals).
Sample size and power
Power calculations according to sample size calculations by Viechtbauer et al. (2015)  for this pilot evaluation suggest N = 59 with a confidence level of 95% and a low probability of the problem to occur of π = 0.05 in total. We therefore chose a number of N = 60 in total, n = 12 for every clinic participating which leads to n = 6 patients and n = 6 staff members of whom n = 3 nurses and n = 3 doctors in each clinic.
Ethics and dissemination
The medical ethics committee of the Medical Chamber of Hamburg reviewed and approved the study protocol (date: 23 October 2017, number: PV5636). With this project it is intended to evaluate implementation of PROs into oncological routine care and to improve psychosocial care for cancer patients. Patients and health care professionals will be asked to participate by joining focus groups and interviews and by filling in questionnaires. Risks or disadvantages on the patient’s side are not expected. Written survey as a method does not involve direct intervention in medical procedures. A written informed consent is mandatory for participation in the study for patients as well as staff members. Patients participating in the study will be informed about voluntariness of participation and the possibility to refuse or discontinue participation at any time without any negative consequences. For further questions concerning the study, contact details of study assistants will be displayed.
The project duration is 36 months. The study was initiated in May 2016. Within the first 7 months, extensive preparatory work was carried out. The recruitment of participants for the preparation phase started in November 2016 and will begin at the end of July 2019 for the study phase. Completion of data collection is planned at the end of December 2019. Data entry, management and analysis as well as the publication of the findings in peer-reviewed journals and at conferences will take place continuously.
The use of PROs in oncological routine care to assess HrQoL in cancer patients can improve health care by assessing relevant symptoms and burdens in HrQoL. Furthermore, instant reaction to critical outcomes on HrQoL measures by clinicians is crucial in order to facilitate optimal treatment of cancer patients. However, implementation of PROs is often unsuccessful and unsustainable. One reason for this could be insufficient evaluation of the implementation process in order to detect possible barriers and facilitators to implementation and respond to those during or after evaluation process. We consider the chosen theoretical models for this study as reasonable and the evaluation tools sufficient in reliability and validity. Qualitative methods in this study are reasonable and chosen approach is feasible. Therefore, this study could contribute to the development of adequate evaluation processes of implementation of PROs to foster sustainable integration of PRO measures into routine cancer care.
Availability of data and materials
Registration of this study at Open Science Framework (https://osf.io/y7xce/).
Consolidated Framework for Implementation Research
Health-related quality of life
Patient reported outcomes
Basch E, et al. Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book. 2018;38:122–34.
Anker SD, et al. The importance of patient-reported outcomes: a call for their comprehensive integration in cardiovascular clinical trials. Eur Heart J. 2014;35(30):2001–9.
Anatchkova M, et al. Exploring the implementation of patient-reported outcome measures in cancer care: need for more real-world evidence results in the peer reviewed literature. J Patient Rep Outcomes. 2018;2(1):64.
Bjordal K. Impact of quality of life measurement in daily clinical practice. Ann Oncol. 2004;15(Suppl 4):iv279–82.
Kenzik KM, et al. How much do cancer-related symptoms contribute to health-related quality of life in lung and colorectal cancer patients? A report from the Cancer Care Outcomes Research and Surveillance (CanCORS) Consortium. Cancer (0008543X). 2015;121(16):2831–9.
Hilarius DL, et al. Use of health-related quality-of-life assessments in daily clinical oncology nursing practice. Cancer. 2008;113(3):628–37.
Duncan EA, Murray J. The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review. BMC Health Serv Res. 2012;12(1):96.
Glasgow RE, et al. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102(7):1274–81.
Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1(1):1.
Ross J, et al. Developing an implementation strategy for a digital health intervention: an example in routine healthcare. BMC Health Serv Res. 2018;18(1):794.
Proctor E, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.
Damschroder LJ, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):1–15.
Zängl, P., Zukunft der Pflege: 20 Jahre Norddeutsches Zentrum zur Weiterentwicklung der Pflege. 2015: Springer-Verlag.
Fixsen, D.L., et al., Implementation research: a synthesis of the literature. 2005.
Wintner LM, et al. Verwendung und Nutzen von Patient-Reported Outcomes in der onkologischen Behandlung: eine Übersicht. Wien Klin Wochenschr. 2012;124(9):293–303.
Mitchell AJ. Screening for cancer-related distress: when is implementation successful and when is it unsuccessful? Acta Oncol. 2013;52(2):216–24.
Antunes B, Harding R, Higginson IJ. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliat Med. 2014;28(2):158–75.
Glasgow, R.E. The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. in NIH Conference on Building the Science of Dissemination and Implementation in the Service of Public Health: Bethesda, MD. 2007.
Lewis CC, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10(1):155.
Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.
Helfferich C. Die Qualität qualitativer Daten. Wiesbaden: VS Verlag für Sozialwissenschaften; 2009.
Barbour R. Doing Focus Groups. Qualitative research kit. Vol. 5. 2014. London: Sage Publishing.
Tariman JD, et al. Validation and testing of the acceptability E-scale for web-based patient-reported outcomes in cancer care. Appl Nurs Res. 2011;24(1):53–8.
Willis, G., et al., Evaluation of a Multistep Survey Translation Process. Survey Methods in Multinational, Multiregional, and Multicultural Contexts. 2010. p. 141–156.
lindig, A., Hahlweg, P., Christalle, E., Scholl, I. , Assessing psychometric properties of the German version of Organizational Readiness for Implementing Change (ORIC). (in preparation).
Bartholomew NG, et al. Counselor assessments of training and adoption barriers. J Subst Abus Treat. 2007;33(2):193–9.
Mayring, P., Qualitative Inhaltsanalyse: Grundlagen und Techniken. 2008, Weinheim und Basel: Beltz.
Bonomi AE, et al. Multilingual translation of the functional assessment of Cancer therapy (FACT) quality of life measurement system. Qual Life Res. 1996;5(3):309–20.
Viechtbauer W, et al. A simple formula for the calculation of sample size in pilot studies. J Clin Epidemiol. 2015;68(11):1375–9.
We thank all participating student assistants of this study for their support.
This work is supported by “Innovationsfonds des Gemeinsamen Bundesausschusses” grant number [01VSF16024].
Ethics approval and consent to participate
This study received approval by ethics committee of the medical association Hamburg. Reference number: PV5636.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1:
Standards for Reporting Implementation Studies: the StaRI checklist for completion
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Görlach, M.G., Schrage, T., Bokemeyer, C. et al. Implementation analysis of patient reported outcomes (PROs) in oncological routine care: an observational study protocol. Health Qual Life Outcomes 18, 3 (2020). https://doi.org/10.1186/s12955-019-1262-2