Research | Open | Published:
Using patient values and preferences to inform the importance of health outcomes in practice guideline development following the GRADE approach
Health and Quality of Life Outcomesvolume 15, Article number: 52 (2017)
There are diverse opinions and confusion about defining and including patient values and preferences (i.e. the importance people place on the health outcomes) in the guideline development processes. This article aims to provide an overview of a process for systematically incorporating values and preferences in guideline development.
In 2013 and 2014, we followed the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to adopt, adapt and develop 226 recommendations in 22 guidelines for the Ministry of Health of the Kingdom of Saudi Arabia. To collect context-specific values and preferences for each recommendation, we performed systematic reviews, asked clinical experts to provide feedback according to their clinical experience, and consulted patient representatives.
We found several types of studies addressing the importance of outcomes, including those reporting utilities, non-utility measures of health states based on structured questionnaires or scales, and qualitative studies. Guideline panels used the relative importance of outcomes based on values and preferences to weigh the balance of desirable and undesirable consequences of alternative intervention options. However, we found few studies addressing local values and preferences.
Currently there are different but no firmly established processes for integrating patient values and preferences in healthcare decision-making of practice guideline development. With GRADE Evidence-to-Decision (EtD) frameworks, we provide an empirical strategy to find and incorporate values and preferences in guidelines by performing systematic reviews and eliciting information from guideline panel members and patient representatives. However, more research and practical guidance are needed on how to search for relevant studies and grey literature, assess the certainty of this evidence, and best summarize and present the findings.
According to the World Health Organization (WHO), “a recommendation [in a practice guideline] tells the intended end-user of the guideline what he or she can or should do in specific situations to achieve the best health outcomes possible, individually or collectively…” . A recommendation does not only depend on the magnitude of an intervention effect, but should incorporate other considerations and criteria that determine the direction and strength of a recommendation, such as the importance or weight of the health outcomes . Recommendations are the deliberate product of inclusively considering these criteria that influence decision-making by a multidisciplinary group through a structured process [3–6]. This multidisciplinary group typically includes content experts, patients, methodologists and other stakeholders [7–9]. These different individuals may choose different treatment options when they are presented with the same evidence. When full understanding of the information is ensured, different choices for recommendations are often the result of disparate values and preferences.
Although infrequently practiced, ideally this information should be based on evidence from thoroughly conducted research, which is collected through a systematic approach . The main reason for incorporating values and preferences in guideline development process is that recommendations aligned with patient values and preferences may be more easily accepted, implemented and adhered to by those intended to benefit from the guidelines. Additionally, in the individual physician-patient encounters, recommendations with consideration of patient’s preferences, can better inform the decision-making process [10–15]. Further motivations for incorporating patient values and preferences in guideline include ethical and moral imperatives, accountability and legitimacy of the guideline developers.
The Grading of Recommendation, Assessment, Development and Evaluation (GRADE) working group developed the Evidence-to-Decision (EtD) framework to facilitate the process of integrating the criteria considered necessary in guideline development and documenting such process for different audiences [4, 5]. With this framework, to formulate a recommendation, these criteria include: balance between desirable and undesirable effects, certainty in the evidence informing the recommendation, resource utilization, and impact on health system equity, feasibility of the recommendation, stakeholder acceptability, patient values and preferences. A number of tools and initiatives explicitly describe the factors that should be considered when developing recommendations with different stakeholders. These include the development of the Guidelines International Network (GIN)-McMaster Guideline Development Checklist , the presentation methods developed in GRADE’s Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) Project  as well as collaborative guideline development activities with professional and governmental organizations. However, we still recognize paucity in practical strategies to incorporate patient values and preferences in the guideline development process.
In fact, the definition of and strategies for determining values and preferences are still under debate. The GRADE approach includes the consideration of patient values and preferences as the relative importance of outcomes or health states of interest [2, 3, 17, 18]. Similarly, in health economics, preference is a general term that includes health utilities elicited under uncertainty (e.g. results from standard gamble), as well as the values elicited under certainty (e.g. time trade off or visual analogue scale) [19–21]. With this GRADE definition, the preference for or against an intervention is conceptually equivalent to the importance placed on outcomes that follow from the decision to undergo an intervention. That is, the preference for or against an intervention is a result of indirectly weighing the health outcomes it causes (e.g. the outcome burden when taking a medication or the consequences of undergoing surgery such as the outcome postoperative pain) . Thus, the preference for or against a treatment intervention is an implicit result of the relative importance of the health outcomes an individual connects to the intervention. However, while values and preferences directly relate to the relative importance of health outcomes in practice guidelines, they also implicitly relate to achieving better health outcomes when judging other aspects that are relevant for a decision. These other aspects such as attitudes, expectations and beliefs are also considered under this umbrella term [22, 23]. In the GRADE EtD, these aspects often fall within other criteria of the EtD framework (e.g., equity, feasibility or acceptability considerations). For example, if a society places low value on avoiding resource expenditure for wide implementation of a new intervention, it may be considered feasible. Patients may find an intervention administered by a health worker other than a physician not acceptable, if they expect the latter to administer it. Thus, feasibility and acceptability are considerations related to values and preferences but not as directly related to the importance patients place on the health outcomes.
Box 1. Relevant criteria in Evidence-to-Decision Framework
|People values and preferences: the relative importance people place on the health outcomes; since we consider an intervention in the context of the consequences it incurs, the preferences for or against an intervention is a consequence of the relative importance people place on the expected or definite health outcomes it incurs.|
|Acceptability and feasibility: views or perspectives or importance of health outcomes placed by stakeholders beyond the target population of the recommendation|
Despite the increasing importance of practice guidelines in the management of health problems, there is a lack of evidence informing about initiatives using values and preferences in the guideline development process. Therefore, we addressed the challenges of integrating values and preferences in practice guidelines. Generally, we utilized the GRADE system for guideline development that is endorsed by over 100 organizations and applied worldwide . Specifically, we first developed an approach for systematically identifying information on values and preferences. Second, we conducted case studies on how to consider local values and preferences evidence in the guideline development process. Our case studies were based on 22 guidelines with 226 recommendations covering diverse clinical areas in a new national guideline program for the Ministry of Health of Saudi Arabia.
For these guidelines, we were specifically interested in identifying values and preferences relevant to the context of the Saudi society. Methodological details of the guideline development process for the Saudi Ministry of Health are described elsewhere [24, 25]. The Ministry of Health of Saudi Arabia had embarked on standardizing and coordinating guideline development nationally to promote the awareness and practice of evidence-based medicine [24, 25]. In this project, we used the definition of “relative importance of outcomes” for patient values and preferences. We undertook several steps to obtain information about patient values and preferences. We performed a systematic review to summarize relevant studies of values and preferences in populations of interest. In addition, we sought input from clinical experts and consulted patient representatives (see the Fig. 1). To assess the feasibility of our approach, we also monitored the workload resulting from conducting systematic reviews on values and preferences during guideline development.
Our approach to comprehensively identifying and understanding existing evidence about values and preferences started with a systematic review summarizing the relevant research evidence . Similar to any systematic review process this included formulation of research questions, literature search, screening according to eligibility criteria, as well as appraisal and summary of the available evidence [7, 14, 27].
Formulation of research question and GRADE definition of values and preferences
We defined the values and preferences as the relative importance of outcomes and formulated the research question for the systematic review of values and preferences as: “what is the relative importance that a population of interest places on the main outcomes?” With this research question, we considered both the studies on the relative importance of outcomes and studies on the preferences for or against an intervention eligible in the 22 guidelines and the detailed recommendations therein.
Studies reporting the “relative importance of outcomes” relevant to the guideline disease topics were included. We included studies that elicited utilities of outcomes through direct measurement techniques including standard gamble, time trade off, visual analogue scales (VAS), and indirect measurement techniques based on generic tools such as EuroQol five dimensions questionnaire (EQ-5D), HUI (health utility index), QWB (quality of wellbeing), as well as utility or health status values transformed (mapping) from quality of life measurement [28–30]. We recognize that not all scientists consider VAS a utility instrument because it does not include a choice under uncertainty. While acknowledging this, we consider VAS measures as eligible to indicate the relative importance of outcomes. Direct choice refers to the technique of asking participants to choose from a set of options. We included studies that expressed the preferences through willingness to pay, probability trade off, discrete choice exercise, ranking, and paired comparison. We also included studies that used other questionnaires and scales, sometimes self-developed to ask preference for outcomes. We also included studies that measured the importance of outcomes in qualitative studies (See Table 1) [23, 31]. Eligible studies included either participants who were experiencing the relevant health states or participants who did not experience the health state of interest but were provided with descriptions of scenarios of the health state [32–34].
We conducted 22 systematic reviews on information suggesting the importance of outcomes; one for each guideline. We developed a broad search filter for values and preferences studies for Ovid Medline, EMBASE and PsychInfo, informed by a search strategy utilized in a previous guideline development process . This search filter included keywords for the following concepts: health state values, preference, utility, attitude to health, patient decision, patient participation, patient satisfaction, patient view, patient perception and their variant formats so as to be as inclusive as possible and capture all potential relevant studies (see Additional file 1). The development of the search strategy is another ongoing project and the detailed development process will be reported in another publication [Selva A, Solà I, Zhang Y, Sanabria AJ, Pequeño S, Rigau D, Martínez L, Mas G, Haynes RB, Schünemann HJ, Alonso-Coello P: Development and use of a content search filter for studies on how patients and other stakeholders value health outcomes (submitted)].
In order to address local values and preferences and enhance contextual information, we also added a geographic search filter that restricted the search to the Kingdom of Saudi Arabia and more broadly to the Middle East. Thus, we developed a complex search strategy based on three search filters: a broad values and preferences filter, the disease specific filters for each guideline, and a geographic filter. These filters were combined using a Boolean “AND”.
Screening and data abstraction
We systematically screened titles and abstracts and retrieved studies for full text screening if they were deemed eligible or if the abstract lacked the detail to determine eligibility by at least one of the screeners. We reviewed the full text articles and summarized the findings stratified according to Table 1 and incorporated them into the GRADE EtD frameworks for each of the 22 guideline areas. We a priori broadened our inclusion criteria and included indirect evidence from other settings when we did not identify information specific for the Saudi Arabia setting.
Input from panel members
Furthermore, we asked guideline panel members (including patient representatives with and without previous experience in the condition of interest) to provide their views on the relative importance of the main outcomes, and their experience related to the disease of interest. We specifically asked clinicians to reflect on patients’ views based on their previous clinical interactions with patients. However, we did not conduct de novo studies on eliciting values and preferences for these guidelines.
Findings of the systematic reviews
We identified a wide variety of eligible studies using utility elicitation, non-utility estimates from questionnaires or scales, as well as qualitative research. Due to heterogeneity of designs and outcomes, we did not pool results and thus provided narrative summaries of the results for each topic. We summarized the information in EtD frameworks for each panel to consider and allow for them to provide feedback. Here, we present guideline-specific examples of the identified studies to illustrate our findings. They are based on a description by utility tools that were used in the original studies.
Utility based estimates
For the antithrombotic guidelines that we produced, utilities for severe, moderate and mild nonfatal intracranial bleeds were identified ranging from 0.10 to 0.51, 0.29 to 0.77 and 0.47 to 0.94, respectively [35, 36]. The utility was 0.63 for nonfatal pulmonary embolism, and 0.44 to 0.84 for major bleed. A systematic review on breast lump-related values and preferences reported the following utilities: 0.96 for disease-free survival, 0.76 to local-regional recurrence, 0.72 to contralateral breast cancer and 0.64 to distant metastasis .
For the guideline on management of breast lump and primary breast cancer, the systematic review identified one study reporting an additional year in life expectancy or 3% in survival rates was sufficient to make adjuvant chemotherapy worthwhile for 68–84% of women .
For the guideline on the screening and treatment of precancerous lesions for cervical cancer prevention, we identified one qualitative research study suggesting that women fear screening and may have a high level of anxiety related to colposcopy or treatment .
Input from panel members
Our consultations with panel members suggested that they were not aware of any studies that were missed by our systematic review process. We also asked them to indicate if indirect evidence from other settings is applicable to the Saudi Arabia setting. Generally, the panellist did not believe there were significant differences except in a few cases. For example, for breast cancer screening, the panel members suggested that in the Saudi Arabia setting, patients place a lower value for any psychological effect of false positive results and frequency of screening compared to the perceived benefits of screening strategies on mortality. In the venous thromboembolism (VTE) treatment guideline development, panellists reflected that oral anticoagulation requires frequent testing and monitoring, diet and medication restrictions, stoppage for procedures. However, anticoagulation would be given for a relatively limited period of time and patients would view potential reduction in mortality and symptomatic VTE favourably .
In the allergic rhinitis guideline, panel members suggested that some patients in Saudi Arabia would not accept sublingual immunotherapy with some allergens of animal origin. The panel evaluating hemodialysis options described that: “the preference to delay dialysis may be stronger in Saudi patients compared to non-Saudi patients (i.e. Saudi patients are more hesitant/resistant to start dialysis)” .
Use of the information as part of decision-making process
The importance patients place on outcomes influences the balance of benefits and harms thereby impacting on the direction and strength of a health recommendation. Thus, being explicit about the relative importance requires a transparent description of how they influenced the recommendation. The panels were made aware that, following the GRADE approach, high variability or uncertainty about the values and preferences typically leads to weak or conditional recommendations .
Table 2 summarizes some examples showing how the guideline panels used the information when formulating recommendations. Panels were instructed to use the information provided about the relative importance of the main outcomes and balance of the desirable and undesirable consequences. Panellists also made judgments about the variability and uncertainty about the values and preferences information.
For example, for the antithrombotic guideline, the systematic review on utilities suggested that major bleeding was equivalent to nonfatal pulmonary embolism; while intracranial bleed overall was 2 to 3 times worse than major bleed or pulmonary embolism . In the Breast Lump guidelines we found that recurrence and metastasis are the most important outcomes for women, and were considered as such by the panel .
How consideration of local values and preferences influenced recommendations
The presumption that local values and preferences differ from those obtained in other settings, questions the usefulness of using the latter. In several cases, local values and preferences contributed significantly to the formulation of recommendations. For example, the allergic rhinitis management guideline stated since “there is important variability about how much people value its ([sublingual immunotherapy, SLIT)] effectiveness because there is a concern that some patients in Saudi Arabia would not accept SLIT with some allergens of animal origin”. Consequently, the recommendation was a weak recommendation suggesting sublingual immunotherapy for treatment of adults with seasonal or intermittent allergic rhinitis based on moderate quality evidence . Although the recommendation was not different from the source guideline , one of the main reasons for this weak recommendation was the expression of local patient values and preferences described above.
The recommendation comparing ultrasonography versus mammography, as part of the triple assessment of palpable breast masses in women aged 30–40 years, was associated with very low certainty in the evidence of effects. The panel suggested “patients would likely favour the use of ultrasonography” because mammography can be more painful and uncomfortable for patients. In the panels’ view this consideration of values and preferences justified a strong recommendation because ultrasonography showed better diagnostic accuracy (sensitivity and specificity) compared with mammography despite very low certainty in the evidence .
Workload related to values and preferences
Incorporating values and preferences in guideline development required resources on the following levels: literature searches, screening and synthesis, preparation of the GRADE EtD frameworks and consideration of values and preferences in decision-making. During development of the search strategy, we noted that many relevant studies were difficult to identify because of the lack of a validated filter or of standardized keywords (Medical Subject Headings: MeSH terms) being used to tag eligible studies. With the definition, measurement and methodology of values and preferences for guidelines still under debate, our aim to not miss relevant information was time and resource consuming. We managed this burden by limiting our search strategy through the stepwise use of a geographic search filter when required. For example, in the Migraine Headache guidelines, we first applied a geographic filter. After identifying no eligible studies, we felt it was necessary to spend additional time and resources to do a larger search for indirect evidence outside of the local context.
Panels recognized the importance of explicitly incorporating the information in the process and considered it in all of the 226 recommendations. The structured summary and presentation of the values and preferences information for each question in the GRADE EtD framework facilitated the process of considering this type of evidence.
We describe an approach for the incorporation of the relative importance of health outcomes in healthcare recommendations. We applied a multi-faceted approach utilizing a systematic review strategy complemented by other information sources. We use illustrative examples to show the usefulness of identifying relevant studies and using their findings in drafting the recommendations.
Strengths and limitations
The systematic and transparent approach to identify and summarize published literature on values and preferences is the strength of the proposed strategy. The feedback from experienced panel members suggested that we did not miss important relevant studies. A second strength is our pre-conceived and structured approach to incorporate both published and elicited local values and preferences in the decision making process. Guideline developers can assume an international or national, or, alternatively, a localized or specific perspective. By considering the appropriate setting the recommendations could potentially be more acceptable to stakeholders. While the former strategy would be helpful for international organizations such as WHO, those adapting recommendations to a specific setting should consider locally relevant evidence, as was the case in this project [1, 38].
This study has some weaknesses. While the study is based on the development of over 20 guidelines and over 200 recommendations, it is restricted to one geographic setting. Also, limited local information was identified for patient values and preferences. The one related advantage is identifying the necessity of conducting more research on local values and preferences. Second, our definition and eligibility criteria for values and preferences were broad. The inclusion of a variety of study designs resulted in challenges with determining the eligibility of individual studies and the category they belong to. The time and resources spent on systematic reviews of values and preferences varied across guideline topics. We also did not formally assess the certainty or the quality of the evidence in the values and preferences from published studies. As for information about values and preferences from panel members, the collected information was unsystematic, potentially biased, and sometimes difficult to use. Furthermore, we were not able to assess publication bias due to the nature of the study question, study design and the geographic filter we used. While we identified studies with a variety of designs providing relevant evidence, the lack of standardized methods for reporting and identifying the evidence places additional limitations on current guideline development but not on our work.
How to interpret and present information about values and preferences in guidelines
Although the integration of values and preferences is considered standard for trustworthy guideline development processes, using systematic reviews to identify values and preferences in a structured approach is still uncommon [1, 7, 12, 39, 40]. The Saudi Arabian panels weighted the relative importance of outcomes using information from literature reviews, the panel members themselves, and patient representatives. This facilitated adoption, adaptation and creating new recommendations according to local values. The GRADE EtD framework helped facilitate the use of values and preferences information in the decision making process by explicitly calling attention to the criterion when balancing benefits and harms. The approach we used has face validity because the panel members did not identify missing studies on local values and preferences. As guideline methodology is refined, how to define, measure, and incorporate patient values and preferences will evolve.
There are other guideline efforts that consider patient values and preferences in the process of developing recommendations. For example, the National Institute for Health and Clinical Excellence (NICE) also considers the impact of values and preferences on the strength of recommendation. The process includes asking patient representatives to reveal their experience in addition to reviews of qualitative research evidence and cross-sectional surveys. However, NICE does not operationalize values and preferences as the importance of outcomes .
Thus, despite recently increasing numbers of available primary studies and systematic reviews on values and preferences [41–44], they are still rarely used in guidelines. This is likely also a result of poor guidance and definitions for how to incorporate this information appropriately. Our study provides a feasible approach to consider patient values and preferences in guideline development. However, other challenges in using this information remain. This includes accepted approaches to assessing the quality or certainty of evidence which is recognized by the GRADE working group and work is ongoing to develop an approach [31, 45–47]. Furthermore, existing systematic reviews seldom have a clear definition, valid search strategy, or transparent synthesis methods to identify evidence about the relative importance of outcomes. Our experience of using GRADE EtD frameworks, that do not yet routinely include modeling based on preferences, need to be seen in the context of other approaches that routinely include modeling [10, 48].
Although considering the relative importance of health outcomes is essential in informing healthcare decision-making, use of this type of information remains a complex area to integrate. Our experience shows that guidelines in general and GRADE EtD frameworks in particular, lend themselves to the incorporation of this aspect in clinical and public health recommendations. To further facilitate this process a methodologically rigorous and consistent approach for reporting, summarizing and interpreting the information is needed due to the great heterogeneity on the definition, perspective and measurement of values and preferences. We provide an empirical approach to address this concern through systematic reviews and panel members’ input.
Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence
Guidelines International Network
Grading of Recommendations Assessment, Development and Evaluation
Health utility index
Medical Subject Headings
National Institute for Health and Clinical Excellence
Quality of wellbeing
Visual analogue scales
World Health Organization
World Health Organization. WHO Handbook for Guideline Development. 2nd ed. Switzerland: World Health Organization; 2014.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–6.
Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M, Brignardello-Petersen R, Laisaar KT, Kowalski S, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186:E123–142.
Schunemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, Scholten R, Langendam M, Leeflang MM, Akl EA, et al. Development of the GRADE Evidence to Decision (EtD) frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76:89-98. doi: 10.1016/j.jclinepi.2016.01.032.
Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Rada G, Rosenbaum S, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.
Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Vandvik PO, Meerpohl J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089.
Kelson M, Akl EA, Bastian H, Cluzeau F, Curtis JR, Guyatt G, Montori VM, Oliver S, Schünemann HJ. Integrating values and consumer involvement in guidelines with the patient at the center: article 8 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9:262–8.
Cluzeau F, Wedzicha JA, Kelson M, Corn J, Kunz R, Walsh J, Schunemann HJ. Stakeholder involvement: how to do it right: article 9 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9:269–73.
Schunemann H, Fretheim A, Oxman A. Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst. 2006;4:22.
Andrews JC, Schunemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, Rind D, Montori VM, Brito JP, Norris S, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013;66:726–35.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–2.
van der Weijden T, Legare F, Boivin A, Burgers JS, van Veenendaal H, Stiggelbout AM, Faber M, Elwyn G. How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol. Implement Sci. 2010;5:10.
Murad MH, Montori VM, Guyatt GH. Incorporating patient preferences in evidence-based medicine. JAMA. 2008;300:2483. author reply 2483–2484.
MacLean S, Mulla S, Akl EA, Jankowski M, Vandvik PO, Ebrahim S, McLeod S, Bhatnagar N, Guyatt GH. Patient values and preferences in decision making for antithrombotic therapy: a systematic review: antithrombotic therapy and prevention of thrombosis, 9th ed: American college of chest physicians evidence-based clinical practice guidelines. Chest. 2012;141:e1S–23S.
Krahn M, Naglie G. The next step in guideline development: incorporating patient preferences. JAMA. 2008;300:436–8.
Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, Davoli M, Flottorp S, Harbour R, Hill S, et al. Developing and Evaluating Communication strategies to support Informed Decisions and practice based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, et al. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:1490.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94.
Dolan P, Gudex C, Kind P, Williams A. Valuing health states: a comparison of methods. J Health Econ. 1996;15:209–31.
Gafni A, Birch S. Preferences for outcomes in economic evaluation: an economic approach to addressing economic problems. Soc Sci Med. 1995;40:767–76.
Torrance GW. Utility measurement in healthcare: the things I never got to. Pharmacoeconomics. 2006;24:1069–78.
Giacomini MK, Cook DJ, Streiner DL, Anand SS. Using practice guidelines to allocate medical technologies. An ethics framework. Int J Technol Assess Health Care. 2000;16:987–1002.
Hofmann B. Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Assess Health Care. 2005;21:312–8.
Holger Schunemann RM, Jan Brozek, Alonso Carrasco-Labra, Romina Brignardello-Peterson, Wojtek Wiercioch, Saudi Arabian Handbook for Healthcare Guideline Development. http://www.moh.gov.sa/endepts/Proofs/Documents/SA%20Handbook%20_AB%20v28-12-2014_18Dec%202014%20(2).pdf. Accessed 25 Aug 2016.
Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, Brignardello-Petersen R, Neumann I, Falavigna M, Alhazzani W, et al. GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol 2016. (Article in Progress).
Garg AX, Hackam D, Tonelli M. Systematic review and meta-analysis: when one study is just not enough. Clin J Am Soc Nephrol. 2008;3:253–60.
Higgins JPT, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org/. Accessed 17 Mar 2017.
Longworth L, Yang Y, Young T, Mulhern B, Hernandez Alava M, Mukuria C, Rowen D, Tosh J, Tsuchiya A, Evans P, et al. Use of generic and condition-specific measures of health-related quality of life in NICE decision-making: a systematic review, statistical modelling and survey. Health Technol Assess. 2014;18:1–224.
Furlong WJ, Feeny DH, Torrance GW, Barr RD. The Health Utilities Index (HUI) system for assessing health-related quality of life in clinical studies. Ann Med. 2001;33:375–84.
Starkie HJ, Briggs AH, Chambers MG, Jones P. Predicting EQ-5D values using the SGRQ. Value Health. 2011;14:354–60.
DeJean D, Giacomini M, Vanstone M, Brundisini F. Patient experiences of depression and anxiety with chronic disease: a systematic review and qualitative meta-synthesis. Ont Health Technol Assess Ser. 2013;13:1–33.
Torrance GW. Preferences for health states: a review of measurement methods. Mead Johnson Symp Perinat Dev Med. 1982;(20):37-45.
Longworth L, Rowen D. NICE DSU Technical Support Document 10: The use of mapping methods to estimate health state utility values. 2011. Available from http://www.nicedsu.org.uk. Accessed 25 Aug 2016.
Gafni A. The standard gamble method: what is being measured and how it is interpreted. Health Serv Res. 1994;29:207–24.
The Saudi Center for Evidence Based Healthcare (EBHC): Clinical Practice Guidelines. 2015. Available from: http://www.moh.gov.sa/endepts/Proofs/Pages/Guidelines.aspx. Accessed 25 Aug 2016.
Al-Hameed F, Al-Dorzi HM, Shamy A, Qadi A, Bakhsh E, Aboelnazar E, Abdelaal M, Al Khuwaitir T, Al-Moamary MS, Al-Hajjaj MS, et al. The Saudi clinical practice guideline for the diagnosis of the first deep venous thrombosis of the lower extremity. Ann Thorac Med. 2015;10:3–15.
Brozek JL, Bousquet J, Baena-Cagnani CE, Bonini S, Canonica GW, Casale TB, van Wijk RG, Ohta K, Zuberbier T, Schunemann HJ. Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines: 2010 revision. J Allergy Clin Immunol. 2010;126:466–76.
World Health Organization. Estonian Handbook for Guidelines Development. Switzerland: World Health Organization; 2011.
National Institute for Health and Clinical Excellence: The guidelines manual. 2012. London: National Institute for Health and Clinical Excellence. Available from: www.nice.org.uk. Accessed 17 Mar 2017.
Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E. Clinical Practice Guidelines We Can Trust. Washington DC: the National Academy of Sciences; 2011.
Lin OS, Kozarek RA, Gluck M, Jiranek GC, Koch J, Kowdley KV, Irani S, Nguyen M, Dominitz JA. Preference for colonoscopy versus computerized tomographic colonography: a systematic review and meta-analysis of observational studies. J Gen Intern Med. 2012;27:1349–60.
Mohiuddin S, Payne K. Utility values for adults with unipolar depression: systematic review and meta-analysis. Med Decis Mak. 2014;34:666–85.
Peeters Y, Stiggelbout AM. Health state valuations of patients and the general public analytically compared: a meta-analytical comparison of patient and population health state utilities. Value Health. 2010;13:306–9.
Sadique MZ, Legood R. Women’s preferences regarding options for management of atypical, borderline or low-grade cervical cytological abnormalities: A review of the evidence. Cytopathology. 2012;23:161–6.
Bremner KE, Chong CA, Tomlinson G, Alibhai SM, Krahn MD. A review and meta-analysis of prostate cancer utilities. Med Decis Mak. 2007;27:288–98.
Cronin M, Meaney S Fau - Jepson NJA, Jepson Nj Fau - Allen PF, Allen PF. A qualitative study of trends in patient preferences for the management of the partially dentate state. Gerodontology. 2009;26(2):137-42.
Jackson LJ, Auguste P, Low N, Roberts TE. Valuing the health states associated with Chlamydia trachomatis infections and their sequelae: a systematic review of economic evaluations and primary studies. Value Health. 2014;17:116–30.
Nutt DJ, King LA, Phillips LD. Drug harms in the UK: a multicriteria decision analysis. Lancet. 2010;376:1558–65.
We are grateful to McMaster University research librarians Jennifer Lawson and Tamara Navarro for their assistance in developing literature search strategies.
The Ministry of Health, Saudi Arabia and McMaster University, Hamilton, Canada.
The sponsor had no role in the design of the study or interpretation of the results except through the employed authors of this study.
Availability of data and materials
The datasets supporting the conclusions of this article are included within the article and its additional file.
YZ, PA, JB, HS designed the systematic review methodology for patient values and preferences and drafted the manuscript; HS and WW designed the methodology for Saudi Arabian healthcare guideline development. YZ, JB, WW, IE, EA, JM, WA, AC, RLM, RAM, JJR, JJY, CC, ZA, VM, MF, IN, RB, NS, BR, AD, MX, YA, CB, RW, and HS conducted the literature search, screening, and data abstraction, and consulted the panellists in the panel meetings. All authors read and approved the final manuscript.
Several authors are members of the GRADE working group and have helped developing the Evidence to Decision Frameworks.
Consent for publication
Ethics approval and consent to participate
Not applicable. This study does not involve de novo patient data collection and describes the process of using preferences and values in guideline development. Information presented here is based on literature reviews and extracts from already published material in the GRADE EtD frameworks, that were approved by all panel members (http://www.moh.gov.sa/endepts/Proofs/Pages/Guidelines.aspx). No patient informed consent and Institutional Review Board approval have been sought.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Search strategy. (DOCX 15 kb)