Open Access

Development and use of a content search strategy for retrieving studies on patients' views and preferences

  • Anna Selva1, 2, 3Email author,
  • Ivan Solà2, 4,
  • Yuan Zhang5,
  • Hector Pardo-Hernandez2, 4,
  • R. Brian Haynes5, 6,
  • Laura Martínez García2,
  • Tamara Navarro5,
  • Holger Schünemann5 and
  • Pablo Alonso-Coello2, 5, 6Email author
Health and Quality of Life Outcomes201715:126

https://doi.org/10.1186/s12955-017-0698-5

Received: 15 July 2016

Accepted: 1 June 2017

Published: 30 August 2017

Abstract

Background

Identifying scientific literature addressing patients’ views and preferences is complex due to the wide range of studies that can be informative and the poor indexing of this evidence. Given the lack of guidance we developed a search strategy to retrieve this type of evidence.

Methods

We assembled an initial list of terms from several sources, including the revision of the terms and indexing of topic-related studies and, methods research literature, and other relevant projects and systematic reviews. We used the relative recall approach, evaluating the capacity of the designed search strategy for retrieving studies included in relevant systematic reviews for the topic. We implemented in practice the final version of the search strategy for conducting systematic reviews and guidelines, and calculated search’s precision and the number of references needed to read (NNR).

Results

We assembled an initial version of the search strategy, which had a relative recall of 87.4% (yield of 132/out of 151 studies). We then added some additional terms from the studies not initially identified, and re-tested this improved version against the studies included in a new set of systematic reviews, reaching a relative recall of 85.8% (151/out of 176 studies, 95% CI 79.9 to 90.2). This final version of the strategy includes two sets of terms related with two domains: “Patient Preferences and Decision Making” and “Health State Utilities Values”. When we used the search strategy for the development of systematic reviews and clinical guidelines we obtained low precision values (ranging from 2% to 5%), and the NNR from 20 to 50.

Conclusions

This search strategy fills an important research gap in this field. It will help systematic reviewers, clinical guideline developers, and policy-makers to retrieve published research on patients’ views and preferences. In turn, this will facilitate the inclusion of this critical aspect when formulating heath care decisions, including recommendations.

Keywords

Patient preference Decision-making Information storage and retrieval Search engine Terminology as topic

Background

Health care decision-making is complex and involves considering several criteria simultaneously. Recently, the GRADE working group has published a series of manuscripts about the development of frameworks to structure and make more explicit the process of moving from evidence to decisions (including recommendations) [13]. Among the suggested criteria to take into consideration when adopting these frameworks, two are particularly relevant to patients. One is the relative importance that patients (or those affected by the decision) place on the main outcomes, crucial when balancing desirable and undesirable effects [4, 5]. The other is whether patients (or other stakeholders) find acceptable the intervention (or the alternative considered) [6, 2]. The GRADE working group used the concept "values and preferences" and conceptualized it as “the processes that individuals use in considering the potential benefits, harms, costs, limitations, and inconvenience of the management options in relation to one another” [7], is closely related with both the relative importance of outcomes importance and acceptability and can be considered an umbrella term.

The consideration of how much patients or those affected by a decision value outcomes is an often-ignored aspect when healthcare panels formulate recommendations (or other type of decisions) [811]. In particular, healthcare guidelines fail to properly incorporate stakeholders’ views [1216]. One of the potential reasons for this poor uptake is likely to be the difficulty in identifying relevant scientific literature, due to the wide range of potentially relevant study designs [17], and their poor indexing [18].

There are different ways of capturing peoples’ views about the importance of outcomes (Table 1). Outcomes can be operationalised as health state utility values, which can be obtained using direct or indirect techniques. The former include the standard gamble [19], the time trade-off [20], or visual analogue scales [21], among others. The latter include multi-attribute instruments in which the relative importance is obtained by regression analysis from quality of life questionnaires, such as the EQ-5D (EuroQol), the SF-6 health survey, or the Health Utility Index (HUI-2 and HUI-3) [22]. People’s views on outcome importance can also be elicited using non-utility quantitative measures, typically provided by surveys or questionnaires [23]. Other non-utility measures include direct choice methods [24] such as those used in decision aids [25], which can provide information about the relative disutility of outcomes. Lastly, qualitative research can provide information on preferences, opinions, perceptions, and attitudes [26].
Table 1

Measures capturing people’s views about healthcare outcomes

Utility measures

Non-utility, quantitative measures

Qualitative findings

Direct techniques

Indirect techniques

Direct choice techniques

Interviews

Discussion groups

Standard gamble, time trade off, or visual analogue scales.

Pre-scored multi-attribute instruments [EQ-5D (euroQoL), the SF-6 health survey, or the health utility index (HUI-2 and HUI-3)]

Decision aids and direct choice studies.

Surveys or questionnaires.

Structured, semi-structured, unstructured, or in-depth.

Focus groups.

Given the wide range of study designs, it is not surprising that standard strategies for systematically eliciting scientific literature on views and preferences are not yet available. Despite the availability of some search filters, these do not cover all the relevant aspects that need to be considered. Some fall short in capturing the methodologies available to obtain the different possible measures [27, 28], while others focus specifically on one type of measure [29, 30].

Systematically identifying evidence addressing views and preferences through standardised approaches is essential because it would facilitate the conduct of systematic reviews while enhancing its efficiency, structure and transparency and would allow guideline panels to develop evidence-based recommendations [5, 31]. We therefore developed a content search strategy to systematically identify this type of evidence in PubMed.

Methods

Development of the initial search strategy

Using an iterative process we developed an initial list of terms, either controlled vocabulary (MeSH terms) and text words. We first explored how related research on this topic was indexed in Pubmed [3239] and checked search strategies of relevant systematic reviews [31, 4044]. We also inspected previous search strategies or filters [2729] as well as other searches used in other related projects and experiences [4547].

Three authors (AS, IS, and PAC) assembled the initial list of all the terms. This list was refined through discussion among all authors, some of whom have extensive experience in healthcare guideline development and methodology.

Testing the performance of the search strategy

We assessed the search strategy performance measuring its relative recall, which allows evaluating and refining the search filters performance against the set of studies eligible for inclusion in relevant systematic reviews [4852]. This approach considers that the methods implemented to identify studies in a systematic review are an efficient alternative to the traditional gold standard obtained from handsearching suggested by other authors [49]. Then, the articles identified in multiple information sources in systematic reviews are representative of the available evidence about a topic, and then the included studies can be used as the reference set to assess the performance of a new search strategy. Relative recall is the proportion of articles that a specific search retrieves of the total relevant studies identified by a systematic review [50]. (Table 2).
Table 2

Relative recall calculation

Relative recall = Number studies included in the SRs retrieved by the search × 100

Number of studies included in the SRs indexed in PubMed

To assess the relative recall of our search strategy we first selected a convenience sample of six relevant systematic reviews that included different and relevant study designs, methods and measures related to our topic of interest [31, 4044]. All six reviews had clear inclusion criteria and provided a clear list of included studies. Their scope and methods are summarized in Table 3.
Table 3

Characteristics of the reference set of systematic reviews

Systematic reviews

Topic

Techniques for elicitation of patient views and preferences of included studies

McLean 2012 [29]

Antithrombotic therapy

Elicitation of health state utility values; Choices patients made when presented with decision aids; Interviews; Surveys.

Peasgood 2010 [41]

Breast cancer

Elicitation of health state utility values.

Petrillo 2011 [42]

COPD

Elicitation of health state utility values.

Bremmer 2007 [38]

Prostate cancer

Elicitation of health state utility values.

Brooker 2013 [39]

COPD

Choices patients made when presented with decision aids; Questionnaires, surveys; Structured and semi-structured interviews.

Morton 2010 [40]

Chronic kidney disease

Qualitative research: focus groups, interviews

We determined which studies included in the reference set of systematic reviews were indexed in PubMed in order to obtain a valid denominator for calculating the relative recall. We aggregated the PubMed unique identifier of each study (PMID) using the Boolean operator OR, and combined them with the search string using the Boolean operator AND.

We determined how many of the primary studies included in the relevant systematic reviews of interest that were indexed in PubMed were retrieved using our search strategy (Table 2). We expressed recall as a proportion and also calculated its 95% confidence interval. We examined the terms used by the primary studies that were not retrieved by the search strategy, and adapted the search accordingly adding the more pertinent terms. We included the search strategy obtained in Additional file 1.

To avoid biases resulting from the initial selection of the reference systematic reviews used to calculate the relative recall and improve the designed search strategy we re-tested its performance against a set of 10 new systematic reviews, similar to those used in the previous step. We conducted a search in PubMed in order to locate systematic reviews of utility measures and other measures. We randomly selected a set of these reviews according to a sequence generated using Microsoft Excel. For the selected reviews we repeated the process described above to calculate the relative recall from our search strategy.

Search strategy implementation in practice

The search strategy obtained was implemented in the following knowledge synthesis projects: 1) the development of a systematic review about the relative importance of outcomes in COPD patients; 2) the development of a practice guideline for the ARIA (Allergic Rhinitis and its Impact on Asthma) initiative; and 3) the update of a clinical guideline on pregnancy and postnatal care [53]. Additionally, we used this strategy to identify systematic reviews for the development of a repository of systematic reviews and primary studies on patients and other stakeholders’ views about health care, linked to Epistemonikos.

For each of these projects, titles and abstracts retrieved using this strategy were independently screened by two researchers using a reference-managing software (EndNote). Eligibility was discussed through standard methods according the inclusion criteria for each of the projects described above. For the purposes of this paper we calculated the precision of the search strategy for each project (percentage of relevant articles in the complete set of articles retrieved) and the number of references needed to read to obtain a relevant reference (NNR) [5456] (Table 4). We calculated these estimates for the complete search string and for each of its domains separately, accounting for the different approaches to capture patients’ views and preferences.
Table 4

Precision and Number Needed to Read (NNR)

Precision: (number of relevant studies identified/total number of references retrieved from the search) X 100

Number Needed to Read (NNR): number of references needed to screen to find one relevant study (1/precision).

Results

Search strategy development and performance testing

We assembled an initial version of the search strategy, grouping its terms in two domains corresponding to major issues in the topic (“patient preferences and decision making” and “health state utilities”.

We tested the relative recall of the search initial version against the included studies of an initial set of six systematic reviews. The reviews included 162 studies of which 151 (93%) were indexed in PubMed. This initial version of the search strategy retrieved 132 of the reference studies, resulting in a relative recall of 87.4% (95% CI 81.2% to 91.8%).

After assessing the studies not retrieved by the initial version of the search strategy we added some new terms, all related to the patient preferences and decision-making domain. We include the improved version of the search strategy (Additional file 1) that showed a relative recall of 92% (95% CI 86.4% to 95.4%) (Table 5). We also obtained the relative recall for each of the two domains form the search strategy, with a better performance for the block of terms related to patient preferences and decision-making, compared to the terms related to health state utility values (85.4% versus 44.4%).
Table 5

Performance of the search strategy

Systematic review

Topic areas

Studies included (n)

Studies indexed in Pubmed (n)

Studies retrieved by the search strategy n (%)

Patient preferences and decision making domain n (%)

Health state utility values domain n (%)

Initial test of the relative recall

 McLean 2012 [29]

Antithrombotic therapy

50

49 (98%)

48 (97.9)

48 (97.9)

3 (6.1)

 Peasgood 2010 [41]

Breast cancer

50

44 (88%)

36 (81.8)

32 (72,73)

34 (77,27)

 Morton 2010 [40]

Chronic kidney disease

18

15 (83%)

15 (100.0)

15 (100.0)

1 (6.7)

 Petrillo 2011 [42]

COPD

11

11 (100%)

10 (90.9)

5 (45.4)

9 (81.8)

 Brooker 2013 [39]

COPD

12

12 (100%)

11 (91.7)

11(91.7)

2 (16.7)

 Bremmer 2007 [38]

Prostate cancer

21

20 (95%)

19 (95.0)

18(90.0)

18 (90.0)

Relative recall results RR*(95% CI) (n)

162

151 (93%)

92% (86.4-95.4) (139)

85.4% (78.9-90.1) (129)

44.4% (36.4-52.3) (67)

Re-test

 Shaw 2016 [57]

CV disease and daibetes

14

13 (93%)

11 (86,62)

11 (86,62)

0 (0.0)

 Smith 2015 [58]

Phyisical activity in joint replacement

13

13 (100%)

9 (69,23)

9 (69,23)

1 (7,69)

 Finlayson 2015 [59]

Metastatic cancer

15

15 (100%)

8 (53,3)

8 (53,3)

0 (0,0)

 Köberich 2015 [60]

Patient-centered nursing care

12

12 (100%)

11 (91,6)

11 (91,6)

0 (0,0)

 Loewen 2017 [62]

Antithrombotic therapy in atrial fibr.

28

26 (93%)

25 (96,15)

25 (96,15)

11 (42,3)

 Cowley 2016 [61]

Esophagogastric cancer

2

2 (100%)

0 (0)

0 (0)

0 (0)

 Brown 2017 [63]

Lung cancer

27

27 (100%)

23 (85,15)

11 (40,74)

20(74,07)

 Jeong 2016 [73]

Colorectal cancer

57

56 (98%)

53 (94,64)

26 (46,42)

51 (91,07)

 Carter 2015 [65]

Gastric cancer

8

7 (87.5%)

7 (100)

3 (42,8)

7 (100)

 Ward 2017 [64]

Brain trauma

5

5 (100%)

4 (80)

4 (80)

2 (40)

Relative recall results RR*(95% CI) (n)

181

176 (97%)

85,8% (79.9-90.2) (151)

61.4 (54.0-68.3) (108)

52.3% (44.9-59.5) (92)

Total. Relative recall results RR*(95% CI) (n)

343

327 (95%)

88,7% (84.8-91.7) (290)

72.5% (67.4-77.0) (237)

48.6% (43.2-54.0) (159)

*RR: Relative Recall

We then tested the relative recall of the improved version of the search strategy against the studies included in a new set of 10 systematic reviews that included a total of 181 primary studies [5765]. The relative recall for this second test was 85,8% (95% CI 79.9% to 90.2%) (Table 5). In this second test the difference in the relative recall between the two domains was lower (61.4% for patient preferences versus 52.3% for health state utility values) (Table 5).

If we consider altogether the studies included in the 16 systematic reviews used as reference at the two steps of the test (n = 327), then the relative recall of the search strategy was of 88.7% (95% CI 84.8% to 91.7%), being the domain on patient preferences and decision making the one that retrieves a higher proportion of relevant studies (relative recall of 72.5% (95% CI 67.4% to 77.0).

Search strategy implementation

We used this search strategy in the development of a systematic review and two clinical guidelines that considered the topic of patients’ views and preferences. For these projects we were not able to obtain a reference standard and calculate the relative recall of our strategy. Instead, we obtained its precision and NNR as valid and relevant performance indicators. The results derived from each search showed a low precision for the strategy (Tables 6, 7 and 8).
Table 6

Use of the search in the development of a systematic review on COPD

COPD systematic reviewa

Retrieved references (n)

Relevant references (n)

Relevant studies on non-utility measures

Relevant studies on utiltiy measures

Relevant qualitative research

Precision (%) 95% CI

NNR

Global search strategy

12,574

252

93

66

110

2.0 (1.77-2.26)

49.8

Patient preferences domain

11,789

203

87

23

93

1.52 (1.32-1.76)b

65.5b

Utility values domain

1008

70

11

59

2

5.85 (4.56-8.08)c

17.1c

aSearch period: from inception until 01/01/2015

bThe precision for the patient preferences domain is calculated as: (number of non-utility studies + qualitative studies)/number of retrieved references by the patient preferences domain. The NNR is calculated as: number of retrieved references by the patient preferences domain/(number of non-utility studies + qualitative studies)

cThe precision for the utility values domain is calculated as: number of studies on utility measures/number of retrieved references by the utility values domain. The NNR is calculated as: number of retrieved references by utility values domain/number of studies on utility measures

Table 7

Use of the search strategy in the development of a healthcare guideline

Allergic rhinitis in asthma guidelinea

Retrieved references (n)

Relevant references (n)

Relevant studies on non-utility measures

Relevant studies on utiltiy measures

Relevant qualitative research

Precision (%) 95% CI

NNR

Global search strategy

1560

33

27

6

0

2,12 (1.51-2.96)

47,27

Patient preferences domain

1487

31

27

4

0

1,82b (1.25-2.63)

55,07b

Utility values domain

92

7

1

6

0

6,52c (3.02-13.51)

15,33c

aSearch period: from inception until May 1st 2015

bThe precision for the patient preferences domain is calculated as: (number of non-utility studies + qualitative studies)/number of retrieved references by patient preferences domain. The NNR is calculated as: number of retrieved references by patient preferences domain/(number of non-utility studies + qualitative studies)

cThe precision the utility values domain is calculated as: number of studies on utility measures/number of retrieved references by utility values domain. The NNR is calculated as: number of retrieved references by utility values domain/number of studies on utility measures

Table 8

Use of the the search strategy in the update of a healthcare guideline

Care during pregnancy and puerperium guidelinea

Retrieved references (n)

Relevant references (n)

Relevant studies on non-utility measures

Relevant studies on utility measures

Relevant Qualitative research

Precision (%) 95% CI

NNR

Global search strategy

668

19

9

1

9

2.84 (1.83-4.40)

35.2

Patient preferences domain

653

19

9

1

9

2.76b (1.75-4.31)

36.3b

Utility values domain

22

2

2

0

0

0

NA

aSearch period: First cycle from 01/01/2012- 31/08/2014; Second cycle from 01/09/2014 to 28/02/2015; Third cycle from 01/03/2015 to 31/08/2015

bThe precision for the patient preferences domain is calculated as: (number of non-utility studies + qualitative studies)/number of retrieved references by patient preferences domain. The NNR is calculated as: number of retrieved references by patient preferences domain/(number of non-utility studies + qualitative studies)

The search used for the systematic review about the relative importance of outcomes in COPD patients showed a low precision (2%, resulting from 252 relevant references from the 12,574 retrieved) and a NNR of 50 references. When the search was used to conduct a review for the development of a guideline on allergic rhinitis the strategy showed a precision of 2.12% (33 relevant references out of 1560 retrieved) with a NNR of 47 references. The strategy used to complete a literature surveillance to update 123 recommendations on pregnancy care from a clinical guideline showed a similar precision 2.84% (19 relevant references out of 668 retrieved), with a NNR of 35 references. Finally the precision of the search for the development of a database on patients’ and other stakeholders’ views about health care was 5% (314 relevant references from 6231 retrieved) with a NNR of 20 references.

Discussion

Main findings

We have developed a content search strategy to systematically identify studies addressing patients’ views and preferences in Pubmed. The strategy includes terms to retrieve references about utilities (and relative disutilities), other quantitative measures of preferences, decisions distributions, and findings from qualitative studies.

We used the relative recall approach to test the performance of this strategy against a reference set of studies included in relevant systematic reviews. After an initial test of the strategy and the addition of some terms to the search string we obtained a relative recall of 85, 8%. We designed the search strategy to use two independent blocks of terms, one related to patients’ preferences and decision making, and one to utilities values. The former showed a better recall than the latter.

When conducting reviews the most relevant performance indicators for the search are relative recall (equivalent to sensitivity) and precision (equivalent to positive predictive value). We tested the precision of the search strategy in a series of knowledge synthesis projects. In all the cases the precision was low, ranging from 2% to 5%. These findings are in concordance with a standard practice in designing search strategies for synthesis of scientific evidence, where the optimization of recall comes at the price of a high reference screening burden [66, 67].

Our results in the context of previous research

The concept of how patients value different health care issues is broad and complex, and can be interpreted from multiple approaches, with several research designs providing valuable information. This variability of conceptual frameworks to approach the topic and study methodologies makes it challenging to identify relevant studies. Furthermore, this area of knowledge is continuously evolving and the terminology used is still immature, adding further challenges to the searching process. Most studies use somewhat different terminology to refer to the same issue, and there is lack of systematic indexing for this topic [17].

The search strategy we present takes into consideration the complexity of conducting an evidence synthesis about this topic, and the need to use a broad vocabulary to ensure its comprehensiveness. We therefore incorporated search terms to retrieve the different measures available (e.g. utilities or qualitative findings), elicited directly and indirectly. It also includes terms to retrieve evidence on preferences, attitudes to health, patient decisions, participation, satisfaction, views or perceptions.

Other researchers have designed search strategies responding to the common necessity to identify studies on this topic but from different perspectives and scopes, making difficult an adequate comparison of their performance. The Scottish Intercollegiate Guidelines Network (SIGN) developed a search strategy for patient issues [27], containing over 200 terms that relate mostly to patients, carers, and relatives’ feelings, emotions, perceptions, concerns, and satisfaction, as well as evidence drawn from support, self-help, and social groups. To our knowledge this filter has not been validated and no details on how it was developed have been published [68, 69]. The Knowledge Institute of Medical Specialists (KiMS) developed a literature search filter specific for patients’ knowledge, views, and values [27]. This filter used the SIGN search as a starting point and was subsequently refined to focus on patient experiences, information needs, unfulfilled needs, preferences, participation in decision-making and satisfaction. The researchers assembled a gold standard to assess the sensitivity, specificity, and precision. Another recently published study [30], used a gold standard approach to validate a set of filters to retrieve studies focusing specifically in patients’ preferences for treatment outcomes, with an excellent performance. To our knowledge, the only filter that also used the relative recall approach to test its performance was restricted to the identification of studies that report on health state utilities [29]. The filter performance was similar to the obtained in our experience, with a slightly better recall (91%) and a lower precision (0.3%).

Our search strategy used in real life experiences obtained a low precision with an appreciable NNR. This is not surprising given the broad array of study designs, the methodologies that can be informative, and the broad vocabulary that has to be considered to improve the search comprehensiveness, which inherently increases the pool of references that may be eligible [17, 57]. This is consistent with the limitations of search strategies to retrieve studies in other fields such as qualitative research. A recent methodological review [70] identified four qualitative filters, developed for MEDLINE (using PubMed), EMBASE (Ovid), PsycINFO (Ovid), and CINAHL (Ovid). While the sensitivity and specificity of these filters were favourable, there are serious concerns with the comprehensiveness of the included terms [70, 71]. Similarly, a recent study [72] assessed the performance of search filters of qualitative research, including eight combinations for MEDLINE, seven for CINAHL, and four for Social Science Citation Index filters. Authors reported that overall precision was disappointingly low, and variable depending on the topic searched, either COPD or early breast cancer.

The design of a search strategy to retrieve studies with a low prevalence in the scientific literature, using an approach that aims to maximize sensitivity will invariably penalize its precision. Consistently, the precision derived from the use of our search is in the range of the precision showed by the searches in samples of systematic reviews [67].

Limitations and strengths

The main strength of our study is that we adopted an explicit methodology to test the performance of our search strategy. We developed the search strategy in different steps, testing the relative recall and precision from independent sets of studies. First, we assembled terms in a comprehensive process, calculated the relative recall of the resulting search against a reference set of relevant studies that allowed us to improve the initial search string. We also tested the final strategy in different projects allowing us to value other important performance characteristics in the development of evidence syntheses.

It could be argued that the validity of relative recall is strongly conditioned by the capacity of the systematic reviews used as reference to adequately identify their included studies. We tried to mitigate this shortcoming retesting the search with a comprehensive set of relevant reviews, that provided at least 100 pertinent reference studies, against which the filter was compared [48, 49].

Implications for practice and research

The availability of a standardised approach to retrieve studies on patients’ views and preferences will likely contribute to ease the use of this issue in the process of conducting systematic reviews, health technology assessments and clinical guidelines. This search strategy fills an important methodological gap and will enable the development of systematic reviews in this field.

In order to lower the burden associated with identifying this type of evidence researchers could implement the proposed search strategy by initially focusing on specific measures (e.g. utilities or qualitative findings) or study designs (e.g. standard gamble, time trade off, or visual analogue scales). Other less-burdensome strategies could entail focusing just on systematic reviews or on more recent or context-specific studies.

There are some valuable implications for future research after the development of this content search strategy. There is a need to validate our strategy with a gold standard approach to evaluate other performance characteristics such as specificity, and to obtain the terms with the best yield. Additionally, the adaptation and evaluation in other databases different from PubMed should also be conducted. Finally, it would be important to determine the relative performance of alternative strategies in this field.

Conclusions

The proposed content search strategy designed for PubMed will help researchers to systematically identify relevant studies addressing patients’ views and preferences. This will facilitate the inclusion of this critical aspect when formulating heath care decisions, including recommendations.

Abbreviations

COPD: 

Chronic Obstructive Pulmonary Disease

Declarations

Acknowledgements

Anna Selva is a doctoral candidate in Public Health and Methodology of Biomedical Research, at the Department of Pediatrics, Obstetrics, Gynaecology and Preventive Medicine at Universitat Autònoma de Barcelona, Spain.

The authors would to thank Dr. Gordon Guyatt (Department of Clinical Epidemiology & Biostatistics, Department of Medicine, McMaster University, Hamilton, ON, Canada) for his invaluable input for the completion of this project.

Funding

This project was funded by research grants from the Instituto de Salud Carlos III through projects PI13/02848 and PI14/02006 (co-funded by the European Regional Development Fund/European Social Fund “Investing in your future”). Pablo Alonso-Coello is funded by a Miguel Servet research contract from the Instituto de Salud Carlos III (CP16/00137).

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Authors’ contributions

Conception and design: AS, IS, YZ, BH, HS, PAC. Undertaking searches: AS, IS, YZ, TN. Extracting data: AS, IS, YZ, HPH. Data analysis and interpretation: all authors. Writing the manuscript: AS, IS, HPH, PAC. Commenting on and editing the manuscript: all authors. Approving final version of the manuscript: all authors.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Clinical Epidemiology and Cancer Screening, Corporació Sanitària Parc Taulí
(2)
Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau (IIB Sant Pau)
(3)
Research Network on Health Services in Chronic Diseases (REDISSEC)
(4)
CIBER Epidemiología y Salud Pública, (CIBERESP)
(5)
Department of Clinical Epidemiology & Biostatistics, McMaster University
(6)
Health Information Research Unit, Department of Clinical Epidemiology & Biostatistics, McMaster University

References

  1. Alonso-Coello P, Schunemann H, Moberg J, Brignardello-Petersen R, Akl E, Marina D, et al., GRADE Working Group. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. doi:10.1136/bmj.i2016.View ArticlePubMedGoogle Scholar
  2. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl E, Marina D, et al. GRADE working group. GRADE evidence to decision (EtD) frameworks: 2. Clinical practice guidelines. BMJ. 2016;353:i2089. doi:10.1136/bmj.i2089.View ArticlePubMedGoogle Scholar
  3. Schünemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, et al., GRADE Working Group. Development of the GRADE evidence to decision (EtD) frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76:89–98.Google Scholar
  4. Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66:719–25.View ArticlePubMedGoogle Scholar
  5. Krahn M, Naglie G. The next step in guideline development: incorporating patient preferences. JAMA. 2008;300:436–8.View ArticlePubMedGoogle Scholar
  6. Mühlbacher AC, Juhnke C. Patient preferences versus physicians’ judgement: does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11:163–80.View ArticlePubMedGoogle Scholar
  7. Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Alonso Coello P, et al. GRADE guidelines: 15. Going from evidence to recommendation- determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013;66:726–35.View ArticlePubMedGoogle Scholar
  8. Chong C, Chen I, Naglie C, Krahn M. Do clinical practice guidelines incorporate evidence on patient preferences? Med Dec Making. 2007;27:E63–4.Google Scholar
  9. Elwyn G, Légaré F, Edwards A, Weijden van der T, May C. Arduous implementation: does the normalisation process model explain why it’s so difficult to embed decision support technologies for patients in routine clinical practice. Implement Sci. 2008;3:57. doi:10.1186/1748-5908-3-57. PubMed CentralView ArticlePubMedView ArticlePubMedPubMed CentralGoogle Scholar
  10. Légaré F, Ratté S, Gravel K, Graham ID. Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals’ perceptions. Pat Educ Couns. 2008;73:526–35. doi:10.1016/j.pec.2008.07.018.View ArticleGoogle Scholar
  11. Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst. 2006;4:22. doi:10.1186/1478-4505-4-22.View ArticlePubMedPubMed CentralGoogle Scholar
  12. Acuña-Izcaray A, Sánchez-Angarita E, Plaza V, Rodrigo G, Montes de Oca M, Gich I, et al. Quality assessment of asthma clinical practice guidelines: a systematic appraisal. Chest. 2013;144(2):390–7. doi:10.1378/chest.12-2005.
  13. Alonso-Coello P, Irfan A, Solà I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19:6 e58.Google Scholar
  14. Brouwers M, Kho ME, Browman GP, Cluzeau F, Feder G, Fervers B, et al., on behalf of the AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J. 2010;182:E839–42. doi:10.1503/cmaj.090449.
  15. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172(21):1628–33.View ArticlePubMedGoogle Scholar
  16. Selva A, Sanabria AJ, Pequeño S, et al. Incorporationg patients’ views in guideline development: a systematic review of guidance documents. Under revision, J Clin Epidemiol 2017. Available online 1 June 2017. In Press, Accepted Manuscript. doi:10.1016/j.jclinepi.2017.05.018.
  17. van der Weijden T, Legare F, Boivin A, et al. How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol. Implement Sci. 2010;5:10.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014;9(6):e99111. doi:10.1371/journal.pone.0099111.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Feeny D, Furlong W, Saigal S, Sun J. Comparing directly measured standard gamble scores to HUI2 and HUI3 utility scores: group- and individual-level comparisons. Soc Sci Med. 2004;58(4):799–809.View ArticlePubMedGoogle Scholar
  20. Ariza-Ariza R, Hernandez-Cruz B, Carmona L, Dolores Ruiz-Montesinos M, Ballina J, Navarro-Sarabia F. Assessing utility values in rheumatoid arthritis: a comparison between time trade-off and the EuroQol. Arthritis Rheum. 2006;55(5):751–6.View ArticlePubMedGoogle Scholar
  21. Rashidi AA, Anis AH, Marra CA. Do visual analogue scale (VAS) derived standard gamble (SG) utilities agree with health utilities index utilities? A comparison of patient and community preferences for health status in rheumatoid arthritis patients. Health Qual Life Outcomes. 2006;4:25. doi:10.1186/1477-7525-4-25.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Bosch JL, Hunink MG. Comparison of the health utilities index mark 3 (HUI3) and the EuroQol EQ-5D in patients treated for intermittent claudication. Qual Life Res. 2000;9(6):591–601.View ArticlePubMedGoogle Scholar
  23. van der Bruel A, Jones C, Yang Y, Oke J, Hewitson P. People’s willingness to accept overdetection in cancer screening: population survey. BMJ. 2015;350:h980. doi:10.1136/bmj.h980.View ArticlePubMedPubMed CentralGoogle Scholar
  24. Kipp R, Lehman J, Israel J, Edwards N, Becker T, Raval AN. Patient preferences for coronary artery bypass graft surgery or Percutaneous intervention in multi-vessel coronary artery disease. Catheter Cardiovasc Interv. 2013;82(2):212–8. doi:10.1002/ccd.24399.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Hersch J, Jansen J, Barratt A, Irwig L, Houssami N, Jacklyn G, et al. Overdetection in breast cancer screening: development and preliminary evaluation of a decision aid. BMJ Open. 2014;4(9):e006016.Google Scholar
  26. Hersch Hersch J, Jansen J, Barratt A, Irwig L, Houssami N, Howard K, et al. Women’s views on overdiagnosis in breast cancer screening: a qualitative study. BMJ. 2013;346:f158. doi:10.1136/bmj.f158.
  27. SIGN [website]. Search filters. [last update 16th May 2014]. Edimburg (Scotland). [accessed 1 Jul 2015). Available at: http://www.sign.ac.uk/search-filters.html.
  28. Wessels M, Hielkema L, van der Weijden T. How to identify existing literature on patients’ knowledge, views, and values: the development of a validated search filter. J Med Libr Assoc. 2016;104(4):320–4.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Arber M, Garcia S, Veale T, Edwards M, Shaw A, Glanville J. Sensitivity of a Search Filter Designed to Identify Studies Reporting Health State Utility Values. HTAi 12th Annual Conference. 15-17 June 2015, Oslo.Google Scholar
  30. van Hoorn R, Kievit W, Booth A, Mozygemba K, Lysdahl KB, Refolo P, et al. The development of PubMed search strategies for patient preferences for treatment outcomes. BMC Med Res Methodol. 2016;16:88. doi:10.1186/s12874-016-0192-5.
  31. MacLean S, Mulla S, Akl EA, et al. Patient values and preferences in decision making for antithrombotic therapy: a systematic review: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141(2 Suppl):e1S–23S.View ArticlePubMedPubMed CentralGoogle Scholar
  32. Arnold D, Girling A, Stevens A, et al. Comparison of direct and indirect methods of estimating health state utilities for resource allocation: review and empirical analysis. BMJ. 2009;339:b2688.View ArticlePubMedPubMed CentralGoogle Scholar
  33. Dolan P, Gudex C, Kind P, et al. Valuing health states: a comparison of methods. J Health Econ. 1996;15:209–31.View ArticlePubMedGoogle Scholar
  34. Drummond M. Introducing economic and quality of life measurements into clinical studies. Ann Med. 2001;33:344–9.View ArticlePubMedGoogle Scholar
  35. Drummond MF, Sculpher MJ, Torrance GW, O’brien BJ, Stoddart GL. Cost-utility analysis. Methods for the economic evaluation of health care programmes. 3rd ed. Oxford: Oxford University Press; 2005. p. 137–209.Google Scholar
  36. Montori V, Devereaux P, Straus S, Haynes B, Guyatt G. Decision making and the patient. In: Guyatt G, Rennie D, Meade M, Cook D, editors. The users’ guides to the medical literature: a manual for evidence-based clinical practice. 2nd ed. New York, NY: McGraw-Hill; 2008.Google Scholar
  37. Muskopf J. Utility Assessment. From Chumney EC and Simpson KN. Methods and Designs for outcomes research. American Society of Health-System Pharmacist. 2006.Google Scholar
  38. Papaioannou D, Braxzier JE, Paisley S. (2011) NICE DSU technical suport document 9: The identification, review and synthesis of Health State utility values from the literature Available from https://www.ncbi.nlm.nih.gov/books/NBK425822/. Accessed 2 June 2017.
  39. Torrance GW. Measurement of health state utilities for economic appraisal. J Health Econom. 1986;5:1–30.View ArticleGoogle Scholar
  40. Bremmer KE, Chong CAKY, Tomlinson G, Alibhai SMH, Krahn MD. A review and meta-analysis of Prostate cancer utilities. Med Decis Mak. 2007;27:288.View ArticleGoogle Scholar
  41. Brooker AS, Carcone S, Witteman W, Krahn M. Quantitative patient preference evidence for health technology assessment: a case study. Int J Technol Assessment Health Care. 2013;29(3):290–300.View ArticleGoogle Scholar
  42. Morton RL, Tong A, Howard K, Snelling P, Webster ACBMJ. The views of patients and carers in treatment decision making for chronic kidney disease: systematic review and thematic synthesis of qualitative studies. BMC. 2010;340:c112. doi:10.1136/bmj.c112. View ArticleGoogle Scholar
  43. Peasgood T, Ward SE, Brazier J. Health-state utility values in breast cancer. Expert Rev Pharmacoecon Outcomes Res. 2010;10:553–66.View ArticlePubMedGoogle Scholar
  44. Petrillo J, van Nooten F, Jones P, Rutten-van MM. Utility estimation in chronic obstructive pulmonary disease. A preference for change? PharmacoEconomics. 2011;29:917–32.View ArticlePubMedGoogle Scholar
  45. Alfadda AA, Al-Dhwayan MM, Alharbi AA, Al Khudhair BK, Al Nozha OM, Al-Qahtani NM, et al. The Saudi clinical practice guideline for the management of overweight and obesity in adults. Saudi Med J. 2016;37(10):1151–62. doi:10.15537/smj.2016.10.14353.
  46. Al-Hameed F, Al-Dorzi HM, AlMomen A, Algahtani F, AlZahrani H, AlSaleh K, et al. Prophylaxis and treatment of venous thromboembolism in patients with cancer: the Saudi clinical practice guideline. Ann Saudi Med. 2015;35(2):95–106. doi:10.5144/0256-4947.2015.95.
  47. Al-Hameed FM, Al-Dorzi HM, Al-Momen AM, Algahtani FH, Al-Zahrani HA, Al-Saleh KA, et al. The Saudi clinical practice guideline for the treatment of venous thromboembolism. Outpatient versus inpatient management. Saudi Med J. 2015;36(8):1004–10. doi:10.15537/smj.2015.8.12024.
  48. Durão S, Kredo T, Volmink J. Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method. J Clin Epidemiol. 2015;68(6):610–6. doi:10.1016/j.jclinepi.2015.02.005. Epub 2015 Feb 18View ArticlePubMedGoogle Scholar
  49. Sampson M, Zhang L, Morrison A, et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Medl Res Methodol. 2006;6:33. doi:10.1186/1471-2288-6-33.View ArticleGoogle Scholar
  50. Hoogendam A, de Vries Robbé PF, Stalenhoef AF, Overbeke AJ. Evaluation of PubMed filters used for evidence-based searching: validation using relative recall. J Med Libr Assoc. 2009;97(3):186–93.View ArticlePubMedPubMed CentralGoogle Scholar
  51. Vincent S, Greenley S, Beaven O. Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Inf Libr J. 2003;20(3):150–9.View ArticleGoogle Scholar
  52. Doust JA, Pietrzak E, Sanders S, Glasziou PP. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol. 2005;58(5):444–9.View ArticlePubMedGoogle Scholar
  53. Grupo de trabajo de la Guía de práctica clínica de atención en el embarazo y puerperio. Guía de práctica clínica de atención en el embarazo y puerperio. Ministerio de Sanidad, Servicios Sociales e Igualdad. Agencia de Evaluación de Tecnologías Sanitarias de Andalucía; 2014. Guías de Práctica Clínica en el SNS: AETSA 2011/10.Google Scholar
  54. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2014. Available from http://handbook.cochrane.org/. Accessed 6 June 2017.
  55. Leeflang M, McDonald S, Scholten RJ, Rutjes A, Reitsma JJB. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database Syst Rev. 2007, Issue 2. Art. No.: MR000022. doi:10.1002/14651858.MR000022.pub2.
  56. Rathbone J, Carter M, Hoffmann T, Glasziou P. A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Syst Rev. 2016;5:27. doi:10.1186/s13643-016-0197-5.View ArticlePubMedPubMed CentralGoogle Scholar
  57. Shaw RL, Holland C, Pattison HM, Cooke R. Patients’ perceptions and experiences of cardiovascular disease and diabetes prevention programmes: a systematic review and framework synthesis using the theoretical domains framework. Soc Sci Med. 2016;156:192–203.View ArticlePubMedGoogle Scholar
  58. Smith TO, Latham S, Maskrey V, Blyth A. Patients’ perceptions of physical activity before and after joint replacement: a systematic review with meta-ethnographic analysis. Postgrad Med J. 2015;91(1079):483–91. doi:10.1136/postgradmedj-2015-133507. Epub 2015 Aug 25View ArticlePubMedGoogle Scholar
  59. Finlayson CS, Chen YT, Fu MR. The impact of patients’ awareness of disease status on treatment preferences and quality of life among patients with metastatic cancer: a systematic review from 1997-2014. J Palliat Med. 2015;18(2):176–86. doi:10.1089/jpm.2014.0222. Epub 2014 Sep 26View ArticlePubMedGoogle Scholar
  60. Köberich S, Farin E. A systematic review of instruments measuring patients’ perceptions of patient-centred nursing care. Nurs Inq. 2015;22(2):106–20. doi:10.1111/nin.12078. Epub 2014 Sep 12View ArticlePubMedGoogle Scholar
  61. Cowley A, Evans C, Bath-Hextall F, Cooper J. Patient, nursing and medical staff experiences and perceptions of the care of people with palliative esophagogastric cancer: a systematic review of the qualitative evidence. JBI Database Syst Rev Implement Rep. 2016;14(10):134–66.View ArticleGoogle Scholar
  62. Loewen PS, Ji AT, Kapanen A, McClean A. Patient values and preferences for antithrombotic therapy in atrial fibrillation. A narrative systematic review. Thromb Haemost. 2017;117(6):1007–22. doi:10.1160/TH16-10-0787.View ArticlePubMedGoogle Scholar
  63. Brown J, Cook K, Adamski K, Lau J, Bargo D, Breen S, et al. Utility values associated with advanced or metastatic non-small cell lung cancer: data needs for economic modeling. Expert Rev Pharmacoecon Outcomes Res. 2017;17(2):153–64. doi:10.1080/14737167.2017.1311210.
  64. Ward Fuller G, Hernandez M, Pallot D, Lecky F, Stevenson M. Gabbe B4.Health state preference weights for the Glasgow outcome scale following traumatic Brain injury: a systematic review and mapping study. Value Health. 2017;20(1):141–51. doi:10.1016/j.jval.2016.09.2398 View ArticlePubMedPubMed CentralGoogle Scholar
  65. Carter GC, King DT, Hess LM, Mitchell SA, Taipale KL, Kiiskinen U, et al. Health state utility values associated with advanced gastric, oesophageal, or gastro-oesophageal junction adenocarcinoma: a systematic review. J Med Econ. 2015;18(11):954–66. doi:10.3111/13696998.2015.1066380.
  66. Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org/.
  67. Sampson M, Tetzlaff J, Urquhart C. Precision of healthcare systematic review searches in a cross-sectional sample. Res Synth Methods. 2011;2(2):119–25.View ArticlePubMedGoogle Scholar
  68. Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman ND, et al. So many filters, so little time: the development of a search filter appraisal checklist. J Med Library Assoc. 2008;96(4):356–61.Google Scholar
  69. ISSG [website]. Search Filters Resource. United Kingdom. [accessed 10 June 2016] Available at: https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-studies-of-public-views-and-patient-issues).
  70. Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016;5(1):74. doi:10.1186/s13643-016-0249-x.View ArticlePubMedPubMed CentralGoogle Scholar
  71. Grant MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Inf Libr J. 2004;21:21–32.View ArticleGoogle Scholar
  72. DeJean D, Giacomini M, Simeonov D, Smith A. Finding qualitative research evidence for health technology assessment. Qual Health Res. 2016;26(10):1307–17.View ArticlePubMedGoogle Scholar
  73. Jeong K, Caims J. Systematic review of health state utility values for economic evaluation of colorectal cancer. Health Econ Rev. 2016;6:36. doi:10.1186/s13561-016-0115-5.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s). 2017