The search for relevant outcome measures for cost-utility analysis of systemic family interventions in adolescents with substance use disorder and delinquent behavior: a systematic literature review

Systemic family interventions are psychotherapeutic treatments, which are increasingly used to treat children and adolescents with mental disorders. These interventions are based on the idea that the behavior of a patient is the result of interactions between himself and the different ‘systems’ he is involved in (i.e. family, peers, school, etc.) and of the interactions between these systems [1–3]. Treatment is directed at improving the disturbing aspects within these interactions [3] and it actively involves the systemic context of the patient. Hence, potential effects are broad and may range from improvements in the interactions with parents, other family members, peers or neighbors, to improvements in educational achievements and work relations, reduction of criminal activity and substance use and reduction of problems with the juvenile justice system [2, 4–6]. Systemic family interventions have shown particularly effective in the treatment of adolescents with substance use disorders and delinquency [7–10]. Examples of these interventions are Multisystemic Therapy (MST), Functional Family Therapy (FFT), Multidimensional Family Therapy (MDFT) and Brief Strategic Family Therapy (BSFT) [7–10].

With the increasing use of systemic family interventions, the question of funding and reimbursement arises. In some countries, like the Netherlands or the United Kingdom, systemic family interventions are reimbursed from social health insurance schemes and, as such, are part of collectively financed health care. Hence, the interventions compete for limited funds with other health care expenditures and, on top of proving effective, need to demonstrate value for money. Common practice in the economic evaluation of medical interventions is the use of cost-utility analysis (CUA) [11, 12] measuring effects in terms of Quality-Adjusted Life-Years (QALYs). QALYs combine length and quality of life gained. Typically, quality of life is measured through preference-based, generic health outcome measures (such as the EQ-5D). These outcome measures typically concentrate on improvements in a number of health domains. A recent publication of our department [13] described the results of a CUA of MDFT versus Cognitive Behavioral Therapy (CBT) in which the effects were measured with the EQ-5D. Yet, in the field of mental health, doubts have been expressed [14, 15] on the use of these generic quality of life measures [16] as these tools might be too limited to cover all relevant treatment effects. Studies on the applicability of these measures in mental health have presented mixed results [14, 15]. Furthermore, there is increasing attention for the inclusion of spillover effects on caregivers and families in economic evaluations. Currently, these effects are not yet included [17, 18], though they may be particularly important in treatment of younger patients. Recently, the Second Panel of Cost-Effectiveness in Health and Medicine has recommended further research on quality of life effects on family members of patients [19].

Both aspects, the assessment of effects specific to mental health treatments and the inclusion of (partial) effects on third parties, seem of particular relevance to the economic evaluation of systemic interventions in delinquency and substance use in adolescents. As outcomes of systemic family interventions are broad and transcend health gains, conventional CUA outcome measures may be too limited and insufficiently connected to clinical practice. This may be one of the reasons why economic evaluations of systemic interventions are still scarce and overall of low quality [20]. Existing economic evaluations of these interventions vary in setting, design and in outcomes measured [20] hence limiting the comparability of results. Furthermore, few studies consider effects on others than the patient [1].

If the aim is to perform economic evaluations of systemic family interventions which account for all relevant effects, a disorder-specific multidimensional measure that captures all relevant systemic contexts would be desirable. Ideally, if such a measure had societal preference-weights attached to its dimensions and levels, it would deviate from the common CUA methodology yet enable CUA-like economic evaluations. In patients with substance use disorder (one of the patient groups treated with systemic family interventions), the need for such a single comprehensive outcome measure capturing the full benefits of treatments has been recognized before [21]. Deas and Thomas [22] and Hogue and Liddle [23] emphasized the necessity of assessing various outcomes beyond effects in the adolescent. In an illustrative pilot study, Jofre-Bonet and Sindelar [21] presented a first example of a preference-based measure for adult populations with substance abuse. However, that measure was not based on standard preference-elicitation techniques but the authors attached patient preference-weights to the eight main domains of the Addiction Severity Index (ASI) [24] by constructing a weight index.

In the current study, we take this line of research further by searching for a multidimensional outcome measure to evaluate systemic family interventions in the populations of adolescents with substance abuse disorder or problems of delinquency. Such a measure could facilitate CUAs of systemic family interventions and could either be based on existing effectiveness measures in this field or fully designed anew. In both cases, the use of an existing measure or the design of a new measure, relevant domains would need to be identified. Based on consultation of the literature on systemic family interventions [1, 25, 26] the domains relating to aspects of the individual patient, family, school (or work) and other community environments (e.g. peers, neighbors) were considered most relevant to the evaluation of the interventions. Figure 1 provides a graphical illustration of these domains, which indicate where potential effects may occur. The strength of the impact on the different systems may obviously differ, depending on the exact underlying problems and other contextual factors.

We perform a systematic literature review to investigate and appraise available instruments in the field of adolescent delinquency and substance use, which cover the relevant domains and which are already accepted and validated in the field. We assess which of these instruments might be most suited to serve as a basis for a preference-based measure in CUA, based on characteristics like comprehensiveness, brevity, accessibility, psychometric properties, etc. Advantage of using an existing instrument would be its being established, accepted and validated in the field and known by clinicians. It would then only be necessary to add preference-weights to the domains to account for differences in impact of each domain. In this way we aim to contribute to the development of adequate outcome measures to assess the economic value of systemic family interventions in the treatment of delinquency and substance use.

We conducted a systematic literature review to identify instruments within the effectiveness and efficacy literature of mental health interventions for adolescents with substance use disorder and delinquency problems. We then assessed the suitability of these instruments for use of preference elicitation techniques. The assessment was based on several characteristics relevant to attain societal preference weights. These characteristics were among others the coverage of the systems displayed in Fig. 1 (i.e. youth, family, peers, school, work, society and neighbors), brevity, practicability of use, accessibility, psychometric properties and acceptance in the field. The review protocol was not registered. Yet, this study adhered to the PRISMA reporting guidelines [27].

Search methods for identification of studies

Databases were selected as to cover both interventions in the medical and in the educational field. The systematic literature review was performed in PubMed, Psychnet (PsycBOOKSc, PsycCRITIQUES, print), Cochrane and ERIC (Education Resource Information Center) to identify all effectiveness studies of mental health interventions for adolescent with substance use disorder or problems of delinquency. The databases were consulted between 5 March 2013 and 8 March 2013. Additional studies were identified based on reference list search. There were no restrictions on the type of publication. The language of publication was required to be English and publication date was 1990 or more recent. The search strategy used is displayed below.

Data analysis

Study selection

First, duplicates were removed. Then, the study selection was performed in two rounds. First, a selection based on title and abstract was performed, then selected articles were subject to a second screening based on full texts. Both rounds of selection were performed by two researchers independently and were each followed by a round of consensus. The eligibility criteria for the first selection based on title and abstract were the following.

Subsequently, when abstracts or titles adhered to the above screening criteria, full texts were independently screened for inclusion based on the following (additional) criteria.

Furthermore, articles from reference lists of reviews were identified. For these, we performed a shortened screening and selection procedure. Titles of these articles were screened based on the following criteria: a) > =1990; b) peer-reviewed article; c) randomized control trial or effect/effectiveness/efficacy study/treatment outcome; d) adolescents; e) delinquency/offenders/substance-abuse; f) mental health intervention (no pharmacotherapy). If this selection resulted in inclusion, the abstract was screened and a final decision on inclusion or exclusion was made. Included articles were added to the database of identified articles for further data synthesis.

Data extraction

Data extraction was performed in MS Access with predefined fields. From all selected studies, general information, such as the title of the study, the name of the author, journal, etc., were recorded, as well as information on the sample size, the studied population and type of intervention (systemic, other [i.e. individual, group intervention], both).

In addition to this general information, instrument-specific information was extracted. This information consisted of instrument names (e.g. Child Behavior Checklist [CBCL]) and covered domains (e.g. family functioning, adolescent behavior, etc.). This information was recorded in order to identify the instruments currently used in the field and their coverage of the different systems relevant for the evaluation of systemic family interventions (Fig. 1).

Synthesis and evaluation of results

As a next step, domain names of the instruments were extracted from the identified articles and linked to the systems relevant for the evaluation of systemic family interventions (Fig. 1): youth, family, peers, school, work, society and neighbors. Domain names were verified with available resources such as guidelines, websites of the developer and other articles using the same instrument. After verification, the domains were translated into the systems mentioned in Fig. 1. For this purpose, domains related to the adolescents themselves, such as ‘substance use and abuse’, ‘physical health’ or ‘mental health’ were linked to the system ‘youth’ whereas domains such as ‘family relations’ were recoded into the system ‘family’, domains like ‘peer relations’, ‘social skills’ or ‘leisure/recreation’ were labeled as ‘peer’ system, domains like ‘educational status’ were labeled ‘school’ and ‘delinquency’ as ‘society’. Table 1 provides an example of the process of recoding for the Problem Oriented Screening Instrument for Teenagers (POSIT). Next, all instruments were classified based on the number of systems (presented in Fig. 1) covered and ranked from highest to lowest. Those covering five or more systems were considered most relevant for our purpose as those covered the majority of effects of systemic family interventions in adolescents with substance use disorder or problems of delinquency.

Problem Oriented Screening Instrument for Teenagers (POSIT)
Domain	Corresponding system
Substance use and abuse	youth
Physical health	youth
Mental health	youth
Family relations	family
Peer relations	peers
Educational status	school
Vocational status	work
Social skills	peers
Leisure/recreation	peers,
Aggressive behavior/delinquency	society

In line with our aim to identify an instrument, which captures most of the systems relevant to the evaluation of systemic family interventions, those instruments covering more than five systems were evaluated in more detail. These were then appraised according to necessarily arbitrary characteristics of brevity, feasibility, practicability, accessibility, psychometric properties and acceptance in the field. These characteristics were set up as to identify one or more instruments suitable to attain societal preference-weights for an instrument by means of preference-elicitation techniques. Within preference-elicitation techniques, such as discrete choice experiments, the number of domains rarely exceeds ten [28, 29]. With higher numbers of domains, the decision task may become too complex and cognitively demanding for the respondent [28]. Hence, a suitable instrument should possess less than 10 domains. A second consideration was the practical use of the instrument itself in clients. An instrument, ideally suitable for self-completion, should put as little strain as possible on the respondent, without loss of important content. Hence, we set a limit to the maximum number of items of the instrument at 500 and a maximum completion time of 1 h, assuming that these would be reasonable amounts of items and time to ask from respondents. Another criterion was the accessibility of the instrument as to ascertain ease of use in future studies. Evaluation of this criterion included the price of use and availability of a (digital) version. Psychometric properties were considered to judge the suitability of the instrument for integration in health economic evaluations. Findings from existing publications on validity and reliability of the instruments were considered in this context. Finally, the frequency of use of the instrument was considered an indicator for the acceptance of the instrument in the clinical field. This was approximated by the number of times that an instrument was used in the studies identified in this review.

The objective of this systematic literature review was to identify existing instruments in the field of adolescent delinquency and substance use, which cover the relevant domains of systemic family interventions. The instruments were appraised based on characteristics relevant for use in economic evaluations such as brevity, accessibility, psychometric properties etc. Euro-ADAD and T-ASI showed favorable characteristics in relation to the criteria for a comprehensive outcome measure, covering multiple relevant systems and being suitable for obtaining preference weights. Both instruments lack preference weights for the outcomes, at present. Attaining these (as a potential next step) would facilitate calculating ‘utility scores’ as common in economic evaluations. Furthermore, the results of the current study may inform future efforts towards standardized and comprehensive core outcome sets as defined by the COMET initiative [54]. The study may be seen as a preparatory step towards a full COMET effort to standardizing the QALY approach to include broader effects.

Some limitations of this study must be noted. First, given our focus on published research up to 2013, we may have missed out on very recent developments in this field. In the Netherlands, for instance, a new, comprehensive instrument for measuring substance abuse in adolescents is being developed, called the MATE-Y [55], which includes nine modules each containing several domains. Yet, up to today there have not yet been publications on the MATE in the field of youth/adolescents. But similar developments may be ongoing elsewhere. Second, we have not investigated the possibility of constructing a new measure by combining different measures into one composite measure. Though this may be a limitation of this paper, we considered it a necessary first step to identify the instruments currently available in the field for direct use. This may also help to highlight the relevant domains to include in a newly developed instrument. With our approach, we were able to identify two instruments as most promising candidates to use in comprehensive evaluations of systemic family interventions. Neither instrument is currently considered ‘gold standard’ in practice. Furthermore, as common for systematic reviews, the results from the current study are based on a limited selection of databases within a limited timeframe. Yet the number of screened and identified articles was extensive and we assume that the consultation of an even larger number of databases would not have yielded significant differences in results. Also, the characteristics for further selection of the instruments were necessarily arbitrary and guided by our goal of selecting one or more instruments suitable to be used to attain societal preference weights and be used in economic evaluations in the long term. We realize that the suitability criterion of a maximum of 500 questions/1 h of completion time may be rather high when considering the busy clinical practice and ongoing evaluation of patient progress. Furthermore, had we considered different or more broad characteristics, additional instruments might have been found suitable. For example, one could think of shortening existing longer instruments first and then proceeding towards steps of attaining societal preference weights. In the light of limited time, this was not considered feasible in the current study.

Notwithstanding these limitations, our review revealed two promising, currently used instruments, which may be made suitable for inclusion in economic evaluations of systemic family interventions: the Euro-ADAD and T-ASI. To make these instruments suitable for health economic evaluations, first of all, more detailed investigation is necessary of their validity, feasibility and comprehensiveness. Current information on this is scarce, yet needed. Moreover, using these instruments in health economic evaluations will require important next steps. In particular, preference weights would need to be derived for the different states described by the instrument, like those available for health-related utility measures such as the EQ-5D. This is possible through preference elicitation techniques, such as discrete choice experiments or time-trade-off techniques, ultimately leading to ‘utility scores’, which can be attached to the different ‘states’ described by the instrument.

Intriguing questions in this context relate to who should indicate the state a person is in and who should provide the values for the different possible states (i.e., whose preferences count). In line with many guidelines for health-economic evaluations [11, 56], and in line with the broad aim of systemic family interventions, one could ask ‘patients’ to provide self-reports based on one of the identified multidimensional instruments. The value attached to this state could then be based on preferences obtained in the general population. This would provide ‘societal weights’ for the broad outcomes of systemic family interventions. These societal weights could thus be attached to the state a person indicates him- or herself to be in on the multidimensional instrument, thus leading to an overall utility score. Given the broad range of outcomes, including effects incurred by others than the patient or even his family (e.g., a safe neighborhood), the score thus relates to a preference ordering over states that include the effects on more than the patient alone. This may be an additional reason for opting for general public preferences. However, whether the general public is the appropriate source (rather than e.g. decision makers or health care professionals) must be further assessed and discussed, as well as their ability to appropriately weight such diverse outcomes. The more fundamental question is whether these scores would count as ‘utilities’ or rather as multi-criteria decision weights.

Other relevant issues in developing a multidimensional utility measure of systemic family interventions may be the diversity and hierarchy of treatment effects. As mentioned earlier, a comprehensive measure would include health as well as non-health effects and would also include both the effects on the patient himself and society as a whole. Obviously, these different effects may be interrelated. Moreover, some observable effects may be considered to be intermediate effects, whereas others may be final outcomes. Related to this point, there may be short-term and long-term effects, which can be important. Hence, in the construction of such a preference-based measure, good care needs to be taken of the possible interaction of the effects.

One may argue that an alternative route to finding an appropriate outcome measure could be to use existing measures in the field of economic evaluation, most notably QALY measures. To our knowledge, so far there have been only a few studies on the validity of preference-weighted health-related quality of life instruments in an adult population of substance abusers [57, 58]. There have been two studies on the degree to which common preference-weighted measures of quality of life (e.g. QWB-SA, SF-12) correlate with substance use severity [58, 59]. Whereas the first study provides evidence for insufficient coverage of all disease dimensions in substance use disorder [58], the second study does suggest moderate to good correlation between quality of life measures and substance use severity measures [59]. In order to verify these results and determine whether the proposed instruments add value in the field of delinquency and substance abuse in adolescents, further research on the suitability and potential of the quality adjusted life year (QALY) measure in this population is recommended.

Keeping these alternatives in mind, further research on the instruments highlighted in the current paper, specifically on the attachment of societal preference weights could bring evaluation of mental health interventions for delinquent and substance abusing adolescents closer to the standard methodology in health economic evaluations of curative medical interventions. Both identified instruments appear suitable and broad enough to capture the effects of family interventions in substance abusing and delinquent adolescents in such CUA. Adding societal preference weights to one of these instruments will create an instrument, which combines the advantage of the specificity of a disorder-specific instrument with compliance with common methodology of health economic evaluations and captures the broad effects relevant to mental health interventions. CUAs of these interventions can then be performed based on a broad and specific measure that includes several systems/dimensions and at the same time acknowledges the relative value that society attaches to improvements in these diverse systems. Though performing CUAs in the field of substance abuse and delinquency in adolescents remains a challenging task, this paper attempted to contribute to confronting one of the major issues in that context: finding a suitable outcome measure.