Psychometric evaluation of the Mental Health Quality of Life (MHQoL) instrument in seven European countries

Abstract

Introduction

To make efficient use of available resources, decision-makers in healthcare may assess the costs and (health) benefits of health interventions. For interventions aimed at improving mental health capturing the full health benefits is an important challenge. The Mental Health Quality of Life (MHQoL) instrument was recently developed to meet this challenge. Evaluating the pyschometric properties of this instrument in different contexts remains important.

Methods

A psychometric evaluation of the MHQoL was performed using existing international, cross-sectional data with 7155 respondents from seven European countries (Denmark, France, Germany, Italy, Portugal, The Netherlands, Portugal and the United Kingdom). Reliability was examined by calculating Cronbach’s alpha, a measure of internal consistency of the seven MHQoL dimensions, and by examining the association of the MHQoL sum scores with the MHQoL-VAS scores. Construct validity was examined by calculating Spearman’s rank correlation coefficients between the MHQoL sum scores and EQ-5D index scores, EQ-VAS scores, EQ-5D anxiety/depression dimension scores, ICECAP-A index scores and PHQ-4 sum scores.

Results

The MHQoL was found to have good internal consistency for all seven countries. The MHQoL sum score and the MHQoL-VAS had a high correlation. Spearman’s rank correlation coefficients were moderate to very high for all outcomes.

Conclusion

Our results, based on data gathered in seven European countries, suggest that the MHQoL shows favourable psychometrical characteristics. While further validation remains important, the MHQoL may be a useful instrument in measuring mental health-related quality of life in the Western European context.

Introduction

Worldwide, more than one billion people are affected by mental health problems. The burden of disease related to mental health problems is very large. For instance, the disability-adjusted life-years (DALYs) lost due to such problems represent seven percent of the total global burden of disease [1]. Additionally, mental health problems are known to have a major economic impact. This can be illustrated by the worldwide costs of lost productivity related to depression and anxiety, the two most commons mental health problems, which have been estimated to amount to US$ 1 trillion annually (1.6% of worldwide GDP) [2]. Such figures highlight the importance of effectively preventing and treating mental health problems.

However, healthcare systems have limited resources to improve the health of the populations they serve. Interventions aimed at preventing or treating mental health problems, even effective ones, in that context compete for scarce resources with interventions aimed at other diseases. Therefore, healthcare decision-makers are confronted with difficult allocation decisions. They may wish to prioritize those interventions that contribute most to the goals of the healthcare system, including using the available resources efficiently. In this context, economic evaluations are increasingly used, in which the benefits and costs of interventions (relative to some relevant comparator) are assessed in order to determine whether they are cost-effective. Such evaluations are most commonly and systematically used in relation to reimbursement decisions regarding pharmaceuticals. However, they are also increasingly applied to other types of interventions, but this broadening of the application of economic evaluations in other contexts comes with specific challenges [3]. Economic evaluations can and are also sometimes used to investigate the cost-effectiveness of mental health interventions, with the aim of informing decisions regarding their reimbursement. For mental health interventions an important challenge is capturing all relevant benefits related to these interventions. In conventional economic evaluations, health benefits are typically measured using a generic health-related quality-of-life measure, such as the EQ-5D [4] or SF-6D [5]. These instruments are prescribed in many guidelines for economic evaluations of health interventions, including the UK [6] and Dutch guidelines [7]. However, it has been questioned whether these commonly used generic health-related quality of life instruments capture all relevant quality of life domains impacted by mental health problems and interventions aimed at improving mental health [8]. This is problematic since it could lead to inaccurate estimations of health benefits related to mental health interventions, hampering well-informed decisions. Such decisions require that the quality of life impact of mental health interventions are captured accurately and completely.

In an attempt to overcome this problem, the Mental Health Quality of Life instrument (MHQoL) was developed [9]. The MHQoL is a self-report, seven-item questionnaire that captures and values dimensions relevant to the quality of life of people with mental health problems, such as self-image, independence and hope. While the instrument was based on previous research highlighting these relevant quality of life dimensions, its psychometric properties, like feasibility, reliability and validity, also need to be demonstrated. This is even more important if healthcare decisions are informed by economic evaluations using the MHQoL as outcome measure. The psychometric properties of the MHQoL so far have only been examined in the Dutch context [9]. This evaluation was performed in a sample of 110 members of the Dutch general public as well as a sample of 479 Dutch mental healthcare users. The results of this study suggested that the MHQoL is a promising instrument, demonstrating favourable psychometrical properties. However, further psychometric evaluation in different populations and contexts remains warranted. This is also true for psychometrical evaluation in an international context. Such evaluation may be considered relevant, not only to investigate the performance of translated versions of the MHQoL, but also given that cultural differences may impact how mental health problems may be experienced, evaluated and perceived [10].

The objective of this study, therefore, is to evaluate the psychometric characteristics of the MHQoL using a large dataset obtained in seven European countries. Specifically, we will investigate the reliability of the MHQoL by examining its internal consistency as well as the construct validity of the MHQoL by investigating the association of the MHQoL sum scores with other validated outcome measures.

Methods

Data source

For this study, we used cross-sectional data obtained in the fourth survey wave of the European COvid Survey (ECOS) project, which is described in detail elsewhere [11]. Generally, this online survey examined support for COVID-19 containment policies, including vaccinations, worries about COVID-19, and trust in different information sources. The data in the fourth wave of this survey was obtained between 5 and 16 November 2020. Respondents (n = 7115) were recruited from the general public in seven European countries (Denmark, France, Germany, Italy, Portugal, the Netherlands, and the United Kingdom) by the market research company Dynata using multisource online panels. To ensure that the sampling frame was representative of the population in each country, the company used various recruiting procedures for different subgroups of the population in each country. It used for example advertised/open recruitment, loyalty programs, affiliate networks and mobile apps [11]. Quotas based on age category, regional distribution and gender were implemented by the authors using the Qualtrics research suite to ensure and control the representativeness based on the country specific census data. Dynata ensured representativeness with regard to educational categories based on their expertise in the differences in educational degrees for each country. The authors proceeded by excluding incompletes answers and speeders (faster than 1/3 of the median time in each country), both of which were replaced by Dynata to ensure the representativeness of the sample. The resulting sample of respondents from each country (with n ~ 1000) was representative of its adult population in terms of region, gender, age group and education level.

The questionnaire was available in the seven languages of the included countries. The MHQoL had existing official versions in Dutch, English and German, which were used in this survey. For the other four countries and languages, the MHQoL instrument was translated by native speakers with a background in health economics.

Respondents completed the MHQoL instrument along with the EuroQol (EQ-5D-5L and EQ-VAS), ICECAP-A (the ICEpop CAPability measure for Adults), and PHQ-4 (Patient Health Questionnaire for Depression and Anxiety) instruments. For these instruments, official translations available from the developers of these instruments were used. Respondents also were asked to answer questions about their demographic characteristics including gender, age, relationship status, and level of education, next to COVID-related questions.

Outcome measures

The MHQoL is an instrument intended to be used to describe and value respondents’ current mental health-related QoL [9]. In the descriptive part, respondents are asked to describe their mental health state using seven specific dimensions of mental health-related quality of life and four answering levels per dimension. The seven dimensions are: self-image, independence, mood, relationships, daily activities, physical health and hope. Levels for self-image for example range from ‘I think very positively about myself’ to ‘I think very negatively about myself’. Preference-based tariffs, allowing scores on the different levels to be converted into utility scores anchored on 0 (dead) and 1 (full mental health-related QoL), are not yet available for the MHQoL. In the absence of tariffs, the MHQoL sum score is used as an alternative, which ranges from 0 (lowest level on all seven dimensions) to 21 (highest level on all seven dimensions), with higher scores indicating better mental health-related quality of life. Next to the descriptive part, the MHQoL instrument also has a direct valuation part in which respondents are asked to rate their psychological wellbeing using a horizontal visual analogue scale (MHQoL-VAS) ranging from 0 (representing ‘worst imaginable psychological wellbeing’) to 10 (representing ‘best imaginable psychological wellbeing’).

The five-level EQ-5D (EQ-5D-5L) is a generic instrument to measure and value respondents’ current health-related QoL [4]. Within the questionnaire, respondents are asked to describe their health using five dimensions and five answering levels per dimension. These five dimensions are: Mobility, Self-Care, Usual Activities, Pain/Discomfort, and Anxiety/Depression. The five answering levels range from having no problems to having extreme problems. Using country-specific, preference-based tariffs, answer scores can be converted into utility scores, with 0 as the equivalent of the state ‘dead’ and 1 as the equivalent of the state ‘perfect health’. In addition to scoring the five dimensions, respondents are asked to rate their current overall health using a vertical visual analogue scale (EuroQol Visual Analogue Scale; EQ-VAS) ranging from 0 (‘worst imaginable health state’) to 100 (‘best imaginable health state’).

The ICE-CAP-A (ICEpop CAPability measure for Adults) is an instrument to measure and value respondents’ overall capability wellbeing and is grounded in Sen’s capability approach [12, 13]. Within the questionnaire, respondents are asked to describe their capabilities in relation to five important life domains: stability, attachment, autonomy, achievement and enjoyment. Each domain is scored using four levels, ranging from full capability to no capability. Using preference-based tariffs, answer scores can be converted to standardised index scores, ranging from 0 (no capability) to 1 (full capability). Currently, tariffs for the United Kingdom [12] and the Netherlands [14] are available.

Finally, the PHQ-4 is a four-item self-complete screening instrument that measures respondents’ likeliness of an anxiety disorder and/or depression [15]. The PHQ-4 is based on the Patient Health Questionnaire (PHQ), a more extensive instrument used by care providers to diagnose patients with mental health disorders. Dimensions are “Feeling nervous, anxious or on edge”, “Not being able to stop or control worrying”, “Feeling down, depressed or hopeless” and “Little interest or pleasure in doing things”. The PHQ-4 has four levels which range from “Not at all” to “Nearly every day”. A sum score can be calculated, which can be used to categorise the respondent’s psychological distress level as none (0–2), mild (3–5), moderate (6–8), or severe (9–12).

Statistical analysis

Using descriptive statistics, the basic characteristics of the respondents were summarized. Mean MHQoL sum score and mean MHQoL-VAS score were compared to scores known for the Dutch general population [9]. For subgroups based on country, gender, and age group, MHQoL sum scores were calculated, also to provide a reference for future studies. In doing so, we do acknowledge and stress the exceptional situation due to the COVID-19 pandemic.

We additionally investigated the MHQoL dimension scores, comparing the youngest and oldest age groups. A lower and upper MHQoL quartile were distinguished using cut-off values (respectively < 12 and > 16) for the MHQoL sum score. We investigated the membership of these quartiles focusing on background characteristics of the respondents including income, mean, minimum and maximum values of the four other outcome measures. EQ-VAS mean scores per country were compared to population norm scores, also given the fact that the survey was conducted during the COVID-19 pandemic.

To examine the internal consistency of the MHQoL, Cronbach’s alpha of the seven dimensions was calculated, both overall and by country. Cronbach’s alpha is a coefficient that represents the extent to which items of a measure are correlated, indicating the extent to which these items measure the same construct, here mental health-related quality of life. Cronbach’s alpha is expressed as a number between 0 (no correlation) and 1 (full correlation), and a score of > 0.7 is seen as indicating good internal consistency [16]. In addition, Spearman’s rank correlation coefficients of the MHQoL sum score and the MHQoL-VAS were calculated, again overall and by country. This correlation was expected to be high and positive since both aim to measure mental health-related QoL. When interpreting results, a correlation coefficient is seen as trivial when < 0.1, as small when 0.1–0.3, as moderate when 0.3–0.5, as high when 0.5–0.7, as very high when 0.7–0.9, and as nearly perfect when > 0.9, following previous evaluation studies like Hoefman et al. [17]. Additionally, the association between the MHQoL-VAS and the MHQoL dimension scores was investigated using a linear regression model (ordinary least squares regression; OLS). A moderate to high positive correlation and a moderate to high adjusted R² were expected since these dimensions levels represent mental health states which determine overall mental health-related QoL. An adjusted R² of > 0.2 was expected, based on previous models which modelled EQ-VAS as the dependent variable and the levels of the EQ-5D dimensions as independent variables [18, 19].

To examine convergent validity, Spearman’s rank correlation coefficients between the MHQoL sum score and respectively: EQ-5D index score, EQ-VAS, EQ-5D anxiety/depression dimension score, ICECAP-A index score, and PHQ-4 sum score were calculated, overall and by country. The resulting coefficients inform whether these measures of theoretically interrelated constructs are correlated, which is an indication of construct validity. We expected the MHQoL sum score to have a moderate to strong positive correlation (0.3–0.7) with the EQ-5D index score, the EQ-VAS, and the ICECAP-A index score. This was based on the assumption that having a better mental health-related QoL is associated with both a better health-related QoL and a better wellbeing. It is acknowledged that these are complex associations, e.g. mental health has direct and indirect effects (e.g.by life style choices) on physical health and vice versa [20].

Furthermore, we expected the MHQoL sum score to have a strong negative correlation (0.5–0.7) with the EQ-5D anxiety/depression dimension and with the PHQ-4 sum score, since a better mental health-related QoL is strongly related to the absence of mental health problems. Tariffs to compute EQ-5D-5L index scores were obtained from the EuroQol website [21]. Tariffs to compute ICECAP-A index scores were obtained from the website of the University of Birmingham [12]. United Kingdom (UK) tariffs were used for EQ-5D-5L and ICECAP-A for all countries. This was done given that tariffs were not available for all countries. Since the current UK tariffs for EQ-5D-5L have been disputed [22] the analyses were also performed using Dutch EQ-5D-5L tariffs [23] to check whether this would influence results.

Beforehand, we did not formulate expectations regarding mean MHQoL sum scores in different countries although they are expected to differ, e.g. based on country differences in depression stigma [24].

All analyses were performed using STATA version 16.1 (StataCorp, 2019. Stata Statistical Software: Release 16. College Station, TX).

Results

Descriptive statistics

Table 1 shows the general characteristics of our respondents. About half of the respondents was female (52%), a third of the respondents lived alone (29%), and a majority of respondents were of working age (18–64 years; 79%). The mean MHQoL sum score was 14.1 (SD ± 3.8), which is below the MHQoL sum score reported for the Dutch general population (15.5; SD ± 2.9) [9]. The mean MHQoL-VAS was 6.6 (SD ± 2.2), which is below the mean MHQoL-VAS reported for the Dutch general population (7.5; SD ± 1.5) [9]. For the dimension scores (scale 0 to 3; higher is better), means ranged from 1.8 (Hope) to 2.2 (Physical health).

Table 1 Respondent characteristics (n = 7115)

The observed mean MHQoL sum score differed somewhat between subgroups, see Table 2. This Table also shows that the mean MHQoL sum score increased with age.

Table 2 Mean MHQoL scores by country, gender and age group

Mean MHQoL sum scores, by country and age group, are presented in Table 3. These scores ranged from 12.3 in young adults (18–24 years) in the UK, to 15.9 in Danish elderly (65+ years).

Table 3 Mean MHQoL sum score by country and age group

Given the observed increase in MHQoL with age, we decomposed the MHQoL sum score into domain scores for the youngest (18–24 years) and oldest (65 + years) age groups, between which the difference was the largest (1.6 points) (see Additional file 1). The youngest group scored worse than the oldest group on all dimensions except for Physical health (+ 0.2), while especially their scores on the dimensions Mood (-0.6) and Independence (-0.5) were relatively low.

Characteristics and mean measure scores are presented in Table 4 contrasting the respondents with a high (> 16) MHQoL score and those with a low (< 12) MHQoL scores. Compared to those in the high group, respondents in the low group were younger, less often in a relationship, more often female, had lower wellbeing scores, and a lower health-related QoL.

Table 4 Descriptive statistics of the highest and lowest MHQoL quartiles

The mean score on the EQ-VAS in our sample was 72.3 (SD ± 23.3), which represents a moderate health-related QoL. For all six countries for which EQ-VAS population norm scores were available—these were not available for Portugal—the mean EQ-VAS-score was below this norm (which ranges from 76.8 in France to 83.7 in Denmark) [25]. Overall, health-related QoL was relatively low compared to reference values.

In contrast to population norms [25], the mean EQ-5D scores in our sample did not decrease with age. For example, overall in the age group 18–24 the EQ-VAS score was 71.6 (SD ± 25.1) while it was 73.0 (SD ± 21.5), for those over 65. We also did not observe a decrease with age in mean scores for capability wellbeing (ICECAP-A scores).

Internal consistency

The seven dimensions of the MHQoL were found to have good internal consistency, with an overall Cronbach’s alpha of 0.82. Comparable levels of internal consistency were observed for all individual countries (presented in Table 5), with alphas ranging from 0.78 for Portugal to 0.87 for the United Kingdom.

Table 5 Chronbach’s alpha by country

MHQoL and MHQoL-VAS correlation

As expected, a significant positive correlation (0.65; p < 0.00) was found between MHQoL sum score and the MHQoL-VAS. For the individual countries, Spearman’s rank correlation coefficient ranged from 0.53 (Italy) to 0.67 (Portugal and the United Kingdom).

The adjusted R² of the multivariate linear model (OLS; Table 6) with MHQoL-VAS as the dependent and the MHQoL dimension level scores as the independent variables was 0.43 for the total sample. For the individual countries, this ranged from 0.36 (Italy) to 0.51 (Portugal).

Table 6 Multivariate regression analysis of MHQoL-VAS

MHQoL-VAS scores were most strongly associated with Mood (all levels), Self-image (‘very negative’) and Future (‘very gloomy about’). This might suggest that these will also receive the most weight in future MHQoL tariffs. Within each dimension lower levels were associated with lower MHQoL-VAS scores. The only exception were the scores associated with the third and fourth level for the domain Independence, which were almost identical. All but two coefficients were statistically significant.

Convergent validity

To examine convergent validity, an element of construct validity, Spearman’s rank correlation coefficients were calculated for MHQoL sum scores with other relevant outcome measures. The resulting rank correlations are presented in Table 7. As expected, strong positive correlations were observed between MHQoL sum score and EQ-VAS, EQ-5D index scores and ICECAP-A index scores. The additional analysis using Dutch instead of UK EQ-5D tariffs resulted in comparable correlation coefficients (not presented). Furthermore, expected strong negative correlations were observed with the EQ-5D dimension anxiety and depression and with the PHQ4 score.

Table 7 Spearman’s rank correlation coefficients

Discussion

In this paper, we presented a psychometric evaluation of the MHQoL questionnaire in seven European countries. This study, therefore, represents one of the first psychometric evaluations of the MHQoL and the first international one to our knowledge. Importantly, this builds on the evidence of the performance of the MHQoL as a reliable and valid measure of mental health-related QoL across Western European countries.

Specifically, we examined reliability by investigating internal consistency and construct validity by investigating convergent validity, using existing survey data obtained through the ECOS survey, which was conducted in seven European countries. Overall as well as for the separate counties, we found good internal consistency between the dimensions. Additionally, the MHQoL-VAS score, in general, was significantly associated with the MHQoL sum scores and domain scores in the expected way. Convergent validity was investigated by investigating the correlations between the MHQoL instrument with the EQ-5D, ICECAP-A as well as specific mental health instruments and showed favourable results. Overall our results suggest that in the investigated countries, the MHQoL appears to be a psychometrically sound measure of quality of life in people with mental health problems, which highlights it is a promising instrument to use and validate further.

The importance in doing so may be emphasised by the fact that sound and concise instruments, capable of adequately capturing important life domains impacted by mental health problems, are currently lacking [26]. Instruments like the MHQoL, therefore, can facilitate broadening the application of HTA to the field of mental health in a way that the benefits of mental health interventions are indeed captured. This may ultimately lead to better (informed) decision-making and, therefore, a more efficient and equitable allocation of resources.

Limitations of the study

The main strengths of our study are its large sample size, the representativeness and international character of its sample, the inclusion of several relevant outcome measures, and the evaluation of multiple psychometric properties. Besides these strengths, several limitations need to be acknowledged. First, the psychometric evaluations performed were not without limitations. Given the nature of the ECOS survey, we did not have any information on clinical diagnosis of respondents. Consequently, we could not evaluate whether MHQoL scores were associated with the absence, presence or severity of mental health problems as determined by clinical professionals. Such an evaluation would have been a valuable and relevant addition to the results presented here, which relied on self-reported outcome measures. Furthermore, since the MHQoL was only included in one of the ECOS survey waves, we were not able to perform a test–retest validation. This would have allowed further evaluation of the reliability (consistency in measurement) and responsiveness (ability to capture changes in mental health) of the MHQoL. In addition, in the absence of MHQoL tariffs, we based our analyses on the sum scores and VAS scores of the MHQoL, while basing them on utility scores would have been interesting and insightful as well.

Second, the data were collected using an online survey, which may have caused some selection bias in respondents and may be associated with lesser engagement with the survey by respondents. Moreover, and very important to stress, the data were collected during the COVID-19-crisis. This situation, the threat or experience of falling ill as well as the strong measures imposed in most countries to mitigate the outbreak, may have affected mental health of the population. The fact that the survey focused on COVID-related aspects (including risks, worries and government intervention) may have increased the awareness of negative consequences and, therefore, negative feelings. The QoL and wellbeing scores observed in our study were, on average, indeed lower than previously observed. The age-profile for mental health, with younger people scoring lower than expected and lower than older people in all domains except for physical health, may also be related to this context, if the mental health of young adults on average would be affected more due to COVID (measures) than that of the old (e.g. by affecting normal activities and social life more in the young). As a competing explanation, the online form of the survey may have caused a selection bias in the group of older people towards those with a higher level of mental health. While this may not have affected the results in relation to our research question on the validity of the MHQoL, it is important to emphasize this when using the here presented MHQoL scores as a reference in future studies.

As a third limitation, the MHQoL instrument was translated from Dutch to four other languages by native speakers with a background in health economics. A more extensive translation procedure, as was done for English and German, using forward–backward techniques, would have been superior.

Future perspective

Given the findings presented here, the MHQoL appears to be a valid and reliable measure of mental health-related QoL in the Western European context. More research, confirming these findings, and expanding the investigation to other aspects of feasibility, reliability and validity, is required in order to gain a fuller understanding of the psychometric properties of the MHQoL, also in different groups (including people with known mental health problems). If the current positive findings are confirmed, the MHQoL can be used in different settings to monitor or evaluate (changes in) mental health-related quality of life. This comprehensive measure may also be a valuable addition to disease-specific clinical outcomes in clinical decision making e.g. to inform patients on the expected impact of an intervention on their quality of life. It could also be applied in economic evaluations, informing decision-makers about the full costs and benefits of mental health interventions. It needs noting that in that context, using the MHQoL rather than generic HR-QoL measures, has the advantage of being more comprehensive in terms of relevant health domains covered but comes at the price of limiting comparability between outcomes. This may be less problematic for decisions within the mental health domain. Moreover, if adequate thresholds would be established for the MHQoL (like for generic HRQoL), decision-making would be more straightforward. However, given that in several Western European countries (with Germany as an exception) [27] the EQ-5D currently is prescribed to be used in economic evaluations (also within the mental health field), for these countries administering the MHQoL instrument alongside the EQ-5D is recommendable in the context of economic evaluations. This will also offer the opportunity to further investigate the relationship between the EQ-5D and MHQoL instruments.

The use of the MHQoL in economic evaluations would require having valid tariffs that transform a MHQoL state as described on the descriptive system into a utility score (typically anchored on the states ‘dead’ (0) and ‘perfect health’ (1). Such tariffs are expected to be developed soon (for the Netherlands).

Conclusion

Our results suggest the MHQoL is a psychometrically sound measure of mental health-related quality of life in the Western European context. While more research remains necessary, this makes the MHQoL instrument interesting to be used in (economic) evaluations of mental health interventions, as it more comprehensively captures their benefits.