Assessing quality of life in older adults: psychometric properties of the OPQoL-brief questionnaire in a nursing home population
Health and Quality of Life Outcomes volume 18, Article number: 1 (2020)
Well-adapted and validated quality-of-life measurement models for the nursing home population are scarce. Therefore, the aim of this study was to test the psychometrical properties of the OPQoL-brief questionnaire among cognitively intact nursing home residents. The research question addressed evidence related to the dimensionality, reliability and construct validity, all of which considered interrelated measurement properties.
Cross-sectional data were collected during 2017–2018, in 27 nursing homes representing four different Norwegian municipalities, located in Western and Mid-Norway. The total sample comprised 188 of 204 (92% response rate) long-term nursing home residents who met the inclusion criteria: (1) municipality authority’s decision of long-term nursing home care; (2) residential time 3 months or longer; (3) informed consent competency recognized by responsible doctor and nurse; and (4) capable of being interviewed.
Principal component analysis and confirmative factor analyses indicated a unidimensional solution. Five of the original 13 items showed low reliability and validity; excluding these items revealed a good model fit for the one-dimensional 8-items measurement model, showing good internal consistency and validity for these 8 items.
Five out of the 13 original items were not high-quality indicators of quality-of-life showing low reliability and validity in this nursing home population. Significant factor loadings, goodness-of-fit indices and significant correlations in the expected directions with the selected constructs (anxiety, depression, self-transcendence, meaning-in-life, nurse-patient interaction, and joy-of-life) supported the psychometric properties of the OPQoL-brief questionnaire. Exploring the essence of quality-of-life when residing in a nursing home is highly warranted, followed by development and validation of new tools assessing quality-of-life in this population. Such knowledge and well-adapted scales for the nursing home population are beneficial and important for the further development of care quality in nursing homes, and consequently for quality-of-life and wellbeing in this population.
Currently, the world faces a shift to an older population;125 million people are now aged 80 years or older . While this shift started in high-income countries (for example in Japan 30% of the population are already over 60 years old), it is now low- and middle-income countries that are experiencing the greatest change. Today, most people can expect to live into their sixties and beyond . Between 2015 and 2050, the proportion of the world’s population over 60 years will nearly double from 12 to 22%; by 2050, the world’s population aged 60 years and older is expected to total 2 billion, up from 900 million in 2015 [1, 2]. All countries in the world face major challenges to ensure that their health and social systems are ready to make the most of this demographic shift .
As people live longer it is important to ensure that the extra years of life are worth living, despite chronic illnesses. Quality-of-life (QoL) and health promotive initiatives for older persons living in nursing homes (NH) will become ever more important in the years to come. The World Health Organization Quality of Life Group  defined QoL as an “individuals’ perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns.” Additionally, global QoL encompasses multiple constructs such as physical health, psychological status, independence level, social relationships, and relationship with significant features of the environment .
QoL conceptual models and instruments for research, evaluation and assessment in diverse populations have been developed since the middle of last century [4, 5]. However, well-adapted and validated QoL measurement models for the NH population are scarce. Accordingly, our understanding of QoL among individuals living in NHs is still limited. During the last decades, spirituality [6,7,8], a sense of meaning-in-life, hope, self-transcendence [9,10,11,12,13,14,15,16,17], social relationships and social support  have become vital aspects of wellbeing and QoL among older adults.
Global QoL is a multifaceted concept, representing a subjective state which is affected by chronic and debilitating health conditions ; such conditions are common among older adults in NHs. Research shows that the NH population is characterized by high age, frailty, mortality, disability, powerlessness, dependency, vulnerability, poor general health and a high symptom burden [19,20,21]. Accordingly, moving to a NH results from numerous losses, illnesses, disabilities, loss of functions and social relations, and facing the end-of-life, all of which detrimental to people’s functioning, independency and QoL. Moreover, older people experience changes in roles, relationships, and living environments that can increase their risk for experiencing social isolation and loneliness ; particularly when moving to a NH. With advancing age, it is inevitable that people lose connection with their friendship networks and that they find it more difficult to initiate new friendships and to belong to new networks. However, a link between QoL and connectedness is emerging in the literature . Despite old age, chronical diseases or frailty; the desire for affiliation and social bonding is an intrinsic human need, also when living in a NH. Hence, the life situation for older adults in NHs might differ significantly from other older adult populations, staying at home or in hospitals. Consequently, a valid and reliable scale assessing QoL in this population is important for the further development of care quality and health promoting intervention in NHs.
While planning for the present study, we searched, broadly and thoroughly, for a valid and reliable measure of QoL suitable for the NH population. Along this road, we found the OPQoL-brief questionnaire , which was developed by a “bottom-up” approach and tested among older adults in Britain . In a frail and vulnerable population such as the NH population, a shorter scale is warranted. Though shorter instruments are more limited in scope and sensitivity than longer measures, the benefits are reduced respondent and research burden and costs. The OPQoL-brief is a shortened version of the OPQOL-35 showing good psychometrics among older adults ; therefore, the OPQoL-brief was selected for this Norwegian study. To the authors knowledge, the OPQoL-brief has not previously been tested by means of confirmatory factor analysis as well as among NH residents.
The aim of this study was to assess the psychometric properties of the Norwegian version of the OPQoL-brief questionnaire in a cognitively intact (not diagnosed with dementia and recognized by the responsible doctor and nurse to have informed consent competency) NH population. The research question was two-fold; (a) how well does the original one-factor measurement model of the OPQoL-brief fit to the observed data? (b) Does the OPQoL-brief reveal good reliability and construct validity in a NH population? We expected the OPQoL- brief to correlate with some established concepts, and tested the following hypotheses:
Hypotheses1 (H1): OPQoL-brief correlates negatively with anxiety and depression.
Hypotheses2 (H2): OPQoL-brief correlates positively with self-transcendence, meaning-in-life, nurse-patient interaction and joy-of-life.
In accordance to the Standards for Educational and Psychological Testing [23, 24], the present research question addressed evidence related to the dimensionality, reliability and construct validity, all of which considered interrelated measurement properties. Dimensionality examines the extent to which the internal components of a test match the defined constructs, and is concerned with the homogeneity of the items . Reliability involves an instrument’s consistence and relative lack of error . This study assessed internal consistence by the reliability coefficients Cronbach’s alpha (α) and composite reliability (ρc). Construct validity refers to how well a scale actually measures the construct it is intended to measure, and is based among others on the constructs’ relationships to other variables . There are two subsets of construct validity: convergent construct validity and discriminant construct validity. Convergent construct validity tests the relationship between the construct and a similar measure; this shows that constructs which are meant to be related are related. Discriminant construct validity tests the relationships between the construct and an unrelated measure; this shows that the constructs are not related to something unexpected. In order to have good construct validity one must have a strong relationship with convergent construct validity and no relationship for discriminant construct validity . In line with the WHO statement of health, salutogenic concepts such as meaning, self-transcendence, joy-of-life and nurse-patient-interaction are found to enhance NH residents’ QoL [10, 17, 20, 27,28,29,30,31,32,33,34,35], and to decrease anxiety and depression [36,37,38]. Therefore, these constructs were selected for assessing convergent construct validity by means of correlational analyses.
Content validity refers to the degree to which a scale has an appropriate, relevant sample of items to represent the construct of interest—that is, whether the content of the specific construct is adequately represented by the items, meaning that the indicators measure all ideas in the theoretical definition . A frequent challenge occurs when the wording of items is too similar—namely, the coefficient alpha, as well as the content validity and dimensionality, are artificially enhanced. Nevertheless, items worded too similarly increase the average correlation among items, which in effect increases the coefficient alpha, yet without adding substantively to the content validity of the measure. Although some similarity among items of a scale is needed to tap into the domain, several items that are mere rewordings of other items are redundant and contain very little new information about the construct . In that sense, theory, validity, reliability, and dimensionality are intertwined.
Design and data collection
Data were collected during 2017–2018 in 27 NHs representing two small and one large urban municipality in Mid-Norway and a large urban municipality in Western Norway. The total sample comprised 188 of 204 (92% response rate) long-term NH residents who met the inclusion criteria: (1) municipality authority’s decision of long-term NH care; (2) residential time 3 months or longer; (3) informed consent competency recognized by responsible doctor and nurse; and (4) capable of taking adequately part in an interview situation. A nurse at the actual ward presented potential participants with oral and written information about the study, their rights as participants and their right to withdraw at any time.
Due to impaired vision, problems holding a pen, fatigue etc., this population have difficulties completing a questionnaire on their own. Therefore, six trained researchers (3 in each part of Norway) conducted one-on-one interviews in the resident’s private room in the NH. Researchers with identical professional background (RN, MSc, trained and experienced in communication with elderly, as well as teaching gerontology at an advanced level) were trained to conduct the interviews in the same manner. The OPQoL-brief was part of a battery of seven scales comprising in total 120 items. To avoid misunderstandings, interviewers read each question loudly, and held a large-print copy of questions and possible responses in front of the participants.
Participants ages ranged between 63 and 104 years (mean 87.4 years, SD = 8.6). The sample consisted of 132 women (73.3%) and 48 men (26.7%), where the mean age for women was 88.3 years (SD = 1.8) and 86 years (SD = 1.2) for the men. In total, 23 were married, 22 cohabitating, 1 was single, 106 were widows/widowers, and 37 were divorced.
The measure of QoL analyzed here is the OPQOL-brief – the short form of the OPQOL-35 questionnaire which was designed to assess QoL among older adults 65+ [41, 42]. The OPQOL-35 has been validated on community-dwelling older populations, and ethnically diverse population samples in Britain [43, 44]. The OPQOL-35 was further tested among geriatric service out-patients in Italy showing excellent applicability to cognitively intact older people, and also to be applicable to most of the people suffering from mild or moderate dementia [45,46,47]. The OPQOL-35 assumes that QoL is a multidimensional concept; the original version includes eight domains [43, 44]. Nevertheless, the factor structure has shown to be unclear; studies of the OPQOL-35 have reported two , four , seven , or nine-factor solutions  based on principal component analysis (PCA). Like the original 35-items version , Chen  extracted eight factors using PCA. No other factor analyses are currently available. A more detailed examination of the factor structure by means of CFA is needed .
The OPQoL-brief comprises of 13 items which are scored Strongly agree = 1, Agree = 2, Neither = 3, Disagree = 4, Strongly disagree = 5 . The items are summed for a total OPQoL-brief score, then positive items are reverse coded, so that higher scores represent higher QoL. The total sum-score ranges from 13 to 65. Examples of items include enjoying one’s life, looking forward to things, staying involved with things, and feeling safe where one lives, etc. (Table 4 in Appendix 1). The OPQoL-brief was found to be a highly reliable and valid measure of QoL in old age . For use in this study, two experts of both languages, English and Norwegian, translated the OPQoL-brief into Norwegian, following the procedure of back-and-forth translation. To better reflect the nuances of the target language , two independent translators did the forward translation into Norwegian (their mother tongue) . One of these was a naive translator who was unaware of the objective of the questionnaire, while the other was a researcher in the field of QoL. No discrepancies appeared. To assure the accuracy of the translation, the initial translation was independently back-translated (from Norwegian into English) by two independent translators. The back-translators were not aware of the intended concept the questionnaire was .
The Self-Transcendence Scale (STS)  assessed interpersonal and intrapersonal self-transcendence. The STS comprises 15 items, each with a score of 1–4, reflecting expanded boundaries of self which are considered to be characteristics of a matured view of life . Total score ranges between 15 and 60, where higher scores indicate higher ST. The STS has shown good psychometric properties [55, 56] and has been translated into Norwegian, and validated in NH patients  showing a two-factor-construct (STS1 & STS2) to be most valid and reliable . The present study applied this two-factor construct (ST1, ST2).
The Purpose-in-Life Test (PIL) assessed meaning-in-life. Based on Frankl’s theory, the PIL was designed to be a general tool assessing meaning [57,58,59,60] and has been commonly used for this purpose [61,62,63]. The PIL is translated into Norwegian  and has previously been used with elderly individuals up to 104 years old [65,66,67]. The Norwegian version has been validated among NH residents, showing good psychometric properties . Each statement is scored from 1 to 7; four represents a neutral value, whereas the numbers from 1 to 7 stretch along a continuum from one extreme feeling to the opposite kind of feeling; higher scores reflect higher meaning-in-life . Total score ranges from 20 to 140.
The Joy-of-Life scale (JoLS) was developed in Norway to assess NH patients’ perceived joy-of-life (JoL) . The intention was to identify essential characteristics of NH patients’ experiences of JoL in their daily life. The JoLS covers domains that identify fundamental qualities stressed in well-being theory [69,70,71,72], nursing care literature [73,74,75,76], and the dimensions found by 29 in-depth interviews on the essence of joy-of-life with NH residents . A 13-items version of the JoLS was found reliable and valid in this population (Haugan, Rinnan et al.2019), and was applied in the present study.
The Nurse-Patient-Interaction Scale (NPIS) developed in Norway, assessed perceived nurse-patient-interaction. The NPIS comprises 14 items identifying essential relational qualities stressed in the nursing literature . This scale is scored from 1 (not at all) to 10 (very much); total score ranges between 14 and 140, where higher numbers indicate better perceived nurse-patient-interaction. The NPIS has shown good psychometric properties with good content validity and reliability among NH residents .
The Hospital Anxiety and Depression Scale (HADS), comprising 14 items, with subscales for anxiety (HADS-A, 7 items) and depression (HADS-D, 7 items) assessed anxiety and depression. Each item is rated from 0 to 3, giving a range of total score between 0 and 21; higher scores indicate more anxiety and depression. The HADS has shown good to acceptable reliability and validity in the NH population .
We obtained approval by the Regional Committee for Medical and Health Research Ethics in Norway (ref.nr 2014/2000/REK Central) as well as from the Management Units at the 27 NHs. Each participant provided voluntarily written informed consent.
The same data were analyzed by descriptive statistics and principal component analysis (PCA) using IBM SPSS version 25, and confirmatory factor analysis (CFA) by means of Stata 15.1 . When evaluating a measurement scale investigating the underlying dimensionality of data and the adequacy of each individual item is central. In these instances, PCA and CFA can provide complementary perspectives on data, giving different pieces of information [25, 80]. The implicit assumption underlying the use of PCA in the present study is the insecurity with respect to the dimensionality of the OPQoL-Brief, which has not been previously tested by means of CFA, neither among NH residents. As previously presented, the OPQoL-brief is a short version of the original OPQoL scale, which has shown 2,4,7,8 and 9 factors. Therefore, a broad perspective on the observed data using PCA followed by the confirmation procedure was used.
Confirmatory factor analysis (CFA) is a sub-model in structural equation modeling that deals specifically with measurement models , accounting for random measurement error, and thus derive a more accurate evaluation of the psychometric properties of the scales used. A high loading of an item indicates that there is much in common between the factor and the respective item . Loadings below 0.32 are considered poor, ≥0.45 fair, ≥0.55 good, ≥0.63 very good, and above 0.71 are excellent . Thus, a good rule of thumb for the minimum loading is .32 , which equates to approximately 10% overlapping variance with the other items in the factor. A “cross-loading” item loads at .32 or higher on two or more factors.
The present study assessed model fit adequacy by χ2-statistics and various fit indices. In line with the ‘rules of thumb’ given as conventional cut-off criteria  the following fit indices were used; χ2-statistics, the Root Mean Square Error of Approximation (RMSEA) and the Standardized Root Mean Square Residual (SRMS) with values below 0.05 indicating good fit, whereas values smaller than 0.10 is interpreted as acceptable . Further, the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI) with acceptable fit set at 0.90 [84, 86] were used. Both skewness and kurtosis were significant and the Robust Maximum Likelihood (RML) estimate procedure was applied. When analyzing continuous but non-normal endogenous variables, the Satorra-Bentler corrected χ2  should be reported .
The OPQoL-brief 13-items mean-scores ranged between 2.99–4.53, showing a total mean of 3.9 (SD = 1.01). In this study, 11 (6.1%) of the NH residents reported QoL mean-score < 3.0 interpreted as a low QoL, 90 (49.7%) stated a high QoL ≥4.0–5.0, while 77 (42.5%) reported QoL mean-scores between 3.0–3.99, representing a modest QoL. Figure 1 displays the distribution of the OPQoL mean-scores, while Table 1 lists the means, standard deviation, Cronbach’s alpha and correlation matrix for the constructs of QoL, ST1, ST2, PIL, JoL, NPIS, HADS-A and HADS-D.
Principal Component Analysis (PCA)
In order to explain as much of the total variance as possible with as few factors as possible, the OPQoL-brief was subjected to PCA. The Kaiser-Meyer-Olkin measure of sampling adequacy exceeded the recommended value of .60 (.84) and Bartlett’s test of sphericity showed statistical significance (p < 0.0001), supporting the factorability of the correlation matrix. We search for the cleanest structure of the concept under investigation and expected the OPQoL-brief to be one- or multi-dimensional with correlated factors. Hence, an oblique rotation such as promax should theoretically render a more accurate solution . PCA with promax rotation and Kaiser Normalization were used; three factors with eigenvalue 1.0 and greater (4.82, 1.72 and 1.06, respectively) were extracted (Table 2). Figure 2 portrays the scree-test of the OPQoL-brief data showing the number of factors to retain is three. Table 2 lists the loadings and variance for this rotated 3-factor solution of the OPQoL-brief suggested by PCA. Yet, this 3-factor solution revealed 9 cross-loadings, with substantial factor loadings on all factors indicating an unclear dimensionality.
Substantial conclusions based solely on PCA should not be drawn ; therefore, we turned to confirmatory factor analysis (CFA).
Confirmatory Factor Analysis (CFA)
Firstly, we checked the original 13-items unidimensional version, revealing a very bad fit to the present data. Consequently, we tested the 3-factor solution suggested by the PCA (Factor 1: items 3, 7, 8, 10, 11, 12; Factor 2: items 2, 5, 6, 7 and Factor 3: items 1, 4, 9, 13). Running CFA, this 3-factor-model did not fully converge and did not provide fit indices; both of which indicating misspecifications. The original OPQoL-brief revealed one dimension; and a 3-factor-solution of the OPQoL-brief construct did not seem theoretically meaningful. Therefore, we turned back to the original unidimensional 13-items model  for further examination.
Model-1 – the original OPQoL-brief unidimensional version
Model-1 comprising 13 items gave significant t-values for all estimates, showing completely standardized factor loadings from .78–.41, and squared multiple correlations (R2) ranging between .61–.16. Some items (item4,5,6,13) disclosed low R2-values (≤0.19) indicating low reliability. The model fit was bad: χ2 = 236.36, (df = 65), χ2/df = 3.64, p = 0.0001, RMSEA = 0.12, p-value for test of close fit = 0.0001, CFI = 0.75, TLI = 0.70, and SRMR = 0.094. However, composite reliability for this one-factor construct was good (ρc = 0.84), indicating good reliability (values ≥0.6 is considered acceptable, while values ≥0.7 are good) [84, 90]. The alpha levels for the various measures indicated an acceptable inter-item consistency with Cronbach’s alpha coefficients of 0.65–0.90 (Table 1) and composite reliability of 0.80–0.84 (Table 3).
An inspection of the standardized residuals and the modification indices (MIs), discovered five significant residuals [item7–3, (0.31) item6–4 (0.21), item5–6 (0.49), item 11–5(− 0.21), item11–6 (− 0.20)]. Furthermore, ten pair of items showed MIs higher than 10, all of which pointing to misspecifications. For the pairs of items 3–7 and items 5–6 the MIs were extremely high (MI = 15.10 and MI = 36.53, respectively).
Item3 (‘I am healthy enough to get out and about’) and item7 (‘I am healthy enough to have my independence’) contain physical functioning and thus share variance. Therefore, it is theoretically rational that they revealed a very high MI. Item3 loaded higher than item7; for that reason, item7 was dismissed from the model. The next step was to consider item6 (‘I try to stay involved with things’), which displayed an extremely high MI with item5 (‘I have social or leisure activities/hobbies that I enjoy doing’). Item6 loaded significantly lower than item5 and was dismissed, and the model was run once more. This 11-items version gave somewhat better fit (χ2 = 137.62, (df = 44), χ2/df = 3.13, p = 0.0001, RMSEA = 0.11, p-value for test of close fit = 0.0001, CFI = 0.83, TLI = 0.78, SRMR = 0.080), although, a poor fit. Now, only one residual was significant, involving the pair of item11–5. Still, several very high MIs were found, involving item11 (‘I take life as it comes and make the best of things’), indicating this item to share unexplained variance with a number of other items. Hence, item 11 was excluded. Next, the estimates pointed at item10 (‘I get pleasure from my home’) and item9 (‘I feel safe where I live’); both concerned with resident’s sense of home while residing in a NH. Item9 showed the best loading and was kept, while item10 was set aside.
Model-2 – the OPQoL-brief 9-items unidimensional version
This modified version (including items 1,2,3,4,5,8,9,12,13), framed Model-2, gave an acceptable fit (χ2 = 54.21, (df = 27), χ2/df = 2.01, p = 0.001, RMSEA = 0.074, p-value for test of close fit = 0.080, CFI = 0.93, TLI = 0.91, SRMR = 0.06). However, even not a good fit.
Model-3 – the OPQoL-brief 8-items unidimensional version.
Finally, dismissing item12 (‘I feel lucky compared to most people’) gave a god fit to the present data: χ2 = 31.55, (df = 20), χ2/df = 1.58, p = 0.048, RMSEA = 0.056, p-value for test of close fit = 0.359, CFI = 0.97, TLI = 0.95 and SRMR = 0.05. This version of the one-dimensional model including eight items (item 1,2,3,4,5,7,9 and 13) was framed Model-3, representing the best fitting model. Figure 3 portrays Model-3, showing the factor loadings, multiple squared correlations (R2), model fit and composite reliability (ρc).
When evaluating a measurement scale, researchers face two important questions: (1) the underlying dimensionality of data (not too many, not too few factors), and (2) the adequacy of the individual items. This study assessed how well the original one-factor measurement model of the OPQoL-brief fit to the observed data, and whether the OPQoL-brief revealed good reliability and construct validity in a Norwegian NH population. Thus, the research question addressed evidence related to the dimensionality, reliability and construct validity of the OPQoL-brief questionnaire in this population.
The scree-test portrayed in Fig. 2 indicated that the number of factors to retain was three. However, two factors showed eigenvalues substantially higher than one, while the third factor was close to one (1.06), along with the next factors showing eigenvalues of 0.98, and 0.88, respectively. Hence, it seems not reasonable to regard the third factor with eigenvalue of 1.06 as ‘major’ and the fourth with eigenvalue of 0.98 as ‘trivial’. When it comes to determining the number of factors, Kaiser’s method (K1) sometimes is problematic and inefficient . As seems to be the case here, the Kaiser-Guttman rule of retaining eigenvalues larger than 1 is not interpretively useful because it tends to result in the retention of too many factors . Despite K1’s widespread use, experts agree that it has deficiencies and that its use is not recommended . PCA needs to balance parsimony with adequately representing underlying correlations, so its utility depends on being able to differentiate major factors from minor ones . By looking at the scree-plot for the PCA in the present study this issue seems evident; one strong factor along with several small factors were portrayed.
Moreover, the rotated 3-factor solution suggested by PCA revealed several cross-loadings with substantial factor loadings on all factors, thwarting the dimensionality. Only four (items 5,6,7,13) loaded solely on one dimension, indicating an unclear dimensionality of the construct and probably a one-dimensional solution like Bowling et al.  presented. Turning to CFA, the analyses suggested a unidimensional solution (Table 3). However, some items seemed troublesome, indicating misspecifications.
Reliability and construct validity are related to the adequacy of the individual items; highly significant standardized factor loadings–preferably > 0.7 indicates that the items perform as good indicators for the QoL construct in the NH population. The square of a standardized factor loading (R2), termed the variance extracted of the item, represents how much variation in an item the latent construct explains . Loadings falling below 0.7 can still be significant, but more of the variance in the measure is error variance than explained variance. Looking at the factor loadings and the R2-values, only three items loaded good-excellent; item1 (λ = 0.82) was excellent, while item2 (λ = 0.69) and item8 (λ = 0.66) displayed good loadings. Contrasting this, item4 (λ = 0.41, R2 = 0.17), item6 (λ = 0.42, R2 = 0.18), and item13 (λ = 0.44, R2 = 0.19) performed like invalid indicators of QoL; the OPQoL-construct explained only a limited amount of the variance in these items. Consequently, the reliability of these indicators was low. The other seven items displayed fair factor loadings ranging between .46–.57. Hence, reliability was acceptable, but not fully supported. An examination of the inter-item correlations revealed plausible correlations (Table 5 in Appendix 2), with the highest values for the pair of items1–2 (r = .62), items5–6 (r = .55), items1–8 (r = .55) and items3–7 (r = .47). Moreover, Cronbach’s alpha (α) (Table 1) and composite reliability (ρc) (Table 3) revealed good values, indicating good internal consistency [84, 90].
Construct validity deals with the accuracy of measurement, reflecting the extent to which a set of measured indicators actually reflect the theoretical latent construct the items are designed to measure . In the present study, convergent construct validity was supported by significant negative correlations between OPQoL-brief and HADS-A and HADS-D as well as positive correlations with ST1, ST2, PIL, NPIS and JOL (Table 1). Both hypotheses (H1 and H2) were supported. Items 1,2 and 8 revealed the best loadings, representing good indicators for QoL in the NH population. Interestingly, item8 (‘I can please myself what I do’) loaded strongly (.66), implying to be a valid indicator of QoL in this population. Considering that NH residents commonly experience idleness, spending many hours doing nothing, waiting, sleeping, this finding is noteworthy. Doing something, being active with something which you like, is essential for QoL among NH residents [95,96,97,98,99].
Content validity is a sub-form of construct validity, referring to whether the OPQoL-brief has an appropriate, relevant sample of items to represent the QoL construct. If the wording of items is too similar, a challenge occurs; items worded too similarly increase the average correlation among items, which in effect increases the coefficient alpha, yet without adding substantively to the content validity of the measure. Firstly, items 3 and 7 possibly are worded too closely; ‘I am healthy enough to get out and about’ (item3) and ‘I am healthy enough to have my independence’ seem to measure the same aspect. Staying in a NH without having dementia means that you on average have 6–7 diagnoses of chronic conditions , which negatively affect health, functioning and independency. Largely, cognitively intact NH residents are not healthy enough to get out and about. Due to illness and health problems, followed by care needs, they have moved to a NH. Consequently, their independency is impeded; many NH residents perceive their institutionalization as the beginning of their loss of independence and autonomy [101,102,103].
The notion that ‘I am healthy enough to have my independence…to get out and about’ might not indicate independence in the NH life situation very well; in fact, it could be the opposite. Striving for independence while you are totally dependent on others might damage your QoL. Although some similarity among items of a scale is needed to tap into the domain, several items that are mere rewordings of other items are redundant and contain very little new information about the construct .
Secondly, the items concerning one’s home (items 9, 10), which for these older adults is a NH, might not be worded specific or precise enough. Many older adults in NHs do not experience the NH as their home , and are grieving over that they had to leave their home, representing a loss to them. The NH is the last stop in their life. The expression that ‘I get pleasure from my home’ (item10) might not be as central as it would be if these individuals were staying in their private home. However, ‘I feel safe where I live’ (item9) seems more appropriate; NH residents highlight the importance of feeling safe to their thriving and QoL [104, 105]. This population is characterized by high age, numerous losses, frailty, mortality, disability, powerlessness, dependency, vulnerability, poor general health, a high symptom burden and facing the end-of-life [19,20,21], all of which increases distress and vulnerability. Thus, feeling safe while staying in a NH seems closely connected to the nurse-patient relationship, care quality and nurse-patient interaction, more than being at ‘my home’. Hence, indicators including the NH working culture, milieu, atmosphere and nurse-patient interaction might be essential domains to include in a QoL measurement for NH residents . Looking at the correlations between the summative scores (Table 1), QoL correlated highly with the nurse-patient-interaction, along with joy-of-life, interpersonal and intrapersonal self-transcendence and meaning-in-life. QoL correlated negatively with anxiety and depression. Thus, convergent construct validity was well supported.
Item12 (‘I feel lucky compared to most people’) did not explain a substantial amount of the variation in the OPQoL-construct (R2 = 0.24). It might be difficult to know who one should compare oneself with. If comparing with the healthy ones coping at home, one might not feel very lucky. Contrary, compared to those who are in hospital waiting for a place in a NH, one might feel lucky. Probably this indicator could be more specified towards the life situation of residing in a NH.
Finally, item11 (‘I take life as it comes and make the best of things’), revealed significant correlations with many items involved in the OPQoL-brief questionnaire. Possibly, item11 covers an attitude and coping mechanism which is very much needed and therefore commonly developed among NH residents. Consequently, this indicator largely relates with the other indicators, sharing variance, and thus blurring the dimensionality and the statistical fit. Including correlated error terms concerning item11 might be an option.
The shortened OPQoL-brief construct was supported by significant factor loadings, several goodness-of-fit indices and significant correlations in the expected directions with the selected constructs. However, a good model fit does not guarantee that we have obtained ‘the true model’; other alternative models might fit the data equally well as the model found .
The effective (listwise) sample size was N = 181, which is considered medium, and close to what is understood as a large sample size. A rate of 10 cases per observed variable is given as a rule of thumb [81, 90]. The models tested in this study included 13 items; accordingly, the sample of N = 181 should be enough. Out of 204 NH patients fulfilling the inclusion criteria, 188 participated, giving a response rate of 92%. This along with almost no missing data represent a strength of this study.
The OPQoL-brief scale was part of a questionnaire comprising 120 items. Accordingly, frail older NH residents might tire when completing the questionnaire, representing a possible bias to their reporting. To avoid such a bias, we carefully selected and trained experienced researchers in conducting the interviews following a standardized procedure, including taking small breaks at specific points during the process. This procedure worked out very well; all participants fulfilled the questionnaire without considerably difficulties. The fact that the researchers visited the participants in the NHs to help fill in the questionnaires might have introduced some bias on the respondents’ responses, which is a limitation of this study.
This study suggests a unidimensional solution of the OPQoL-brief. However, five of the original 13 items emerged to be poor indicators of the OPQoL-construct showing fair reliability and an insufficient validity. The present study suggests that the nine- and eight-items versions revealed an acceptable and a good fit to the data, respectively. Further development and testing of a well-adapted scale assessing QoL in the NH population are required.
Availability of data and materials
The datasets generated and/or analysed during the current study are not publicly available due to Norwegian Act on medical and health research (ACT 2008–06-20 no. 44):§ 38 but are available from the corresponding author on reasonable request. All raw data is in Norwegian.
Comparative Fit Index
Hospital Anxiety and Depression Scale
Hospital Anxiety and Depression Scale, subscale assessing anxiety
Hospital Anxiety and Depression scale, subscale assessing depression
Master of Science
Quality of life
Root Mean Square Error of Approximation
Structural Equation Modelling
Standardized Root Mean Square Residual
Tucker Lewis Index
The World Health Organization
World Health Organization. World report on ageing and health. 2018 2018, February 5th 2019, July 17th].
Kinsella K, He W. An Aging World: 2008, U.C. bureau, editor. Washington, DC: U.S. Department of Health and Human Services National Institutes of Health NATIONAL INSTITUTE ON AGING U.S. Department of Commerce economics and statistics; 2009. p 1-191.
The WHOQOL Group, W., Development of the World Health Organization WHOQOL-BREF quality of life assessment. The WHOQOL Group. Psychol Med., 1998;28(3): p. 551–8.
Bowling, A., Measuring health: a review of quality of life measurement scales. third ed. 2004, Maidenhead, GB. : Open University. 224.
Cooney A, et al. Exploration of the meaning of connectedness for older people in long-term care in context of their quality of life: a review and commentary. Int J Older People Nursing. 2014;9(3):192–9.
Bredle JM, et al. Spiritual well-being as a component of health-related quality of life: the functional assessment of chronic illness therapy—spiritual well-being scale (FACIT-Sp). Religions. 2011;2(1):77–94.
Chirico F. Spiritual well-being in the 21st century: it is time to review the current WHO’s health definition. J Health Soc Sci. 2016;1(1):11–6.
Dhar N, Chaturvedi S, Nandan D. Spiritual Health Scale 2011: Defining and Measuring 4th Dimension of Health. Indian J Community Med. 2011;36(4):275–82.
Bradshaw SA, Playford ED, Riazi A. Living well in care homes: a systematic review of qualitative studies. Age Ageing. 2012;41(4):429–40.
Haugan G. Nurse-patient interaction is a resource for hope, meaning-in-life, and self-transcendence in cognitively intact nursding-home patients. Scand J Caring Sci. 2014;2014(28):74–8.
Haugan G. Meaning-in-life in nursing-home patients: a valuable approach for enhancing psychological and physical well-being? J Clin Nurs. 2014;23(13–14):1830–44.
Haugan, G., et al., Development and psychometric properties of the joy-of-life scale among cognitively intact nursing home patients. Scand J Caring Sci, 2019.
Drageset J, Haugan G, Tranvåg O. Crucial aspects promoting meaning and purpose in life: perceptions of nursing home residents. BMC Geriatrics, Open Access. 2017;17(1):254.
Duggleby W, et al. Hope, older adults, and chronic illness: a metasynthesis of qualitative research. J Adv Nurs. 2012;68(6):1211–23.
Hammer KOEOC. The meaning of hope in nursing research: a meta-synthesis. Scand J Caring Sci. 2009;23(3):549–57.
Reed PG. Theory of self-transcendence, in Middle range theory for nursing. In: Smith MJ, Liehr PR, editors. . New York: Springer Pub; 2018. p. 119–46.
Haugan G, et al. Self-transcendence in nursing home patients - a resource for well-being. J Adv Nurs. 2013;69(5):1147–60.
Bakas T, et al. Systematic review of health-related quality of life models. Health Qual Life Outcomes. 2012;10(34).
Barca ML, et al. Factors associated with depression in Norwegian nursing homes. Int J Geriatr Psychiatry. 2009;24(4):417–25.
Haugan G. Meaning-in-life in nursing-home patients: a correlate to physical and emotional symptoms. J Clin Nurs. 2014;23(7–8):1030–43.
World Health Organization. Global status report on noncommunicable diseases 2010. 2011. p. 176. ISBN: 978 92 4 156422 9.
Bowling A, et al. A short measure of quality of life in older age: the performance of the brief older People's quality of life questionnaire (OPQOL-brief). Arch Gerontol Geriatr. 2013;56(1):181–7.
American Psychological Association and National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999.
Goodwin LD, Leech NL. The meaning of validity in the new standards for educational and psychological testing: implications for measurement courses. Meas Eval Couns Dev. 2003;36(3):181–92.
Netemeyer R, Bearden W, Sharma S. Scaling procedures. Issues and Applications. In: Thousand oaks. London, New Dehli: SAGE Publications, Inc.; 2003.
Ginty, A.T., Psychometric Properties, in Encyclopedia of Behavioral Medicine, G. M.D. and T. J.R., Editors. 2013, Springer: New York, NY.
Haugan G. The relationship between nurse-patient-interaction and meaning-in-life in cognitively intact nursing-home patients. J Adv Nurs. 2014;70(1):107–20.
Haugan G, Moksnes UK, Espnes GA. Nurse-patient-interaction: a resource for hope among cognitively intact nursing home patients. J Holist Nurs. 2013;31(3):152–63.
Haugan, G., Self-transcendence, well-being and nurse-patient interaction in cognitively inatct nursing home patients, in Faculty of Social Sciences and Technology Management, Department of Social Work and Health Science. 2013, Norwegian University of science and technology: Trondheim, Norway p 88.
Haugan G, Hanssen B, Moksnes UK. Self-transcendence, nurse-patient interaction and the outcome of multidimensional well-being in cognitively intact nursing home patients. Scand J Caring Sci. 2013;27(4):882–93.
Haugan G, et al. Self-transcendence and nurse-patient interaction in cognitively intact nursing-home patients. J Clin Nurs. 2012;21:3429–41.
Haugan G, Moksnes UK, Løhre A. Intra-personal self-transcendence, meaning-in-life and nurse-patient interaction: powerful assets for quality of life in cognitively intact nursing home patients. Scand J Caring Sci. 2016;30(4):790–801.
Drageset J, et al. Sense of coherence as a resource in relation to health-related quality of life among mentally intact nursing home residents - a questionnaire study. Health Qual Life Outcomes. 2008;6:85.
Drageset J, et al. Emotional loneliness is associated with mortality among mentally intact nursing home residents with and without cancer: a five-year follow-up study. J Clin Nurs. 2013;22(1–2):106–14.
Drageset J, et al. The impact of social support and sense of coherence on health-related quality of life among nursing home residents--a questionnaire survey in Bergen, Norway. Int J Nurs Stud. 2009;46(1):66–76.
Haugan G, Innstrand ST, Moksnes UK. The effect of nurse-patient-interaction on anxiety and depression in cognitively intact nursing home patients. J Clin Nurs. 2013;22(15–16):2192–205.
Haugan, G. and S.T. Innstrand, The effect of self-transcendence on depression in cognitively intact nursing home patients. ISRN Psychiatry, 2012. 2012(Article ID 301325): p. 10.
Drageset J, Espehaug B, Kirkevold M. The impact of depression and sense of coherence on emotional and social loneliness among nursing home residents without cognitive impairment - a questionnaire survey. J Clin Nurs. 2012;21(7–8):965–74.
Waltz CF, Strickland OL, Lenz ER. Measurement in nursing and health research. 3rd ed. New York: Springer; 2005.
Clark LA, Watson D. Constructing validity: basic issues in objective scale development. Psychol Assess. 1995;73:309–19.
Bowling A, Gabriel ZL. Theories of Quality of Life in Older Age. Aging Soc. 2007;27(827–48).
Bowling, A. And S. P., Which measure of quality of life performs best in older age? A comparison of the OPQOL, CASP-19 and WHOQOL-OLD. J Epidemiol Community Health, 2010. 65: p. 273–280.
Bowling, A., Psychometric properties of the older People’s quality of life questionnaire validity. Curr Gerontol Geriatr Res, 2009. 298950.
Bowling A, Stenner P. Which measure of quality of life performs best in older age? A comparison of the OPQOL, CASP-19 and WHOQOL-OLD. J Epidemiol Community Health. 2011;65:273–80.
Bilotta C, et al. Dimensions and correlates of quality of life according to frailty status: a crosssectional study on community-dwelling older adults referred to an outpatient geriatric service in Italy. Health Qual Life Outcomes. 2010;8(56).
Bilotta C, et al. Older People’s Quality of Life (OPQOL) scores and adverse health outcomes at a one-year follow-up. A prospective cohort study on older outpatients living in the community in Italy. Health Qual Life Outcomes. 2011;9(72).
Bilotta C, et al. Quality of life in older outpatients living alone in the community in Italy. Health Soc Care Commun. 2012;20:32–41.
Mares J, Cigler H, Vachkova E. Czech version of OPQOL-35 questionnaire: the evaluation of the psychometric properties. Health Qual Life Outcomes. 2016;14(93).
Chen Y, Hicks A, While A. Validity and reliability of the modified Chinese version of the Older People's Quality of Life Questionnaire (OPQOL) in older people living alone in China. Int J Older People Nursing. 2014;9(4):306–16.
Hendricson W, et al. Development and initial validation of a dual-language English-Spanish format for the arthritis impact measurement scales. Arthritis Rheum. 1989;32:1153–9.
Beaton, D., et al., Recommendations for the Cross-Cultural Adaptation of the DASH and Quick DASH Outcome Measures. 2007, Toronto: : Institute for Work and Health.
Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: Literature review and proposed guidelines. J Clin Epidemiol. 1993;46:1417–32.
Reed, P.G., Theory of Self-Transcendence, in Middle Range Theory for Nursing, M.J. Smith and P.R. Liehr, Editors. 2008, Springer publishing company, LLC: New York p 105-129.
Reed PG. Demystifying self-transcendence for mental health nursing practice and research. Arch Psychiatr Nurs. 2009;23(5):397–400.
Reed PG. Toward a nursing theory of self-transcendence: deductive reformulation using developmental theories. Adv Nurs Sci. 1991;13(4):64–77.
Haugan G, et al. The self-transcendence scale - an investigation of the factor structure among nursing home patients. J Holist Nurs. 2012;30(3):147–59.
Crumbaugh JC. Cross-validation of purpose-in life test based on Frankl's concepts. J Individ Psychol. 1968;24:74–81.
Crumbaugh JC, Henrion R. The PIL test: administration, interpretation, uses theory and critique. Int Forum Logother. 1988;11(2):76–88.
Crumbaugh JC, Maholick LT. An experimental study in existentialism: the psychometric approach to Frankl’s concept of noogenic neurosis. J Clin Psychol. 1964;20(2):200–7.
Crumbaugh JC, Maholick LT. Manual of Instructions for The Purpose-in-life test. Psychometric Affilitates, in Viktor Frankl Institute of Logotherapy 1969. 1981.
Crumbaugh JC, Maholick LT. Manual of Instructions for the Purpose-in-Life Test. Illinois: Psychometric Affiliates: (P.O. Box 2852, Saratoga, CA 95070) Murfreesboro: Psychometric Affiliates; 1969.
Steger MF. An illustration of issues in factor extraction and indentification of dimensionality in psychological assessment data. J Pers Assess. 2006;86:263–72.
Steger MF, et al. The meaning in life questionnaire: assessing the presence of and search for meaning in life. J Couns Psychol. 2006;53:80–93.
Bondevik, M., The Life of the Oldest Old. Studies concerning loneliness, social contacts, activities of daily living, purpose in life and religiousness, in Department of Public Health and Primary Health Care, Division for Nursing Science. 1997, University of Bergen: Bergen.
Bondevik M, Skogstad A. Loneliness, religiousness, and purpose in life in the oldest old. J Relig Gerontol. 2000;11(1):5–21.
Haugan G, Moksnes UK. Meaning-in-life in nursing-home patients: a validation study of the purpose-in-life test. J Nurs Meas. 2013;21(2):296–319.
Flood M, Scharer K. Creativity enhancement: possibilities for succesful aging. Issues Ment Health Nurs. 2006;27:939–59.
Haugan, G., et al., Development and psychometric properies of the joy-of-life scale in cognitively intact nursing home patients. Scand J Caring Sci, 2019.
Keyes, C.L., Mental health as a complete state: how the salutogenic perspective completes the picture in Bridging Occupational, Organizational and Public Health, G.F. Bauer and O. Hämmig, Editors. 2014, Springer: Netherlands. p. 179–92.
Keyes CL. Promoting and protecting mental health as flourishing: a complementary strategy for improving national mental health. Am Psychol. 2007;62(2):95–108.
Keyes CL, Lopez SJ. Toward a science of mental health. In: Snyder CR, Lopez SJ, editors. Oxford handbook of positive psychology. Oxford: Oxford University Press; 2009. p. 89–95.
Seligman M. Flourish: a visionary new understanding of happiness and well-being. New York: Simon and Schuster; 2012.
Smith M. Review of research related to Watson's theory of caring. Nurs Sci Q. 2004;17(1):13–25.
Watson, J., Nursing: human science and human care. A theory of nursing. 1988, New York: National League for Nursing.
Nåden D, Sæteren B. Cancer patients’ perception of being or not being confirmed. Nurs Ethics. 2006;13(3):222–35.
Watson J. Watson's theory of human caring and subjective living experiences: carative factors/caritas processes as a disciplinary guide to the professional nursing practice. Texto Contexto - Enferm. 2017;16:129–35.
Rinnan E, et al. Joy of life in nursing homes: a qualitative study of what constitutes the essence of joy of life in elderly individuals living in Norwegian nursing homes. Scand J Caring Sci. 2018.
Haugan G, Drageset J. The hospital anxiety and depression scale—dimensionality, reliability and construct validity among cognitively intact nursing home patients. J Affect Disord. 2014;165:8–15.
StataCorp. Stata: Release 16. Statistical Software. College Station: StataCorp LLC; 2019.
Hurley AM, et al. Exploratory and confirmatory factor analysis: guidelines, issues, and alternative. J Organ Behav. 1997;18:667–83.
Brown T. Confirmatory factor analysis for applied research. New York: The Guilford Press; 2006.
Sharma S. Applied multivariate techniques. New York: Wiley; 1996.
Tabachnick BG, Fidell LS. Using Multivariate Statistics. 7th ed. Harlow: Pearson Education Inc; 2019.
Mehmetoglu, M. and T.G. Jakobsen, Applied Statistics using STATA. A guide for the social sciences. 2017, Los Angelos - London - New Dehli - Singapore - Washington DC - Melbourne: SAGE.
Mehmetoglu, M. and T. Jakobsen, Applied Statistics using STATA. A guide for the social sciences. 2017, Los Angelos - London - New Dehli - Singapore - Washington DC - Melbourne: SAGE.
Acock AC. Discovering structural equation modeling using Stata revised ed. Texas: STATA Press; 2013.
Satorra A, Bentler P. Corrections to Test Statistics and Standard Errors in Covariance Structure Analysis. In: Von Eye A, Cloggs C, editors. Latent variables analysis: Applications for developmental research. Thousand Oaks: Sage; 1994. p. 399–419.
Kline R. In: Little TD, editor. Principles and Practice of Structural Equation Modeling. 3rd ed. New York: The Guildford Press; 2011.
Osborne, J.W. and A.B. Costello, Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most From Your Analysis. Practical Assessment, Research & Evaluation, 2005. 10(7, July 2005): p. 1–9.
Hair Jj, et al. Multivariate data analysis. Upper Saddle River: Prentice Hall; 2010.
Fabrigar LR, et al. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods. 1999;3:272–99.
Nunally JC, Bernstein IH. Psychometric theory. New York: McGraw-Hill; 1994.
Raykov T. Estimation of congeneric scale reliability using covariance tructure analysis with nonlinear constraints. Br J Math Stat Psychol. 2001;54(2):315–23.
Fayers P, Machin D, editors. Quality of Life. The assessment, analysis and interpretation of patient-reported outcomes. 2nd ed. Chisester: John Wiley & Sons Ltd.; 2007.
Slettebo A, et al. The significance of meaningful and enjoyable activities for nursing home resident's experiences of dignity. Scand J Caring Sci. 2017;31(4):718–26.
Grönstedt H, et al. Effects of individually tailored physical and daily activities in nursing home residents on activities of daily living, physical performance and physical activity level: a randomized controlled trial. Gerontol. 2013;59(3):220–9.
Brownie S, Horstmanshof L. Creating the conditions for self-fulfilment for aged care residents. Nurs Ethics. 2012:1–10.
Haugland BØ. Meningsfulle aktiviteter på sykehjemmet [Meaningful activities in nurisng homes]. Sykepleien Forskning. 2012;7(1):42–9.
Harper Ice G. Daily life in a nursing home - has it changed in 25 years? J Aging Stud. 2002;16:345–59.
FABBRI E, et al. Aging and Multimorbidity: New Tasks, Priorities, and Frontiers for Integrated Gerontological and Clinical Research. J Am Med Dir Assoc. 2015;16(8):640–7.
Otsuka S, et al. Prospects for introducing the Eden Alternative to Japan. J Gerontol Nurs. 2010;36(3):47–55.
Choi N, Ransom S, Wyllie R. Depression in older nursing home residents: the influence of nursing home environmental stressors, coping, and acceptance of group and individual therapy. Aging Ment Health. 2008;12(5):536–47.
Tuckett A. The meaning of nursing-home: ‘waiting to go up to St. Peter, OK! Waiting house, sad but true’: an Australian perspective. J Aging Stud. 2007;21(2):119–33.
Bergland A, Kirkevold M. Thriving in nursing homes in Norway: contributing aspects described by residents. Int J Nurs Stud. 2006;43(6):681–91.
Bergland A, Kirkevold M. Resident-caregiver relationships and thriving among nursing home residents. Res Nurs Health. 2005;28(5):365–75.
Bollen KA. Structural equations with latent variables. New York: Wiley; 1989.
The authors acknowledge the nursing home residents who voluntarily participated in this study, and the research assistants Eva Rinnan, Brit Moene Kuven, Wenche Mjanger Eide, and Siw Eriksen Taasen who collected the data.
This study received funding from the Norwegian Council of Research, project code 238331.
Ethics approval and consent to participate
Ethical approval was obtained by the Regional Committee for Medical and Health Research Ethics in Norway (ref.nr 2014/2000/REK Central) as well as by the Management Units at the 27 nursing homes which participated in this study. Each participant provided voluntarily written informed consent.
Consent for publication
Each author participated sufficiently in the work to take public responsibility for appropriate portions of the content. All authors agreed to be personally accountable for the author’s own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.
The authors declare no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Haugan, G., Drageset, J., André, B. et al. Assessing quality of life in older adults: psychometric properties of the OPQoL-brief questionnaire in a nursing home population. Health Qual Life Outcomes 18, 1 (2020). https://doi.org/10.1186/s12955-019-1245-3