Health and Quality of Life Outcomes

Background: The 'Schedule for the Evaluation of Individual Quality of Life – Direct Weighting' (SEIQoL-DW) is an instrument developed to measure individual quality of life. Although this instrument has been used in numerous studies, data on validity and reliability are sparse. This study aimed to examine aspects of validity, reliability and responsiveness of the SEIQoL-DW on data obtained in adults with congenital heart disease, by using the new standards of psychological testing. Methods: We evaluated validity evidence based on test content, internal structure, and relations to other variables, as well as the stability and responsiveness of the SEIQoL-DW. Evidence was provided by both theoretical considerations and empirical data. Empirical data were acquired from two studies. Firstly, using a cross-sectional study design, we included 629 patients with congenital heart disease. Secondly, 130 of the 629 initially included patients readministered the questionnaires approximately one year after the first data collection. In addition to the SEIQoL-DW, linear analog scales were used to assess overall quality of life and perceived health. Results: We found that the SEIQoL-DW is not a valid measure of quality of life, but rather assesses determinants that contribute to individuals' quality of life. The SEIQoL-DW consistently proved to be valid and reliable to assess those determinants. However, responsiveness in patients with congenital heart disease may be problematic. Conclusion: Based on theoretical and empirical considerations, the SEIQoL-DW cannot be considered as a quality of life instrument. Nonetheless, it is a valid and reliable instrument to explore determinants for patients' quality of life. Background Quality of life is an increasingly popular concept, as illustrated by an exponential growth of quality of life studies in medical, nursing and health services literature. It has emerged as an important variable for evaluating the quality and outcome of provided health care. For this purpose, a vast amount of tools have been developed to measure quality of life. Most of them are standardized questionnaires or test batteries to obtain information on patients' functioning or self-perceived health. Published: 28 May 2004 Health and Quality of Life Outcomes 2004, 2:27 Received: 23 March 2004 Accepted: 28 May 2004 This article is available from: http://www.hqlo.com/content/2/1/27 © 2004 Moons et al; licensee BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.


Background
Quality of life is an increasingly popular concept, as illustrated by an exponential growth of quality of life studies in medical, nursing and health services literature. It has emerged as an important variable for evaluating the qual-ity and outcome of provided health care. For this purpose, a vast amount of tools have been developed to measure quality of life. Most of them are standardized questionnaires or test batteries to obtain information on patients' functioning or self-perceived health.
About 15 years ago, the use of standard tools for measuring quality of life began to be criticized for several reasons. First, such predetermined tools contain items that may not be relevant for all individuals whose quality of life is assessed [1]. Even when tools are constructed based on data from in-depth interviews with experienced patients, they do not represent the perspective of all patients. Second, standardized tools assume that all aspects applied are of equal importance for all respondents [1], neglecting the variation of importance of different life areas for individual subjects [2]. Third, quality of life questionnaires are mostly focused on limitations and impediments, without considering positive elements that contribute to the quality of life [3]. Measurement of quality of life should therefore include the possibility that quality of life can be evaluated both in positive and negative terms.
Because of these critiques, a paradigm shift in the measurement of quality of life has taken place. Indeed, there are accumulating arguments that an individualized approach of quality of life is preferable above the use of standard questionnaires. Individual quality of life measurements provide the possibility to respondents to indicate the domains that are important for their quality of life, and to subsequently rate how important the respective domains are [2]. One instrument put forward to measure individual quality of life is the 'Schedule for the Evaluation of Individual Quality of Life' (SEIQoL) [4] and its short form: the 'Schedule for the Evaluation of Individual Quality of Life -Direct Weighting' (SEIQoL-DW) [2,5]. The SEIQoL and SEIQol-DW consists of three successive steps. Patients are asked 1) to name the five most important domains for their quality of life, 2) to rate the actual status on each domain, and 3) to indicate the relative weighting of each domain. In the SEIQoL, the third step is done by a judgement analysis of a series of presented cases, while in the SEIQoL-DW, patients can quantify the relative importance of each nominated domain using a colored 5-segment disk. Several studies have been published using these instruments in healthy and nonhealthy populations.
The creators of the SEIQoL and SEIQoL-DW based their instrument on the definition: "quality of life is what the individual determines it to be" [6,4]. From this perspective, it is argued that both instruments have high face and content validity [1,2,5,7,8]. Data on other psychometric properties is, however, sparse [7,8]. Yet, to support further use of these instruments as valid quality of life tools, additional evidence on validity and reliability is required.
Since the development of the SEIQoL and SEIQoL-DW, more than 15 years ago, the concept of quality of life has evolved dramatically. Several concept analyses have been undertaken to increase clarity in the conceptualization of quality of life [9][10][11][12][13][14]. Most of them indicated that quality of life is most appropriately defined in terms of life satisfaction [9][10][11]. Hence, validity testing of quality of life instruments should be based on the newer quality of life conceptualization. Therefore, we aimed to examine aspects of validity, reliability and responsiveness of the SEIQoL-DW on data obtained in adults with congenital heart disease, relying on recent conceptual work in the field of quality of life. Furthermore, the new standards and terminology of psychometric testing were applied [15].

Current conceptualization of quality of life
Before validity of a quality of life measurement can be evaluated, it is necessary to point out what quality of life is [16]. A spectrum of definitions of quality of life exists in the literature. In the early 1990s, Ferrans developed a useful taxonomy of the conceptualizations of quality of life [17,18,9], grouping them into six broad categories: (1) normal life, (2) social utility, (3) happiness/affect, (4) satisfaction with life, (5) achievement of personal goals, and (6) natural capacities.
To critically appraise the appropriateness of the respective conceptualizations, we used 6 criteria inferred from conceptual pitfalls and clarifications with regard to quality of life: 1) quality of life must not be used interchangeably with health status or functional abilities; 2) quality of life relies on a subjective appraisal, rather than on objective parameters; 3) there is a poor distinction between indicators and determinants of quality of life; 4) quality of life can change over time, but does not fluctuate greatly; 5) quality of life can be positively or negatively influenced; 6) assessment of overall quality of life is preferred over health-related quality of life (Moons P, Budts W, De Geest S: Pitfalls in the conceptualization of quality of life: A guide for conceptual clarity, manuscript under review).
Evaluating the different conceptualizations in the context of the conceptual pitfalls reveals that the only conceptualization that successfully deals with all the conceptual problems is the one that considers quality of life in terms of satisfaction with life. Accordingly, we defined quality of life as "the degree of overall life satisfaction that is positively or negatively influenced by individuals' perception of certain aspects of life important to them, including matters both related and unrelated to health". This definition suggests that quality of life should be measured by assessing overall life satisfaction. Other factors, such as family, work, health, etc., may have a positive or negative impact on patient's quality of life. Because these variables are external factors impacting on quality of life, they can be considered as determinants.

Lines of evidence
Validity and reliability are evolving concepts. Validity refers to the degree to which a test or an instrument measures what it intends to measure, and is currently considered to be a unitary concept. This means that there are no distinct types of validity that are mutually exclusive, but that there are several aspects of evidence to consider in ensuring validity. The process of validation involves accumulating evidence to provide a sound scientific basis for the interpretation of test scores entailed by the proposed use of tests, based on former and actual testing of lines of evidence for a specific instrument [15].
The proposed use of the SEIQoL-DW is to measure quality of life from an individual perspective. The individual character of the SEIQoL-DW implies, however, that the conventional psychometrics may be irrelevant, because there is no standard against which the instrument can be tested [8]. Therefore, it is argued that internal reliabilities and validities would be of more interest [8]. However, the new standards of psychometric testing allow alternative approaches to validity testing. We evaluated several sources of validity evidence with respect to the SEIQoL-DW: evidence based on test content, internal structure, and relations to other variables [15]. Although the incorporation of evidence on response processes and on intended and unintended consequences of the use of an instrument received increasing attention [15], these issues will not be addressed in the present study, because of their limited relevance for validity testing of the SEIQoL-DW.
Their relevance lies more in educational and employment testing than in testing clinical phenomena.
We also provide evidence about the reliability of the SEIQoL-DW. Stability of the instrument can be determined by a test-retest in patients who are in a stable clinical and psychosocial condition. With respect to the SEIQoL-DW, assessment of the internal consistency is irrelevant because the items nominated by the patients are not intended to be interrelated. Furthermore, determining inter-rater reliability of the SEIQoL-DW is not useful because it is a self-report instrument.
Evidence about validity and reliability can rely both on theoretical considerations and empirical data. Hence, some of the evidence provided is based on logical reasoning, while other evidence relies on testing of hypothetical relationships. Based upon the validity and reliability evidence to be provided, several research questions and hypotheses were developed (Table 1).

Evidence based on test content
Test content refers to the themes, wording, and format of the questions of an instrument, as well as the guidelines for procedures regarding administration and scoring [15]. Because the SEIQoL-DW is put forward as an individual quality of life measurement, allowing the respondents themselves to nominate the items that are important for their quality of life, it is argued that the content of the SEIQoL-DW is by definition valid [1,2,5,7,8]. Given the Table 1

Lines of validity, reliability and responsiveness
Evidence based on test content Q1: Does the SEIQoL-DW measure quality of life, given the new conceptualization? Q2: In how many patients are the responses on the SEIQoL-DW invalid? Q3: What is the percentage of positive and negative domains nominated by the patients? Evidence based on internal structure H1: There is a low to moderate correlation between the scores of the actual status and the relative importance.

Evidence based on relations with other variables
H2: There is a high correlation between the scores of the actual status in patients who reported health as important and the linear analog scale of health status. H3: There is a lower score in the actual status of financial means and material well-being in patients who are unemployed or looking for work than in others. H4: There is a lower score in the actual status of job/education in patients who are unemployed or looking for work than in others. H5: There is a lower score in the actual status of health in patients who are not able to work due to disabilities. H6: There is a low to moderate correlation between the SEIQoL-DW index score and the linear analog scale of quality of life. new conceptualization of quality of life, it is however necessary to re-evaluate whether the SEIQoL-DW does indeed measure quality of life (Q1).
The application of the SEIQoL-DW is complex. Even respondents with normal cognitive functioning may have difficulty understanding the system [19,7]. It is therefore appropriate to examine for how many respondents the SEIQoL-DW scores are invalid (Q2).
Quality of life is increasingly considered to be a positivistic concept (see above: Moons P, et al, manuscript under review). Indeed, traditional quality of life assessments focus primarily on limitations and impediments, without considering positive elements that contribute favorably to quality of life. However, our definition entails that quality of life can be determined both by positive and negative aspects. It is therefore appropriate to assess the number of positive and negative issues expressed by the respondents (Q3).

Evidence based on internal structure
Analysis of the internal structure of a test indicates the degree to which the relationship among the items and components conform to the construct as operationally defined [15]. The SEIQol-DW consists of three successive steps, assessing different relevant aspects. If respondents do not understand the distinction between the actual status (step 2) and the relative importance (step 3), a high correlation between the two scores can be expected. Therefore, evidence on the internal structure of the SEIQoL-DW is provided if the scores on the actual status and the relative importance are low to moderately correlated (H1).

Evidence based on relation with other variables
This aspect of validity corresponds with the association between the test scores and other variables that the test is expected to correlate with or predict, and also other variables that the test is not expected to correlate with [15]. The SEIQoL-DW index scores, as well as scores on particular domains or components, can be assessed for correlations with external variables, such as demographics or scores of other quality of life tools. In this respect, six hypotheses were proposed (H2 to H6) ( Table 1).

Evidence on reliability
Reliability refers to the consistency of a measurement when the testing procedure is repeated on a population of individuals or groups, in other circumstances or at other time points [15]. In the present study, stability of the SEIQoL-DW was determined by a test-retest in patients who are in a stable clinical and psychosocial condition (assessed by interview and medical record). It was hypothesized that the SEIQoL-DW index score would remain sta-ble in patients in whom no medical or psychosocial changes occur during an interval of one year (H7).

Evidence on responsiveness
Responsiveness refers to the ability of an instrument to detect clinically important changes. In this respect, it is assumed that changes in the scores of the actual status in patients who reported health as important are highly correlated with changes on the linear analog scale of health status (H8). On the other hand, it can be hypothesized that changes in health status are not or only marginally correlated with changes in SEIQoL-DW index score (H9). Furthermore, it could be assessed whether there is a floor or ceiling effect in the responses (Q4).

Study population
Empirical evidence in this paper is based on data from two studies employing the SEIQoL-DW in congenital heart disease. These studies have been approved by the Institutional Review Board. Using a cross-sectional study design, we examined 629 adults with congenital heart disease. Fifty of the 629 patients (8%) evaluated in our study had to be excluded because their responses were considered invalid for the following reasons: the respondents failed to completely understand the SEIQoL-DW, their answers were inaccurate, or the person accompanying the respondent provided the answers instead of the patient. Demographic characteristics of the remaining 579 patients are specified in table 2. Patients could be included in this study if they were 18 years or older, literate, Dutch-speaking, and provided verbal informed consent. Exclusion criteria were: first visit to our outpatient clinic, mental retardation, and referral for or follow-up after percutaneous closure of an atrial septal defect or a patent foramen ovale. A detailed description of the sampling method and the data collection procedure was given in a related article (Moons P, Van Deyk K, Marquet K, Raes E, De Bleser L, Budts W, De Geest S: Individual quality of life in adults with congenital heart disease: A paradigm shift, under review).
In a 9-month period, a subset of 144 of the initial 579 patients was asked to complete the questionnaires a second time, to perform a test-retest. The time interval between the two data collections was approximately one year. Seven patients (5%) declined to participate in the follow-up study, three (2%) indicated that their condition was unchanged and felt that readministration of the questionnaires was unnecessary, and four (3%) were excluded for practical reasons. This resulted in a sample of 130 individuals who were followed-up longitudinally. Except for the median frequency of follow-up, characteristics of these 130 patients were equivalent to those of the entire sample (Table 2).

Variables and measurement
The instrument under study was a Dutch version of the SEIQoL-DW. The use of the SEIQoL-DW permits the calculation of an index score, by summing the products of the rated level and weighting for each of the 5 areas. This index ranges from 0 to 100.
In addition to the SEIQoL-DW, both overall quality of life and perceived health status were measured using a Linear Analog Scale (LAS). This is a vertical, graded, 10-centimeter line, ranging from 0 to 100. The use of these LASs allows patients to give their own rating of their overall perceived quality of life or subjective health. Several studies have shown that both the LAS for quality of life and health status is valid, reliable and responsive to changes in clinical conditions [20][21][22]. Demographics and relevant clinical information were collected from medical records.

Data analysis
Statistical analyses were performed with SPSS statistical software version 10.0 (SPSS inc., Chicago, IL). For the analysis of the nominated areas using the SEIQoL-DW, a standard qualitative analytic procedure was used. The domains, as described by the respondents, were transcribed verbatim. The individual statements, in their original form, were subsequently sorted and clustered, according to common content. Each cluster was subjectively labeled according to the best description of the meaning of the statements in that cluster, e.g. family, financial means and material well-being, health.
Descriptive statistics were expressed in percentages, medians and quartiles. In testing hypothetical relationships, the Pearson's correlation coefficient and Student's t-test were calculated to reject or confirm the hypotheses. Twosided tests were used, and the level of significance was set at p < 0.05.

Evidence based on test content
The SEIQoL-DW was initially developed to be a quality of life instrument. However, rather than being an indicator of quality of life, we believe that the SEIQoL-DW measures determinants of quality of life because respondents are explicitly asked to nominate domains that are most important for their quality of life (Q1). From this point of view, the SEIQoL-DW has to be considered as a tool to explore relevant determinants of quality of life instead of measuring quality of life itself.
We calculated the number of patients for whom the responses on the SEIQoL-DW were invalid (Q2). From the 629 patients initially included, 50 of them (8%) did not provide valid responses. Reasons for this were: an apparent non-understanding of the SEIQoL-DW procedure, inaccuracy of the answers, and interference from the accompanying person.  This means that respondents are keener to emphasize positive aspects, contributing to a better quality of life, than to focus on problems and concerns.

Evidence based on internal structure
The internal structure of the SEIQoL-DW was evaluated by calculating the association between the actual status and the relative importance of the nominated domains (H1).
We found a correlation of r = 0.26 (p < 0.001), confirming the hypothesized low to moderate correlation between the scores on the actual status and the relative importance.

Evidence based on relation with other variables
It was hypothesized that the score on the LAS for perceived health would be highly correlated with the score on the actual status in patients who reported health as important for their quality of life (H2). Indeed, we found a correlation coefficient of r = 0.69 (p < 0.001) between the two variables.
In patients who were unemployed or looking for work, it could be assumed that their rating of the actual status on financial means and material well-being was lower than in other patients (H3). The results corroborated this hypothesis, as the score of unemployed patients was significantly lower (t = 2.46; p = 0.015).
The same group of patients was also expected to score lower on actual status of job/education (H4). Patients who were unemployed did, indeed, report a significantly lower score on job/education than patients who were employed or students (t = 7.9; p < 0.001).
Patients who are not able to work due to disabilities probably tend to perceive their health as worse than their nondisabled counterparts (H5). This hypothesis was substantiated by the significant difference between the two groups of patients (t = 2.76; p = 0.006).
Because it is assumed that the SEIQoL-DW is not measuring quality of life, but rather determinants of quality of life, a low to moderate correlation between the SEIQoL-DW index score and the LAS of quality of life was hypothesized (H6). A correlation coefficient of r = 0.48 (p < 0.001) was found confirming the hypothesis that the SEIQoL-DW is not an indicator for quality of life.

Evidence on reliability
To evaluate the stability of the SEIQoL-DW, we performed a test-retest in 98 patients for whom no medical or psychosocial changes occurred during an interval of one year (H7). A paired t-test showed no difference in scoring between the test and retest (t = 0.59; p = 0.56).

Evidence on responsiveness
As the scores on the LAS for perceived health are highly correlated with the scores on the actual status in patients who reported health as important for their quality of life, it was hypothesized that the changes in scores on both scales between the test and retest were highly interrelated as well (H8). However, we found that the changes in health state using the LAS and the SEIQoL-DW were not correlated (r = 0.16; p = 0.23).
Health is only one determinant of quality of life. It is therefore hypothesized that a deterioration of the health status does not necessarily result in a decreased SEIQoL-DW index score (H9). Twenty-two patients experienced complications between the test and retest, such as arrhythmias, pulmonary embolism, endocarditis, or non-cardiac co-morbidities. This change in health status corresponded with a decrease in perceived health on a LAS, while the quality of life on the LAS remained stable. The score on the SEIQoL-DW index increased from 74.1 to 80.8 (t = 2.74; p = 0.012), demonstrating that the SEIQoL-DW index scores are independent from changes in health.
Floor and ceiling effects were evaluated by calculating the percentage of patients with the lowest (0) and highest possible score (100) on the SEIQoL-DW index (Q4). Low floor and ceiling scores were observed with 0% (0/579) having the lowest possible score and 1% (6/579) having the highest possible score.

Discussion
In this study, we aimed at a detailed appraisal of the psychometric properties of the SEIQoL-DW, by analyzing data obtained in adults with congenital heart disease. For this purpose, we used the new standards on psychometric testing [15].

Validity
The SEIQoL-DW was created to measure individual quality of life. However, the present study indicates that this instrument measures determinants of quality of life, rather than quality of life itself. Indeed, patients are asked to nominate the 5 domains that are most important to maintain, enhance or impair their quality of life. Furthermore, we found a moderate correlation between the SEIQoL-DW index score and the score on LAS of quality of life, indicating that these two instruments are measuring different concepts. Therefore, the SEIQoL-DW cannot be considered as a quality of life instrument, but more as a tool to appraise individually relevant determinants of quality of life.
The SEIQoL-DW procedure may be difficult to understand by respondents [19,7]. In the present study, this problem was observed in 50 of the 629 patients (8%). This means that only a few patients were not able to provide valid answers. Post-hoc analysis revealed that 29 of these 50 patients (60.4%) had followed or were pursuing vocational high school, indicating that this problem may be related with a lower level of education.
Traditionally, quality of life is measured in terms of limitations and impediments. However, it is increasingly considered to be a positivistic concept. This issue is confirmed by the present validation study, since only 5.3% of the nominated issues were expressed negatively. This means that respondents spontaneously emphasize positive aspects, contributing to a better quality of life, than focus on problems and concerns.
All hypotheses put forward in this validation study could be accepted. The relationships between scores on the actual status of specific areas and other clinical or demographic variables were confirmed. Conversely, if no or only a weak relationship was assumed, this was corroborated by the data. All these aspects offer critical evidence on the validity of the SEIQoL-DW in measuring determinants of quality of life.

Reliability
Stability of the SEIQoL-DW was assessed by conducting a test-retest in patients with a stable clinical and psychosocial condition. The SEIQoL-DW index scores did not change over an interval of 1 year, supporting the stability of the instrument. Since the SEIQoL-DW is a self-report instrument in which respondents nominate the five most important domains in life, inter-rater reliability and internal consistency were not relevant.

Responsiveness
The instrument did not suffer from a floor or ceiling effect. Nonetheless, the expected high correlation between the change in health status on the LAS and the change in the actual status of patients who reported health as important was not confirmed. Hence, this hypothesis was rejected, indicating that the responsiveness of the SEIQoL-DW in patients with congenital heart disease might be problematic. Previous research, in which individual quality of life was assessed before and after a surgical procedure, significant improvements have been found [4,23]. Therefore, this issue needs to be scrutinized in future studies.

Methodological issues
The new standards of psychometric testing indicate that "a sound validity argument integrates various strands of evidence into a coherent account of the degree to which existing evidence and theory support the intended interpretation of test scores for specific uses" [15]. The present article provides additional evidence on the validity and reliability of the SEIQoL-DW. However, this does not warrant valid and reliable results when the SEIQoL-DW is used in other settings or patient populations. Indeed, data in this study were obtained in adult patients with congenital heart disease during their regular followup visit at an outpatient clinic.
It was previously argued that the conventional psychometric testing may be irrelevant for the SEIQoL-DW, and that internal reliabilities and validities could be more of interest [8]. Nonetheless, the present study was able to assess typical psychometric properties using the new standards, and can therefore serve as an exemplar of how validity and reliability of this instrument can be evaluated. More specifically, the techniques used can be replicated in future validation studies.
Before the validity of an instrument can be evaluated, it is critical to define the underlying concept [16]. In this respect, we previously undertook an in-depth conceptualization of quality of life (see above: Moons P, et al. manuscript under review). Based on this conceptual work, a definition of quality of life was constructed. This definition was imperative to check whether the SEIQoL-DW was measuring quality of life or not.
In this study, we did not question whether the calculation of the SEIQoL-DW index is appropriate. It has been previously argued that such aggregation of potentially unrelated domains may be improper [24].

Conclusion
This study aimed to provide additional evidence on validity, reliability and responsiveness of the SEIQoL-DW. For this purpose, the new standards of psychological testing were applied. Relying on theoretical and empirical considerations, we found that the SEIQoL-DW does not measure quality of life, but rather determinants that contribute to the individual quality of life of the respondent. Therefore, the SEIQoL-DW cannot be considered as a quality of life instrument as such. However, we provided consistent evidence that the instrument is valid and reliable to assess the determinants of quality of life. From this point of view, the use of the SEIQoL-DW in research and clinical practice is supported, because the instrument can offer crucial information for health care professionals to better understand the consequences of a medical condition on patients' quality of life. Further validation studies in other patient populations and other settings are, however, suggested to obtain more evidence on the psychometric properties of this scale.