A brief symptom index for advanced renal cell carcinoma
© Eton et al; licensee BioMed Central Ltd. 2006
Received: 28 June 2006
Accepted: 26 September 2006
Published: 26 September 2006
Our objective was to test a brief, symptom index for advanced renal cell carcinoma, a disease affecting over 38,000 Americans each year and often diagnosed in late stages.
We conducted secondary data analyses on patient-reported outcomes of 209 metastatic renal cell carcinoma patients participating in a Phase III clinical trial. Patient-reported outcomes, obtained from the FACT-Biological Response Modifier (FACT-BRM) scale, were available at baseline, 2, and 8 weeks. We analyzed data from eight FACT-BRM items previously identified by clinical experts to represent the most important symptoms of advanced renal cell carcinoma. Items comprising this index assess nausea, pain, appetite, perceived sickness, fatigue and weakness, with higher scores indicating fewer symptoms. We determined reliability and validity of the index and estimated a minimally important difference.
The index had excellent internal reliability at all three time points (alphas ≥ 0.83). Baseline scores were able to discriminate patients across Karnofsky performance status, number of metastatic sites, and risk group categories (p s < .01). Mean index scores declined over time likely indicative of the toxic nature of the administered treatments. Distribution- and anchor-based methods converged on a minimally important difference estimate of 2 to 3 points.
The 8-item index of patient-reported symptoms of renal cell carcinoma appears to be a psychometrically sound measure. It is a brief, reliable, and valid measure that can easily be adapted for use in clinical trials and observational studies.
Renal cell carcinoma (RCC) is estimated to affect over 38,000 Americans annually and account for over 12,000 deaths . It is often diagnosed in advanced stages, with varied sites of metastases. Due to uniform resistance to most cytotoxic chemotherapy agents , patients with metastatic RCC typically have very poor prognoses. Though not curative, biological response modifiers such as interleukin-2 and interferon-alfa have proven effective at delaying tumor growth and disease progression. However, these treatments can result in severe and sometimes dose-limiting toxicities such as fever, chills, nausea, vomiting, hypotension, and fatigue .
Recently, less-toxic targeted therapies have shown anti-tumor activity in clinical trials of metastatic RCC. Among the most promising agents are bevacizumab (Avastin), sorafenib (Nexavar) and sunitinib (Sutent). Bevacizumab, approved for the treatment of metastatic colorectal cancer in 2004, has also shown clinical activity in metastatic RCC . Sorafenib, an oral multi-kinase inhibitor, has demonstrated anti-tumor activity in several solid tumors including RCC [5, 6]. A recent Phase II trial demonstrating the efficacy of second-line sunitinib – a multi-targeted, tyrosine kinase inhibitor – has led to the rapid approval of this drug for the treatment of metastatic RCC . Though better tolerated than immunotherapy, such targeted therapies have been associated with a host of toxicities including fatigue, diarrhea, nausea, dyspepsia, hypertension, proteinuria, rash, and malaise. Hence, regardless of treatment type, the clinical picture of advanced RCC features an array of symptoms and complications.
Measurement of symptoms and complications has often been done using common toxicity criteria, global measures of performance status, and/or formal quality of life assessment. Though the latter has proven highly effective at measuring subjective patient status in many functional areas, regulatory agencies and clinicians sometimes prefer briefer, symptom-focused instruments in clinical trials. To meet the need for brief and focused assessment of symptoms and complications associated with advanced RCC and its treatment, we used clinical experts to identify an 8-item index of questions from an available health-related quality of life (HRQL) questionnaire. Here we attempt to validate this index using data from a Phase III clinical trial. A secondary objective was to determine a minimally important score difference for the index.
Development of the index
I have a lack of energy
I have nausea
I have pain
I feel sick
I am forced to spend time in bed
I get tired easily
I feel weak all over
I have a good appetite
We maintained the 5-point ordinal rating scale of the original FACT-BRM, with responses ranging from 0 – 'not at all' to 4 – 'very much'. An aggregated symptom index score is created by first reverse-coding the negatively-worded items (all items except "I have a good appetite") then summing the responses to all 8 items so that a higher score indicates fewer symptoms and better well-being. Hence, the worst possible score is 0 and the best possible score is 32.
Validation of the index
We conducted secondary data analyses on 209 advanced RCC patients with available HRQL data (70% male; mean age = 58.1 years) who were entered on a multi-center randomized Phase III trial comparing interferon with or without 13-cis-retinoic acid. Participating centers included Memorial Sloan-Kettering Cancer Center (54 patients) and member institutions of the Eastern Cooperative Oncology Group (155 patients). The trial was approved by the institutional review boards of each participating center. All patients provided informed consent. For more complete trial details see Motzer and colleagues (2000) . Patients in the trial were required to have a pretreatment Karnofsky Performance Status greater than 60% and an estimated life expectancy of more than 3 months. Over half (53%) of patients in this analysis had a prior nephrectomy. The full 40-item FACT-BRM scale was used in the trial. For validation purposes, we extracted and scored the eight FACT-BRM items constituting the proposed symptom index. Assessments were conducted at baseline (prior to treatment) and at 2, 8, 17, 34, and 52 weeks after the initiation of therapy. Since completion rates fall well below 50% by 17 weeks, we considered only the baseline, 2 week, and 8 week data. Data were aggregated across treatment.
Internal reliability was determined by computing Cronbach's alpha coefficients and inter-item correlations at each of the three time points. Concurrent validity was determined by comparing baseline index scores across various clinical parameters using one-way analyses of variance (ANOVA) and independent sample t-test. Index scores were compared across Karnofsky performance status, number of identified metastatic sites, and prognostic risk group. These parameters were either directly available from clinical report forms or, in the case of risk group, determined from a composite of pretreatment features available on the forms. We determined responsiveness to change over time by comparing baseline to follow-up index scores using repeated-measures ANOVA. Least significant difference post-hoc tests were used to specify pairwise differences for any significant omnibus F-test. Finally, we computed an estimate for a minimally important difference (MID) for the index using both distribution- and anchor-based analyses . The MID is the smallest difference in score likely to be clinically meaningful to patients and clinicians. Distribution-based measures included 1/3 and 1/2 of the standard deviation and one standard error of measurement (SEM). Anchor-based analyses compare index score differences between clinically distinct groups. Magnitude differences falling within an effect size range of 0.20 to 0.60 will be those considered minimally important. The effect size is computed as a mean difference between clinically distinct groups divided by the pooled within-group standard deviation.
Internal consistency reliability for the index was quite good in this sample of patients. Cronbach's alpha coefficients were 0.83, 0.86, and 0.86 at baseline, 2 weeks, and 8 weeks, respectively. Inter-item correlations demonstrate that the symptoms and complications represented in the items appear to co-vary with one another. After reverse-coding of negatively-worded items, the number of inter-item correlations reaching at least rho = 0.25 was 23 of 28 (82%) at baseline, 27 of 28 (96%) at 2 weeks, and 26 of 28 (93%) at 8 weeks. The number of inter-item correlations reaching at least rho = 0.50 was 7 of 28 (25%) at baseline, 10 of 28 (36%) at 2 weeks, and 11 of 28 (39%) at 8 weeks.
Mean Index Scores By Baseline Clinical Parameters
Karnofsky rating 2
90 (n = 113)
80 (n = 45)
70 (n = 30)
Adjacent category mean differences & effect sizes
Index score 1
24.76 (sd = 5.62)
22.89 (sd = 6.84)
18.34 (sd = 4.65)
F(2, 185) = 14.67***
80,90 > 70
1.87 (0.30), 4.55 (0.73)
No. of metastatic sites
0 or 1 site (n = 72)
2 or more sites (n = 116)
Mean difference & effect size
24.81 (sd = 5.57)
22.34 (sd = 6.42)
t(186) = 2.69**
Risk group 3
Favorable (F) (n = 47)
Intermediate (I) (n = 95)
Poor (P) (n = 41)
Adjacent category mean differences & effect sizes
26.96 (sd = 3.82)
23.35 (sd = 6.20)
19.47 (sd = 6.06)
F(2, 180) = 19.18***
F > I > P
3.61 (0.58), 3.88 (0.63)
A previously developed prognostic factor model was used to define and categorize patients into risk groups. The model, derived from an analysis of 670 advanced RCC patients treated in clinical trials at the Memorial Sloan-Kettering Cancer Center between 1975 and 1996, classifies patients into risk groups based on pretreatment clinical features associated with shorter survival time . These include: low performance status (KPS < 80), high lactate dehydrogenase (≥ 1.5 × 200 U/L – the upper limit of normal), low hemoglobin level (≤ the lower limit of normal: normal Hgbmale = 13 g/dL and normal Hgbfemale = 11.5 g/dL), high corrected serum calcium (≥ 10 mg/dL), and absence of nephrectomy. Patients with 0 factors present are classified as favorable risk, those with 1 or 2 factors present are classified as intermediate risk, and those with 3 or more factors present are classified as poor risk. Baseline index scores were compared across these three risk groups (see Table 1). Patients with a favorable risk had significantly higher mean baseline index scores than did patients with either an intermediate or a poor risk (p s < .001). Patients with an intermediate risk had a higher mean baseline index score than patients with a poor risk (p < .001).
Responsiveness to change
Minimally important difference
A minimally important difference (MID) was estimated by combining multiple distribution- and anchor-based analyses. The standard deviations of baseline, 2 week, and 8 week index scores were divided by 3 and 2 to determine 1/3 and 1/2 standard deviation (SD) estimates for an MID. Estimates based on 1/3 SD ranged from 2.07 to 2.38; estimates based on 1/2 SD ranged from 3.11 to 3.58. The SD-based criteria were supplemented by a second distribution-based parameter, the standard error of measurement (SEM). The SEM was computed at each of the three time points as, SD × (1 - reliability)1/2. SEMs for the three time points ranged from 2.56 to 2.68. Using these three distribution-based parameters, it appears that the MID for this index falls within the range of 2 to 3 points.
Anchor-based analyses were used to refine or confirm the MID estimate. Score differences between distinct groups were computed along with corresponding effect sizes. The baseline cross-sectional comparisons in the table were used to define these groups. Clinically distinct groups were defined as adjacent response categories for each of the clinical parameters used. For example, in analyses of Karnofsky performance status, patients with a KPS rating of 90 were compared to patients with a KPS rating of 80 and patients with a KPS rating of 80 were compared to patients with a KPS rating of 70. Comparing index scores of patients with a KPS rating of 90 to those with a KPS rating of 70 was not used since this contrast exceeds a minimal level of clinical distinction. These adjacent category differences in index scores as well as their corresponding effect sizes are shown in the last column of Table 1. Only the magnitude differences that fall within an effect size range of 0.20 to 0.60 were considered minimally important. This corresponds to a 'small' to 'medium' effect , which we have found to be a reasonable approximation of an MID on FACT measures . Three of the five minimal group differences fell within this effect size range. These three mean differences ranged from 1.87 to 3.61. Combining these results with the results of the distribution-based analyses, we conclude that the MID on the index most likely falls within the range of 2 to 3 points.
Discussion and conclusion
Our newly developed index for assessing symptoms and complications associated with advanced RCC performed well psychometrically. The index had excellent internal reliability across all three time points. The moderate to high inter-item correlations indicate that this set of items represents a constellation of coexisting symptoms. This is further borne out by the longitudinal changes in item scores. All symptoms tended to worsen from pretreatment to the on-therapy follow-up points. Hence, aggregating these eight items into a common index score appears justified and is in line with recent FDA guidance on the construction and use of patient-reported outcome measures (PROs) . Supportive of its validity, baseline scores on the index differentiated clinical patient groupings based on Karnofsky performance ratings, number of metastatic sites, and prognostic risk. A combination of distribution and anchor-based analyses identified an MID estimate of 2 to 3 points.
There are a few noteworthy limitations to this study. First, the development of the index relied entirely on physician input. Results could have been different had input from patients also been sought. Future efforts to improve and refine the index would benefit from patient query. Second, some of the newer targeted, anti-cancer agents may result in toxicities (i.e., rash and diarrhea) that are not directly addressed in the current index. To address these issues, it may be possible to augment the index with a few items drawn from other established measures including those within the FACT measurement system (see review by Eton, Yost & Cella) . However, any substantial modification to the existing index will require re-validation. Third, some of the group sample sizes in the baseline index comparisons are modest (N's < 50). This may limit the generalizability of the validation and MID results. Finally, we used only cross-sectional (baseline) data in the anchor-based analyses for determining the MID. Confirmation of the MID using longitudinal methods is warranted. Furthermore, replication of the MID analyses in other RCC samples, especially in patients receiving therapies different from those studied in this report, would enhance the generalizability of the estimate.
Based on these results this index would appear to be a reasonable choice as an endpoint in clinical studies of advanced RCC. It is a brief, clinically-derived measure with excellent psychometric properties. Many current and planned Phase II and III clinical trials in the RCC setting include therapeutic combinations in which an array of complications may emerge [6, 15]. Some of these trials combine standard immunotherapy with newer targeted agents (e.g., bevacizumab, sorafenib, sunitinib). Others are directly comparing the clinical benefit of newer targeted therapies versus standard immunotherapy. Given this picture, symptom assessment will need to be broad enough to capture toxicities associated with both immuno- and targeted therapy as well as symptoms of metastatic disease. The index reported herein provides the clinical investigator with a good foundation for an assessment of PRO endpoints in metastatic RCC.
This study was supported in part by a grant from the National Cancer Institute (CA 05826) and by Genentech, Inc., South San Francisco, CA. We thank Joel Wallace and Anita Chawla at Genentech, Inc. for conceptual assistance and technical support.
- American Cancer Society: Cancer Facts & Figures 2006. Atlanta, GA, American Cancer Society; 2006.Google Scholar
- Motzer RJ, Russo P: Systemic therapy for renal cell carcinoma. J Urol 2000, 163: 408–417. 10.1016/S0022-5347(05)67889-5PubMedView ArticleGoogle Scholar
- Borden EC, Parkinson D: A perspective on the clinical effectiveness and tolerance of interferon-alpha. Semin Oncol 1998, 25: 3–8.PubMedGoogle Scholar
- Yang JC, Haworth L, Sherry RM, Hwu P, Schwartzentruber DJ, Topalian SL, Steinberg SM, Chen SX, Rosenberg SA: A randomized trial of bevacizumab, an anti-vascular endothelial growth factor antibody, for metastatic renal cancer. N Engl J Med 2003, 349: 427–434. 10.1056/NEJMoa021491PubMed CentralPubMedView ArticleGoogle Scholar
- Ratain MJ, Flaherty KT, Stadler WM, O'Dwyer P, Kaye S, Xiong H, Patnaik A, Gore M, Lee RJ, Eisen T: Preliminary antitumor activity of BAY 43–9006 in metastatic renal cell carcinoma and other advanced refractory solid tumors in a phase II randomized discontinuation trial (RDT) [abstract 4501]: ; New Orleans, LA. Volume 22. ASCO; 2004:381s.Google Scholar
- Stadler WM: Targeted agents for the treatment of advanced renal cell carcinoma. Cancer 2005, 104: 2323–2333. 10.1002/cncr.21453PubMedView ArticleGoogle Scholar
- Motzer RJ, Michaelson MD, Redman BG, Hudes GR, Wilding G, Figlin RA, Ginsberg MS, Kim ST, Baum CM, DePrimo SE, Li JZ, Bello CL, Theuer CP, George DJ, Rini BI: Activity of SU11248, a multitargeted inhibitor of vascular endothelial growth factor receptor and platelet-derived growth factor receptor, in patients with metastatic renal cell carcinoma. J Clin Oncol 2006, 24: 16–24. 10.1200/JCO.2005.02.2574PubMedView ArticleGoogle Scholar
- Bacik J, Mazumdar M, Fairclough DL, Murphy BA, Mariani T, Motzer RJ, Cella D: The Functional Assessment of Cancer Therapy-BRM (FACT-BRM): A new tool for the assessment of quality of life in patients treated with biologic response modifiers. Qual Life Res 2004, 13: 137–154. 10.1023/B:QURE.0000015297.91158.01PubMedView ArticleGoogle Scholar
- Motzer RJ, Murphy BA, Bacik J, Schwartz LH, Nanus DM, Mariani T, Loehrer P, Wilding G, Fairclough DL, Cella D, Mazumdar M: Phase III trial of interferon alfa-2a with or without 13-cis-retinoic acid for patients with advanced renal cell carcinoma. J Clin Oncol 2000, 18: 2972–2980.PubMedGoogle Scholar
- Yost KJ, Eton DT: Combining distribution- and anchor-based approaches to determine minimally important differences: The FACIT experience. Evaluation & the Health Professions 2005, 28: 172–191. 10.1177/0163278705275340View ArticleGoogle Scholar
- Motzer RJ, Mazumdar M, Bacik J, Berg W, Amsterdam A, Ferrara J: Survival and prognostic significance of 670 patients with advanced renal cell carcinoma. J Clin Oncol 1999, 17: 2530–2540.PubMedGoogle Scholar
- Cohen J: Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Hillsdale, NJ, Erlbaum; 1988.Google Scholar
- Center for Drug Evaluation and Research: Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Rockville, MD, US Food and Drug Administration; 2006:1–32.Google Scholar
- Eton DT, Yost KJ, Cella D: Future trends in patient-reported outcomes assessment for patients with advanced-stage lung cancer receiving targeted therapy. Clin Lung Cancer, in press.Google Scholar
- Rini BI, Sosman JA, Motzer RJ: Therapy targeted at vascular endothelial growth factor in metastatic renal cell carcinoma: Biology, clinical results and future development. BJU Int 2005, 96: 286–290. 10.1111/j.1464-410X.2005.05616.xPubMedView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.