Validation study of a web-based assessment of functional recovery after radical prostatectomy
© Vickers et al; licensee BioMed Central Ltd. 2010
Received: 24 May 2010
Accepted: 5 August 2010
Published: 5 August 2010
Good clinical care of prostate cancer patients after radical prostatectomy depends on careful assessment of post-operative morbidities, yet physicians do not always judge patient symptoms accurately. Logistical problems associated with using paper questionnaire limit their use in the clinic. We have implemented a web-interface ("STAR") for patient-reported outcomes after radical prostatectomy.
We analyzed data on the first 9 months of clinical implementation to evaluate the validity of the STAR questionnaire to assess functional outcomes following radical prostatectomy. We assessed response rate, internal consistency within domains, and the association between survey responses and known predictors of sexual and urinary function, including age, time from surgery, nerve sparing status and co-morbidities.
Of 1581 men sent an invitation to complete the instrument online, 1235 responded for a response rate of 78%. Cronbach's alpha was 0.84, 0.86 and 0.97 for bowel, urinary and sexual function respectively. All known predictors of sexual and urinary function were significantly associated with survey responses in the hypothesized direction.
We have found that web-based assessment of functional recovery after radical prostatectomy is practical and feasible. The instrument demonstrated excellent psychometric properties, suggested that validity is maintained when questions are transferred from paper to electronic format and when patients give responses that they know will be seen by their doctor and added to their clinic record. As such, our system allows ready implementation of patient-reported outcomes into routine clinical practice.
Radical prostatectomy is a mainstay of treatment for early stage prostate cancer. Although associated with excellent rates of cure, the procedure leads to erectile and urinary dysfunction. Patients typically experience severe urinary incontinence and erectile dysfunction immediately after surgery, but recover gradually over the course of the first post-operative year. Nonetheless, some patients experience long-term difficulties with sexual function and urinary control. The uncertain nature of return to function is a major source of anxiety for prostate cancer patients recovering from surgery.
There are treatments available for urinary and erectile dysfunction after radical prostatectomy. Pelvic floor exercises ("Kegels") have been shown to improve return of urinary control, and procedures such as the male sling can be successful for patients with persistent incontinence. Comparably, PDE5 inhibitors such as Viagra can be used to treat post-prostatectomy erectile dysfunction with some urologists advocating daily use for the first few months after surgery as a form of "penile rehabilitation"[6, 7].
Clearly, good clinical care of patients after radical prostatectomy depends on careful assessment of post-operative morbidities. Yet there is accumulating evidence that physicians do not always judge patient symptoms accurately. This has been demonstrated in fields as diverse as chemotherapy, primary care and dermatology. Specifically with respect to radical prostatectomy, Sonn et al. compared the results of patient questionnaires with clinical documentation of their urologists in 1,366 men following curative treatment for prostate cancer, about 70% of whom underwent surgery. Physicians consistently underestimated patient dysfunction when compared to questionnaire responses from patients. For example, at long term follow-up, 42% of patients reported urinary dysfunction whereas physicians documented urinary dysfunction only in about half as many patients (22%). Similarly, whereas nearly all patients (94%) reported some level of erectile dysfunction, only 62% of patients were coded as such by their urologist.
While patient-reported outcomes are clearly preferable, considerable logistical problems are associated with their integration into routine clinical care. Paper questionnaires need to be administered to patients, checked and then the responses tallied, for example, by summing specific questions to calculate domain specific scores, with the results then entered into the clinical record. All this must be done by busy clinic staff in a timely manner such that the doctor can access the questionnaire results before the consultation.
At Memorial Sloan-Kettering Cancer Center, we have started to integrate electronic recording of patient-reported questionnaires into our clinics. These allow automatic checking of patient responses and direct porting of summary information into the clinic record. Known as the STAR system ("Symptom Tracking and Reporting"), this system was initially developed for use by patients receiving outpatient chemotherapy and was found to be feasible for both clinic-based and home-based internet reporting in diverse populations, including those with no prior computer experience, lower educational levels, and high symptom burdens[12, 13]. Use of the platform was expanded into the clinical trial setting with ongoing national multi-center evaluations in the National Cancer institute-sponsored cooperative groups, and a system in development for the NCI under contract based on the STAR model, called the PRO-CTCAE.
STAR is a flexible interface which allows administration of questions to patients via the web, either from clinic-based computers or from home. Reminder emails can be triggered to patients who do not complete scheduled questionnaires. The platform sends automated notifications to study staff if a patient misses a scheduled questionnaire, or reports a concerning symptom which merits clinical evaluation. Data collected by STAR are exported to the electronic health record and clinical trials database for clinicians and investigators to view results.
We adapted the STAR system to record data on functional recovery after radical prostatectomy. Here we report our initial experience of obtaining patient-reported outcomes through a web interface. In particular, we were interested in patient response rates and in evaluating the psychometric validity of our instrument.
Development of the instrument
We have previously validated a questionnaire for post-treatment recovery in prostate cancer patients. This questionnaire was deemed too long and complex for web implementation as it included 68 questions, some of which had up to 18 response options. Accordingly, we choose a subset of questions for our web-instrument. We chose six questions from the erectile function domain that constitute the International Index of Erectile Function-6 (IIEF-6), a well-validated and widely used instrument. We then analyzed data from questionnaire responses of radical prostatectomy patients to identify 6 of 16 questions from the urinary domain that demonstrated good internal consistency (Cronbach's alpha of 0.82) and, when summed, had excellent correlation with the full urinary domain score (0.90). Bowel symptoms are rare after radical prostatectomy, and we found that just two questions about bother from bowel symptoms that had good correlation (0.91) with the total of all questions from the bowel domain score. We added a simple 0 - 10 scale of global health related quality of life to give a total of 15 questions.
Having chosen the items for our instrument, we added two interactive features. First, the IIEF6 asks several questions about erectile function that depend on sexual activity. For example, a question about erection frequency and another about erection hardness both have "no sexual activity" as a response. Accordingly, we designed the questionnaire so that if a patient responded "no sexual activity" to the first question about erections, further questions were skipped and scored as zero. Second, assessment of recovery after radical prostatectomy is subject to what is known as interval censoring. A patient given a questionnaire at three and six months and who, for example, reports use of incontinence pads at the first but not the second questionnaire, is typically recorded as having regained continence at six months. In truth, the patient likely stopped using pads at some point in the preceding three month period, and not on the exact day that the questionnaire was administered. Thus, when a patient first responds that he is not using pads, an additional question is implemented concerning when the patient first stopped needing pads (within the past month; 1 - 2 months; 2 - 3 months; more than 3 months). Similarly, when a patient first reports erections sufficient for penetration, he is asked about when this was first achieved. Additional file 1 shows the instrument and scoring.
STAR hardware includes two servers that reside in the MSKCC New Jersey Data Center. One is the web server and is located in the "DMZ" where it is accessible to the Internet. The other server is the database server and is completely protected by the MSKCC firewall.
The web server uses Microsoft Internet Information Services (IIS) and Secure Sockets Layer (SSL) encryption to protect the data through the Internet. The web application is developed using Microsoft .Net technologies and uses the highest security and privacy features that exist to date.
The database server hosts Microsoft SQL server where the data is stored and backed up daily. All information in STAR is saved in the database where it is completely protected by the firewall.
STAR database shares information with CAISIS, an open source, web-based cancer data management system that is used for clinical management of patients at MSKCC. Patient information is entered in CAISIS and "pulled" by STAR. STAR in turn registers the patients automatically and emails them in due time to take the survey, based on surgery and appointment dates provided by CAISIS. Once the survey is taken, STAR will "push" the data back to CAISIS where it can be seen by clinicians and downloaded for research purposes.
Implementation of the web-based tool in clinical practice
Patients who provide an email address to the hospital are sent an email inviting them to complete a questionnaire at 3, 6, 9, 12, 18, 24, 36, and 48 months after surgery. However, patients with a clinic appointment up to six weeks before any of these time points are sent reminder emails two weeks prior to their appointment so that their responses are available to their urologist in good time. The text of the email reminder is given in additional file 2: the key point is that patients are told that their answers will go directly into their clinic record so that their doctors can see how they are doing.
As an attempt to improve clinical practice, the web-based tool was not subject to an IRB protocol. Data for this evaluation was obtained under an IRB waiver for routinely collected data to be used for research purposes.
Our initial aim was to investigate the validity of the STAR questionnaire to assess functional outcomes following radical prostatectomy. For assessment of internal consistency within domains, Cronbach's alpha coefficient was calculated for all domains of the questionnaire, except overall quality of life, which is a single-item measure.
To assess instrument validity, we evaluated the association between survey responses and known predictors of sexual and urinary function. To account for men who completed more than one survey, we used multivariable generalized estimating equations. Pre-specified predictors included time from surgery, age at surgery, nerve sparing status (none, unilateral or bilateral), comorbidities (0, 1 or >1). Separate models were built for the outcomes of urinary and sexual function. For these analyses, sexual and urinary function domains were rescaled to a 0 - 100 range, in order to allow direct comparison. To account for a possible non-linear relationship between recovery of function and either time from surgery or age, we included non-linear terms (restricted cubic splines with knots at the tertiles). We hypothesized that valid measures of erectile and urinary function would show decreasing scores with age and comorbidity, and increasing scores with time from surgery, and nerve sparing surgery.
We also compared the surgeon and patient-reported assessments of function. We restricted this analysis to occasions when the patient- and physician-reported assessment where within six weeks of one another. We used the first eligible assessment time for patients with more than one time point for which both patient- and physician reported data were available. Physician-reported erectile and urinary function are on a five point scale. Sexual function was defined as physician-assessed score of 1 or 2 (normal, full erections or full, but diminished erections satisfactory for sexual activity); continence as classified as physician-assessment of 1 (no pads). All analyses were conducted using Stata 11.0 (Stata Corp., College Station, Texas).
Summary of patient characteristics.
All men who completed a questionnaire
N = 1235
Men who were invited, but did not complete questionnaire
N = 346
Age at surgery (years)
62 (57, 67)
60 (56, 65)
5.00 (3.48, 6.95)
5.30 (3.98, 7.14)
Pathologic Gleason Score
Lymph node involvement
Positive Surgical Margins
Seminal Vesicle Invasion
Nerve sparing status
Number of comorbidities
Nearly all surveys were completed in their entirety (n = 1761; 93%); 78 (4%) questionnaires included one missed question and only 48 (3%) included more than one missed question.
Summary of response rate and average scores for each domain.
Number of items
Median (IQR) score (original scaling)
Range of scores (original scaling)
8 (3, 23)
18 (15, 20)
8 (7, 8)
Overall quality of life
8 (7, 9)
We hypothesized that, if our instrument was valid, there should be a positive correlation between function and quality of life, with urinary function having a higher correlation than sexual function. This is indeed what we found: the correlations between total sexual and urinary function and overall quality of life were 0.27 and 0.47, respectively.
Predictors for urinary and sexual function scores.
Coefficient (95% CI)
Coefficient (95% CI)
Time from surgery
Nerve sparing status
9.0 (3.8, 14.2)
3.0 (-4.8, 10.7)
9.1 (5.0, 13.1)
17.2 (11.2, 23.2)
2.0 (-1.4, 5.3)
-3.3 (-8.3, 1.7)
-1.4 (-5.4, 2.7)
-6.5 (-12.5, -0.4)
-6.9 (-13.0, -0.8)
-13.1 (-22.2, -3.9)
Our third hypothesis was that patient-reported sexual and urinary function scores would predict physician-reported assessments of potency and continence. Among those with patient and physician reported scores that were measured within six weeks of each other (n = 365 and 469 for sexual and urinary function, respectively), we found that patient-reported sexual and urinary function scores predicted physician-assessed function with high discrimination (sexual function AUC: 0.86 and urinary function AUC: 0.87; p < 0.001 for both).
We have found that web-based assessment of functional recovery after radical prostatectomy is practical and feasible. The tool clearly has good patient acceptability, as evidenced by the very high response rate of 78%. Indeed, we believe that this is an underestimate as it includes as non-responders patients who gave non-functioning email addresses. For example, we became aware that some of the addresses given by patients were those of relatives.
The patient-reported outcomes instrument has excellent psychometric properties. There was very high internal consistency within domains, and excellent validity: domain scores not only predicted overall health related quality of life, but do so exactly as hypothesized, with urinary function showing a stronger relationship than sexual function; there was a tight association between domain scores and physician assessments; correlation between items within domains was much higher than correlation between items from different domains; domain scores discriminated between groups known to differ, such as decreases in erectile function with age and nerve resection and increases in urinary function with time from surgery. These findings might not be surprising given that our instrument consists of a subset of questions from a previously validated questionnaire. Nonetheless, we have shown that validity is maintained when the questions are transferred from paper to electronic format. Moreover, our findings show that validity is not strongly affected when patients give responses that they know will be seen by their doctor and added to their clinic record, in contrast to providing confidential answers, as is typical in research studies.
Our system was costly to establish, requiring considerable programming time, but incurs near zero marginal costs for new patients. A member of the clinic staff has been assigned to ensure that the salutation field is appropriate (e.g. "Dear Mr. Jones" rather than "Dear Mr. Jones Jr.") and to be the first point of call for patient enquiries. Some of these enquiries are technical and so are forwarded to a programmer. But the total number of enquiries is low, approximately 1 - 2 per month, and few enquires require more than a couple of minutes to resolve.
An obvious limitation of our system is that it is available only for individuals with email access. Accordingly, we are currently developing a version that can be used by patients in clinic. We will purchase inexpensive notebook computers that we will then "lock down" so that only our web-interface is available. Patients will be logged in by clinic staff, who will be available to advise patients if they have difficulties. Our previous experience of implementing STAR in chemotherapy clinics at MSKCC is that patients find it no more challenging to use than an ATM machine. Currently, an automated telephone system (interactive voice response system) is being developed as an optional add-on to STAR to administer items by telephone.
We have not discussed assessment of baseline function in this paper as our focus has been the evaluation of post-treatment questionnaires. Assessment of function before surgery is naturally a key part of good clinical care, and essential for interpreting a patient's post-operative function. Baseline data are currently collected using a paper version of the web-based questionnaire. Paper forms are used as we are yet to establish systems for email contact before a patient's first session. We will transition to the web-based system this year when computers are made available in the clinic, as discussed above.
Our focus on the psychometric evaluation of the web-based questionnaires has also left unaddressed the issue of how the system is used by surgeons, and the degree to which they find it useful. We plan to evaluate systematically how surgeons use the system to guide patient care and follow up.
We have developed a web-based assessment of functional recovery after radical prostatectomy. We have obtained very high response rates, and the instrument demonstrates excellent psychometric validity. As such, our system allows ready implementation of patient-reported outcomes into routine clinical practice.
Supported in part by funds from David H. Koch provided through the Prostate Cancer Foundation, the Sidney Kimmel Center for Prostate and Urologic Cancers, a R21-CA133869 grant to Andrew Vickers and a P50-CA92629 SPORE grant from the National Cancer Institute to Dr. P. T. Scardino.
- Stephenson AJ, Kattan MW, Eastham JA, Bianco FJ Jr, Yossepowitch O, Vickers AJ, Klein EA, Wood DP, Scardino PT: Prostate cancer-specific mortality after radical prostatectomy for patients treated in the prostate-specific antigen era. J Clin Oncol 2009,27(26):4300–4305. 10.1200/JCO.2008.18.2501PubMed CentralPubMedView ArticleGoogle Scholar
- Sanda MG, Dunn RL, Michalski J, Sandler HM, Northouse L, Hembroff L, Lin X, Greenfield TK, Litwin MS, Saigal CS, et al.: Quality of life and satisfaction with outcome among prostate-cancer survivors. N Engl J Med 2008,358(12):1250–1261. 10.1056/NEJMoa074311PubMedView ArticleGoogle Scholar
- Penson DF, McLerran D, Feng Z, Li L, Albertsen PC, Gilliland FD, Hamilton A, Hoffman RM, Stephenson RA, Potosky AL, et al.: 5-year urinary and sexual outcomes after radical prostatectomy: results from the Prostate Cancer Outcomes Study. J Urol 2008,179(5 Suppl):S40–44. 10.1016/j.juro.2008.03.136PubMedView ArticleGoogle Scholar
- Filocamo MT, Li Marzi V, Del Popolo G, Cecconi F, Marzocco M, Tosto A, Nicita G: Effectiveness of early pelvic floor rehabilitation treatment for post-prostatectomy incontinence. Eur Urol 2005,48(5):734–738. 10.1016/j.eururo.2005.06.004PubMedView ArticleGoogle Scholar
- Sandhu JS: Treatment options for male stress urinary incontinence. Nat Rev Urol 2010,7(4):222–228. 10.1038/nrurol.2010.26PubMedView ArticleGoogle Scholar
- Bannowsky A, Schulze H, van der Horst C, Hautmann S, Junemann KP: Recovery of erectile function after nerve-sparing radical prostatectomy: improvement with nightly low-dose sildenafil. BJU Int 2008,101(10):1279–1283. 10.1111/j.1464-410X.2008.07515.xPubMedView ArticleGoogle Scholar
- Mulhall JP, Parker M, Waters BW, Flanigan R: The timing of penile rehabilitation after bilateral nerve-sparing radical prostatectomy affects the recovery of erectile function. BJU Int 2010,105(1):37–41. 10.1111/j.1464-410X.2009.08775.xPubMedView ArticleGoogle Scholar
- Basch E: The missing voice of patients in drug-safety reporting. N Engl J Med 2010,362(10):865–869. 10.1056/NEJMp0911494PubMed CentralPubMedView ArticleGoogle Scholar
- Weingart SN, Gandhi TK, Seger AC, Seger DL, Borus J, Burdick E, Leape LL, Bates DW: Patient-reported medication symptoms in primary care. Arch Intern Med 2005,165(2):234–240. 10.1001/archinte.165.2.234PubMedView ArticleGoogle Scholar
- Consolaro A, Vitale R, Pistorio A, Lattanzi B, Ruperto N, Malattia C, Filocamo G, Viola S, Martini A, Ravelli A: Physicians' and parents' ratings of inactive disease are frequently discordant in juvenile idiopathic arthritis. J Rheumatol 2007,34(8):1773–1776.PubMedGoogle Scholar
- Sonn GA, Sadetsky N, Presti JC, Litwin MS: Differing perceptions of quality of life in patients with prostate cancer and their doctors. J Urol 2009,182(5):2296–2302. 10.1016/j.juro.2009.07.027PubMedView ArticleGoogle Scholar
- Basch E, Artz D, Dulko D, Scher K, Sabbatini P, Hensley M, Mitra N, Speakman J, McCabe M, Schrag D: Patient online self-reporting of toxicity symptoms during chemotherapy. J Clin Oncol 2005,23(15):3552–3561. 10.1200/JCO.2005.04.275PubMedView ArticleGoogle Scholar
- Basch E, Iasonos A, Barz A, Culkin A, Kris MG, Artz D, Fearn P, Speakman J, Farquhar R, Scher HI, et al.: Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. J Clin Oncol 2007,25(34):5374–5380. 10.1200/JCO.2007.11.2243PubMedView ArticleGoogle Scholar
- Sobin LH, Wittekind C: UICC TNM classification of malignant tumors. New York: John Wiley & Sons; 1997.Google Scholar
- Basch E, Artz D, Iasonos A, Speakman J, Shannon K, Lin K, Pun C, Yong H, Fearn P, Barz A, et al.: Evaluation of an online platform for cancer patient self-reporting of chemotherapy toxicities. J Am Med Inform Assoc 2007,14(3):264–268. 10.1197/jamia.M2177PubMed CentralPubMedView ArticleGoogle Scholar
- Befort CA, Zelefsky MJ, Scardino PT, Borrayo E, Giesler RB, Kattan MW: A measure of health-related quality of life among patients with localized prostate cancer: results from ongoing scale development. Clin Prostate Cancer 2005,4(2):100–108.PubMedView ArticleGoogle Scholar
- Rosen RC, Cappelleri JC, Gendrano N: The International Index of Erectile Function (IIEF): a state-of-the-science review. Int J Impot Res 2002,14(4):226–244. 10.1038/sj.ijir.3900857PubMedView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.