Development and content validity of the Lupus Foundation of America rapid evaluation of activity in lupus (LFA-REAL™): a patient-reported outcome measure for lupus disease activity

Lupus is a heterogeneous autoimmune disease characterized by periods of flares and remission, with a high impact on quality of life. Patients with increased lupus disease activity are known to have poor quality of life [1]. Disease activity is traditionally assessed by physicians on the basis of non-systematic history taking, physical examinations, and interpretation of laboratory test results, which may not be available for weeks. Validated, clinician-derived outcome measures that are commonly used in clinical trials include the British Isles Lupus Activity Group (BILAG), SLE Disease Activity Index (SLEDAI), Physician’s Global Assessment (PGA), and composite indices based on these instruments, none of which are simple or easy to assess in a busy clinic [2, 3]. Patient-reported outcomes (PRO) are usually included in trial results, but rarely as a key endpoint and are not usually used in clinical practice in the care of SLE patients, despite the fact that their pivotal importance in good medical care has been increasingly recognized [4, 5].

The difficulty in using PRO as measures of disease activity in lupus clinical trials stems from the discordance between the physician and patient estimates of disease activity [6–11]. Potential reasons behind this discordance have been studies by some and point to the relative importance paid to different concepts by physicians vs. patients in evaluating disease activity [12, 13]. Although this may be due to physician misjudgment of the degree of change experienced by patients, the discrepancy between the clinician and patient reports might also suggest that patients incorporate drug side effects or other co-morbidities in with lupus activity while physicians are trained to consider those as separate entities. While knowing whether to attribute symptoms to active lupus disease is often challenging for patients and their clinicians, instructing patients to think about this difference when scoring the PRO should improve the likelihood that disease activity is being scored, as opposed to permanent damage or side effects of medications. Neglect of patient input could affect the accuracy of trial results, the quality of patient care decisions, as well as the patients’ adherence to prescription medication, compliance with follow-up appointments, and trust in their care. The parallel assessment of the most common symptoms using the PRO/ClinRO might help explain the frequent discordance between patient and physician evaluations.

The recent publication by Holloway reviewed in detail al the PROs in the lupus arena. There are more than 60 PRO measures that have been used in studies of lupus, with only a few that are SLE-specific, and fewer still that have provided guidance to try and distinguish active lupus symptoms from treatment side effects or the collateral damage of a chronic disease [14]. No existing PRO measure integrates correlative assessments between patients and their doctors to facilitate working together to evaluate disease activity and making fully informed treatment decisions. The Systemic Lupus Activity Questionnaire (SLAQ) is a validated disease activity PRO, derived from the SLAM, that has been validated and used in epidemiological research [15]. Symptom checklists have been proposed by others [16]. Additionally, the SIMPLE Index that includes patient reported items, medication use and laboratory tests was developed as a surrogate for disease activity [17]. The Lupus Foundation of America—Rapid Evaluation of Activity in Lupus (LFA-REAL) system includes both clinician-reported outcomes and patient-reported ratings developed in parallel that capture complementary information. Studying and validating these instruments together may create a more complete picture of lupus disease activity, while helping to distinguish between discordance based on differing opinions and discordance based on addressing completely different topics. The LFA-REAL PRO is different from the SLAQ and symptom checklists by focusing on continuous quantification of current symptoms, with specific instructions aimed at distinguishing active disease from damage and depression.

The main advantages of a disease activity-driven PRO that correlates with a ClinRO are: 1. By including a focus on overlapping assessment approaches, this could be used in practice as a way to better communicate about symptom progress. In a patient centered care model this might improve the relationship between providers and patients by improving communication, trust, and adherence to treatment plans. 2. Could be self-administered and used to alert the health care providers of a change in health status that would prompt an earlier in-person evaluation of a patient. 3. Could be used in clinical trials as outcome measures. The inclusion in the PRO of both patient-based and clinician-based assessment approaches, differences between the assessments might be better understood as either a different focus or a different assessment with the same focus. 4. By improving the correlation between clinician and patient-reported outcomes, both could be used by practitioners to justify need for increased therapy and use of biologics by payers. 5. Could be used in clinical epidemiological research to provide a better proxy for disease activity measures that cannot be obtained without in-person visits. 6. Could allow for increased uptake in the community based on the possibility to decrease the duration of assessments while increasing meaningful communications. The providers and the patients would more effectively and precisely interact, resulting in shorter visits. 7. Could be scaled in prospective studies to provide a clinically meaningful target for therapy for a treat-to-target approach in everyday care.

The FDA Clinical Outcomes Assessment PRO guidance recommends using an iterative process of qualitative research involving concept elicitation to identify the concepts that are most important to patients and cognitive debriefing interviews (to ensure that the draft PRO is easy to understand and use by patients), in order to assure content validity of the measure in the target population [18–22]. Steps to ascertain content validity of the PRO were important to address before the two instruments could be studied side by side. The purpose of this paper is to describe the methods and results of the qualitative work performed to complete this process.

Early work that included focus groups, patient and stakeholder input informed the development of a PRO to match the CLIN-RO. The health domains of the PRO were tested prior to the work described in this paper [23]. Further concept identification included six focus groups (N = 35 patients) conducted by the LFA staff in different geographical locations (New York, Atlanta, Oklahoma, Boston, Washington DC, Los Angles) and was analyzed in a full report [24].

This qualitative study was performed through interviews with 25 adult participants with lupus from the Washington D.C. area. The interviews were conducted by Evidera, a healthcare research firm with expertise in outcomes research, instrument development, and validation. The first phase included 10 patients in 45–60 min concept elicitation interviews conducted by telephone. The second phase included 15 patients with in-person or over the phone cognitive debriefing interviews.

Prior to the interviews, IRB approval was obtained. Written informed consent and a medical release of information form were obtained from the participants. Interviews were performed by experienced qualitative interviewers. A semi-structured interview guide for each phase was used to ensure consistency across interviews. In addition to the questions asked during the interviews, participants were invited to complete the LFA-REAL PRO questionnaire, a health-related quality of life questionnaire, and a socio-demographic form. All interviews were audio-recorded. Participants’ treating physicians were contacted to complete a brief clinical form to confirm lupus diagnosis.

Measures

LFA-rapid evaluation of activity in lupus PRO (LFA-REAL PRO)

With a recall period of four weeks, the LFA-REAL PRO asks patients to evaluate the severity and progress of symptoms associated with their current lupus activity along a series of anchored visual analog scales (VAS). Items evaluated separately include rash, symptoms of arthritis (i.e., joint pain, joint swelling, and joint stiffness), a summary arthritis score that integrates those three features, muscle pains or aches, body symptoms (e.g., shortness of breath), fatigue, fever, and hair loss. These items evolved and were modified in Phase II based on patient concept elicitation results, and the final instrument incorporated feedback from Phase II. A pre-defined component of the instrument was to break arthritis and mucocutaneous symptoms down into discrete components that might allow patients to describe similar features to those that physicians consider in the LFA REAL™ CLINRO. For example, a score for overall arthritis is not elicited until after the patient scores each component of joint pain, joint stiffness, and joint swelling, in effect, teaching patients through this process how physicians evaluate arthritis. The LFA-REAL was not shown to the patients until after the concept elicitation and cognitive debriefing interviews were completed, both of which confirmed the validity of the descriptors and provided edits to the wording used for the instructions and labels. The current version of the PRO is presented in Fig. 1.

SF-36 (version 1.0 standard; Rand scoring) questionnaire

This 36-item questionnaire reports current experiences or experiences over the past four weeks using Likert-type response scales to assess patient health status and quality of life. Scores range from 0 to 100, with higher scores indicating higher levels of functioning. Scores were generated for each of the eight domains (i.e., physical functioning, role limitations due to physical problems, bodily pain, general health perceptions, vitality, social functioning, and role limitations due to emotional problems and mental health) and two summary domains (i.e., physical component summary [PCS] and mental component summary [MCS]).

The SF-36 was chosen to describe the population based on its extensive use in SLE studies [25]. We acknowledge the shortcomings of the SF-36 in SLE and its potential for floor effects and in the validation study we plan to include several other PROs [26]. SF-36 scoring was based on the original orthogonal rotation [27, 28].

“Questions about You” form (a socio-demographic and clinical questionnaire)

Participants completed the brief “Questions About You” form, a socio-demographic and clinical questionnaire, after completing the semi-structured interviews. The “Questions About You” form asked participants to provide socio-demographic information including, but not limited to, age, race, employment, education, and basic clinical information. This information was used for descriptive purposes to characterize the population sample.

Clinical form

For eligible participants, treating physicians were asked to complete a clinical form to confirm and date the diagnosis of lupus.

This paper describes the methods and results of qualitative research with lupus patients used to ensure content validity of the LFA REAL™ PRO, and supports our hypothesis that a patient-centered approach can be used to evaluate the content validity of the LFA-REAL PRO. The final draft of this instrument is shown in Fig. 1 as a series of simple, landmarked VAS global assessments of symptoms that are confirmed as being commonly experienced and important to SLE patients, in terms that they understand and approve of. Instructions are brief and guide the patient to a consideration of what represents active lupus disease as opposed to longstanding symptoms that are more likely due to damage. A global score and organ-based or symptom-specific evaluations can be obtained from the patient’s perspective that is comparable to the physician’s ratings of the most common lupus features on the partnered CLINRO [29, 30].

Routine use of PROs will enhance clinical practice. Unfortunately, in lupus, PROs in use have been problematic to execute and interpret. In trials they often provide disparate outcomes. We hope that a measurement that complements the physician’s judgment will be able to increase the use of PROs in routine clinical care.

The poor correlation between PROs and physician assessments of disease activity or disease severity tell us that PROs are measuring information not captured in physician assessments. PROs measure the impact of disease on patients’ daily lives. Disease activity as assessed by physicians is based on symptoms, signs and laboratory assessments that are proximal to the impact on daily lives. Current PROs used in SLE emphasize these impacts rather than symptoms and disease activity. The Lupus Impact Tracker (LIT) [31, 32] derived from the LupusPRO [33, 34] was shown to be responsive to physician assessed changes in disease activity (SLEDAI, PGA, and SRI), but also to patient assessed changes in healthy (HRQoL), and disease activity (SLAQ). LIT has also been shown to be responsive to physician assessed treat to target health outcomes being used recently in SLE [35]. As expected, achieving lupus low disease activity and remission correlates with improvements in QoL. Although, PROs – including the LIT- have not shown any association with damage measures [36] it is possible that lupus PROs are contaminated by damage when reports of pain refer to pain from joints affected by avascular necrosis or osteoarthritis. The LFA-REAL PRO was designed to mainly evaluate disease activity related physical health and not impacts on daily life or SLE related damage. While the LFA-REAL does not cover the full spectrum of disease activity in the listed items, the “is there anything else to report” can include any additional issues deemed important by the patient and can inform further changes to the instrument. Mental health, impacts and damage are clearly important and should continue to be measured and addressed in lupus care and research.

A ClinRO disease activity instrument, the Lupus Activity Index (LAI), was developed as a physician measure that incorporates, along with a number of other components, eight visual analog scales which are restricted to 4 symptoms (fatigue, rash, joint involvement, serositis) and 4 signs (neurologic, renal, pulmonary, and hematologic involvement) [31, 32]. The LFA-REAL was designed to include both the physician and patient perspectives of disease activity and allow for a simple yet comprehensive evaluation of any and all active features, individually and in composite. Unlike the LAI, which uses the landmarks as flags to suggest approximate grades of disease severity the LFA-REAL uses these landmarks as defined cut-offs (mild disease is < 1) [29, 30]. This promotes more consistency in scoring between observers and gives the same line length to adjust gradations of mild, moderate or severe manifestations. The ClinRO pre-specifies the most commonly affected organ/systems/symptoms of lupus but leaves room to expand the list for individual patients. The PRO described here has been refined with patient input to include the symptoms that are important to patients but overlaps with the ClinRO in specified ways to elicit an assessment that might correlate to what the physicians evaluate.

We readily acknowledge several limitations, the SLE diagnosis was not confirmed for all participants, the PRO may not cover all aspects of SLE, data on fibromyalgia was not collected. During the development of the draft PRO, which was tested in the current study, volunteer patients had listed disease features that were most important to them, and it became clear that, unlike rash, the concept of arthritis is not universally understood. However, patients do have a granular understanding of each feature that a physician incorporates into an assessment of inflammatory arthritis, such as joint pain, swelling, and stiffness. For this reason, the patients are instructed to rate these components separately and then, only after that, to factor them together in a global arthritis rating. We hypothesize that this may produce an understanding of arthritis more equivalent to that of a trained clinician than previous PROs have been able to achieve. If so, it might be easier to determine whether discrepancies between physician and patients’ perception of improvement are due to different interpretations of the question or different assessments of the same domain. For example, if swelling is markedly improved, but pain is not, this will be apparent in the arthritis section of the PRO where it might not be as apparent to physicians when completing the CLINRO.

To complete the PRO in a consecutive series of visits, patients are instructed to begin a new scoring session with review of what they scored at the last visit and to consider whether they would rate each scale as same, improving, or worsening by moving their mark (or not) along the line accordingly. This approach has been shown to improve the consistency of scoring PGAs in lupus studies and is mandated for use in the SELENA SLEDAI PGA that clinicians routinely score [37, 38]. By providing an equivalent scoring methodology for clinicians and patients for the most common lupus manifestations, the LFA-REAL may offer additional insights into the physician-patient discordance that has so often been observed in trials [39].

In the next steps, assessment of the quantitative performance of LFA-REAL PRO scores will include tests of reliability, validity, and responsiveness, including correlations of the REAL PRO scores with those of other PRO measures used in lupus clinical trials (e.g. SF36, Lupus QOL, PtGA).

SLE is an unpredictable condition with a variable course and prognosis ranging from mild, manageable disease to a severe, life threatening disorder. Some patients may experience predominantly skin manifestations, while others can have musculoskeletal involvement, renal disease, neurological involvement, pericarditis, pleurisy or solid organ inflammation, or a combination of these [40]. A chronic, waxing and waning, complicated disease such as lupus requires a tool that is not only responsive and dimensional but also resonant to the concerns of the lupus population without the burden of many layers of complexity. By directing the patient’s focus to active disease symptoms, the LFA-REAL PRO may help make the assessment of treatments both in trials and clinical practice more efficient, accurate, and useful. Furthermore, the PRO may provide important information about the direction of progress for the whole patient, as well as individual features of disease. Given the known negative correlation of disease activity with quality of life [41], this may add to the understanding of which features have the greatest impact on a person’s experience.