Measurement challenges

Unfortunately, across most illness conditions TS and PS research has been consistently hampered by serious measurement problems, including; distributional skew, ceiling effects, and missing response data [39–47]. Since ceiling effects and data skew reduce the power of statistical methods to detect meaningful group differences, numerous attempts have been made to resolve these problems including; the use of very extreme anchors, the use of non-neutral midpoints, and the expansion of the number of scale response options [48–50]. Nevertheless, systematic comparisons of these approaches have been sporadic and there remains a longstanding debate over the relative merits of such methods. Results from one of the few empirically-based comparisons by Ware and Hays [51], suggest that Likert-type scales might perform slightly better than Visual Analogue Scale (VAS) methods. Advocates of VAS methods contest this assertion and refer to the ease of use, brevity and condensed layout of VAS rating scales [52].

None of these scaling solutions, however, have been shown to wholly resolve the distributional problems associated with the cross-sectional measurement of TS and PS. Yet, there remains a persistent and largely unquestioned assumption that normal distributions of satisfaction scores can be obtained if only the construct were measured correctly. As a result, there are quite a few examples in the literature where patients' satisfaction ratings are suspect of social desirability or acquiescence responses bias [49, 53]. There is a risk, however, of over generalizing an assumed respondent bias to all types of TS measures. For example, TS-M ratings may be less susceptible to social desirability bias compared to PS ratings of clinical care, as the latter is more likely to be influenced by patients' relationships with primary caregivers [54]. Moreover, if respondents tend to acquiesce and provide satisfied responses, it is more likely to occur when answering questions about less important or irrelevant aspects of care. Scales composed of large numbers of detailed and treatment-specific content typically contain a large number of items that are irrelevant to the experiences of a specific patient and thus are more susceptible to receiving a satisfied rating from respondents. In contrast, more generally worded questions are composed of items that allow respondents to interpret their meaning based on important aspects of their own experiences. Respondents are less likely to provide an acquiescent response to questions that are considered personally relevant.

An alternate mechanism may help explain the skewed distribution of TS-M ratings. Over time, clinical-selection may affect the composition of patient samples (sample drift) and result in a skewed cross-sectional distribution of satisfaction scores. It is hypothesized that such selection occurs over time as patients for whom a medication is working continue to take the medication, while those for whom it is not working, or for whom unpleasant side effects occur, seek alternative treatments. In general, one might expect sample drift to be greatest during the initiation of a new course of medication and, conversely, least when either a satisfactory medication has been found or when treatment alternatives have been exhausted. In the latter case, access to fewer treatment alternatives may be more likely among those with severe and persistent disease. Currently, it is unknown to what extent these various influences shape the observed distribution of satisfaction results in cross-sectional patient samples.

Rationale for current study

Although numerous disease-specific measures of patients' TS and TS-M have been reported in the literature [55–60] less attention has been paid to developing a more general measure of TS-M; one that would permit comparisons across medication types and patient conditions. Also, as addressed earlier, there is an unresolved controversy over the optimal method for scaling satisfaction items. Therefore, two central objectives have been identified for the current study:

Background item generation

The design of test items for the new instrument was based on a generalized conceptual framework of treatment satisfaction. The initial formation of the conceptual framework was grounded in a thorough review of the scientific literature that dealt with the core TS-M domains across a diversity of therapeutic areas. Subsequently, the draft conceptual framework was more fully elaborated using qualitative data from patient focus group interviews. Focus group participants (n = 30) were recruited to take part in one of three, two-hour sessions conducted in Los Angeles, Chicago, and Boston. Participants consisted of patients with at least one the following illness conditions: asthma, arthritis, cancer, cardiovascular disease, depression/anxiety, diabetes, infectious disease, migraine, and psoriasis. The focus group discussions were guided by a trained interviewer who, in accordance with established qualitative research procedures [61], focused on aspects of the treatment satisfaction framework, outlined in a discussion guide [62, 63].

Over the course of the three focus group sessions, the discussion guide and conceptual framework on which it was based, were evolved through integration of the patients' perspectives from each preceding group. In this way the guide was iteratively refined to reflect the participants' perspectives. Once the framework was fully elaborated, the domains of TS-M included; (1) side effects, (2) symptom relief, (3) convenience, (4) effectiveness, (5) impact on daily life, and (6) tolerability/acceptability. Fifty-five draft TS-M items were designed to measure aspects of the conceptual framework and its domains. Further details of the qualitative methods and results can be found in a sister manuscript describing the development of the TS-M conceptual framework.

Initial item reduction and scaling (patient interviews)

In-depth patient interviews were conducted in order to reduce the 55-item pool by approximately half, leaving only those items that were most relevant across respondents. The interview sample consisted of 17 patients taking medication for the same conditions represented by focus group participants. During the 45–60 minute interviews, patients rated the importance or relevance of each item to their satisfaction with their medication using a 5-point scale (where 1 was most important and 5 was not important at all). These ratings were used to select items that were most relevant across all illness groups. When items were ranked equally, the conceptual framework was used to help assure adequate representation of theoretical dimensions in the framework. The final test item pool contained 31 items.

Two scaling methods, visual analogue scaling (VAS) and Likert-type scaling, were considered for use in the final instrument. In order to compare the relative performance of the two methods, two sets of TSQM items were created that differed only in terms of the rating scale used. For both sets, TSQM items were scaled using either a 5-point or 7-point scale. Five-point scales were used for unidimensional continua (e.g. extremely to not at all), while 7-point scales were used for bipolar continua (e.g., extremely positive to extremely negative). This provided roughly equivalent rating intervals across items. Non-neutral midpoints were used for 7-point scales, resulting in a greater range of positive response options than negative options for these items. This approach has been suggested elsewhere as a way of helping to address scale resolution problems associated with the upper end of skewed distributions [48].

Psychometric testing and refinement (national panel survey)

The remaining sections of this article describe the reliability and validity characteristics of the test items and scaling methods using a large sample of patients participating in the NFO – World Group's CAP. The NFO CAP consists of over 250,000 people suffering from one or more of over 60 chronic ailments and conditions. The panel is a representative sampling of one out of every 191 households in America, prescreened for more than 50 pieces of demographic information so as to represent the demographic characteristics of the population of the citizenry of the USA (for more information see: http://www.nfow.com).

Patients were recruited for this portion of the study that had the same illness conditions as represented within the focus groups and interviews (anxiety/depression, arthritis, asthma, cancer, cardiovascular disease, diabetes, infectious disease, migraine, and psoriasis). They were also required to be at least 18 years of age, able to read English, and able to complete a questionnaire on-line. The broad sampling provided a range of treatment intents (i.e., curative, preventive and symptom management) as well as routes of medication administration (i.e., injection, oral, topical, inhalation).

Invitations were sent electronically to 10,000 NFO panel members across the United States. Participants that accessed the study site via the Internet were assessed for eligibility, equally stratified by illness condition and gender, and then randomly assigned into 1 of the 2 scale conditions (VAS or Likert-type scaling methods). Since many participants had multiple illness conditions, and were on several medications at the same time, respondents were helped to clearly identify which particular medication and illness condition were the subject of study. A total of 6,713 individuals responded (a response rate of 67.2%), from this pool individuals were sequentially offered the opportunity to participate based on the availability of participant slots in each stratum. Five hundred and eighty seven individuals passed screening and were enrolled, of these, 567 provided complete data sets, with 287 respondents in the VAS arm and 280 in the Likert-type arm.

In addition to completing the test items, respondents were asked to provide information about the length of time they had been on their medication, the method of its administration, the frequency and severity of any side effects they might have experienced, and the likelihood that they would continue to take the medication given its current level of effectiveness and side effects. They were also asked about perceptions of their current state of health, the severity of their illness, and some basic socio-demographic information (e.g., age, gender, educational level, and ethnic background).

Respondent characteristics

Construct dimensionality of the TSQM

Multi-step exploratory factor analyses (EFA) were employed to investigate the construct validity of the TSQM. Two separate EFAs were conducted, one using global TS-M items, and another using items that referred to more specific domains of medication experiences (e.g., Effectiveness, Side effects, Convenience) [64]. Such multi-step EFA procedures have been recommended by Gorsuch [65, 66] and Russell [67] as a way to evaluate the structure and dimensionality of measures that include both global and specific item content. The global TS-M items are super ordinate conceptually and psychologically, and thus may be redundant and confounding measures of the construct. As the goal is to identify the underlying construct or factor, redundant and confounding variance should be minimized. The confounding of subordinate construct dimensionality by global items shows up as unwanted covariance, manifest as cross-loading of global items across the more specific factors. As a result, separate analyses of global and specific items provide scales with greater cohesion and homogeneity than when such a process is not followed.

Scale characteristics and scaling comparisons

Possible reasons for the distributional skew of SIDEF were explored further. The removal of respondents who reported rare or very infrequent side effects from the sample resulted in an essentially normal distribution (skew = -.13, <4% of scores at ceiling value). This suggested that respondents appropriately provided high satisfaction ratings in situations where the side effects of the medication were very infrequent. Thus, the skew and ceiling effects associated with this particular scale do not seem to be simply due to problems associated with an uninterpretable respondent bias.

Medication and illness characteristics associated with treatment satisfaction

No significant differences in mean TSQM scale scores were observed by gender or education level. Significant differences in satisfaction levels were found on all TSQM scales by route of medication administration (Figure 1). As documented elsewhere, individuals using injectables reported low satisfaction with convenience of use [68]. Also, despite low ratings on SIDEF and CONV by the injectable group, the GLOBAL and EFFECT ratings were highest – presumably due to the influence of insulin-dependence in our injectable sample. Also, consistent with other research, high GLOBAL and CONV ratings were associated with orally administered medications [68–70] although satisfaction with the effectiveness of oral medications was a bit lower than with the injectables. The topicals were associated with the highest levels of satisfaction with SIDEF and CONV, but the lowest levels of on the GLOBAL and EFFECT scales – likely due to their safety and ease of use, but relative ineffectiveness at treating the condition, which in this case was psoriasis [71]. Taken together, these observations provide some evidence for the criterion-related validity of the TSQM scales.

Determinants of overall satisfaction and medication persistence

In order to explore the effects of TS-M on patients' choice to either continue or discontinue using a medication, a composite variable was derived – "Likelihood to Discontinue" medication. This variable was computed by taking the respondents' ratings of their 'likelihood to continue on the current medication given its current level of effectiveness' and subtracting it from ratings of their 'likelihood to request a change in medication due to current side effects. The four TSQM scale scores were then entered as independent variables into a stepwise multiple regression analysis. The final model contained 3 of the 4 satisfaction scales; GLOBAL, SIDEF and EFFECT (Adjusted R² = .50, F(3,563) = 186.2, p < .0001). GLOBAL accounted for more variance than did the more specific TSQM scales and was the most significant independent variable accounting for medication non-persistence (standardized Beta coefficient = -.35, p < .0001). This scale was followed by SIDEF (Beta = -.22, p < .0001) and EFFECT (Beta = -.28, p < .0001). On its own, CONV was not found to be significantly correlated to respondents' ratings of Likelihood to Discontinue medication.

Sampling considerations

The response by two-thirds of the NFO panel invitees was much greater than that usually found for broad and less targeted Internet-based health surveys, which typically range between 20–30% [72–75]. Nevertheless, our samples should not be considered representative of the general US population, and only, at best, an approximate sampling of membership within each of the illness panels. Fortunately, concerns about sampling bias are rarely raised as a serious criticism in this type of psychometric study, since the main objective is to empirically examine the content and measurement dimensions that underpin a theoretical construct. So long as the constructs of measure remain fairly consistent across the illness populations, it is unlikely that a moderate degree of sampling and selection bias alters the item-item covariance structure used to identify the dimensionality of such constructs. Such bias becomes more problematic where determination and comparison of score levels is of interest, as is the case when estimating population parameters or testing group score differences.

A more serious threat to demonstration of the construct validity of the TSQM, however, is the interdependency between illness conditions, illness severity, and medication type. Such interdependencies preclude a clear delineation of their independent effects on TSQM results. This concern became most apparent during our examination of the effects of the route of medication administration on TSQM results. It was not possible to clearly separate the effects of medication route from illness group membership, a fact that we acknowledged when describing these results.

TSQM performance

Given the heterogeneity of the sample, the reliability and construct validity characteristics of the TSQM scales were surprisingly strong. The internal consistency estimates of the scales were high given the small number of items and their broad content coverage. The patterns of item loadings within the factor analyses provide clear evidence for the orthogonal dimensionality of the three specific TSQM scales and, by inference, discrete sub-dimensionality of the TS-M construct. In turn, when predicting GLOBAL ratings, the regression weights associated with specific TSQM scales differed across illness groups, reflecting the different importance of TS-M dimensions to overall satisfaction across illness/medication types. This finding provides further evidence that GLOBAL cannot be simplified to a generalizable summation of specific scales across all illness groups since the relative weighting of specific aspects of TS-M on GLOBAL scores appears to be influenced by both medication-specific and disease-specific experience across patient groups. Despite its non-uniform derivation, the importance of GLOBAL was manifest through its correlation with patients' perception of treatment persistence (likelihood to continue/discontinue medication), which was stronger than any specific dimension of TS-M, even the effectiveness dimension.

The results of scaling comparisons (VAS versus Likert-type method) support earlier work by Ware and Hays [51] that reported better predictive performance of a Likert-type scaling compared to VAS scaling of TS items. Despite our best efforts to assure both metric and sample equivalence between the two scaling conditions, better distributional characteristics and lower measurement error was associated with the Likert approach, particularly on GLOBAL. As a result, this scaling method was associated with a greater proportion of meaningful variance across a variety of parametric analyses. Moreover, a commonly cited advantage of VAS type scales is ease of completion, yet when asked, the patients in the two scaling conditions did not differ on the reported ease of questionnaire completion. As a result, the Likert-type scaling method was selected to scale the final version of the TSQM.

Atypical cross-sectional score distributions

As expected, our scaling efforts did not effectively correct the distributional problems. Indeed, some of the most consistent findings in the PS literature are the persistent distributional skew and ceiling effects associated with this type of data. One of major causes of such skew in the current study was item relevance. This was clearly demonstrated using SIDEF items, where satisfaction ratings were consistently high when problems with side effects were rare or non-existent. Approximately 50% of the sample reported rarely experiencing side effects and, when this was taken into account, the distribution of SIDEF satisfaction scores became essentially normal. A similar pattern of results might have occurred for CONV, however, information on the frequency of inconvenient medication-related events was not collected.

In addition to item relevance, we hypothesized that a portion of the skew and ceiling effects might be due to a continuous self- and clinical-selection process, leading to sample drift as over time, individuals who are less satisfied with either the effectiveness or side effects of their medication seek alternatives. Supporting this idea, respondents' length of time on medication was positively associated with mean differences on both EFFECT and SIDEF. Those on medications for more than two months expressed higher levels of satisfaction on the two dimensions of TS-M. Moreover, the distributions of scores had greater skew towards the more satisfied end of the continuum.

Acting against such a trend may be a lack of effective treatment alternatives for those with more serious conditions. Respondents who rated themselves as either in worse health, or as more ill, were less satisfied across all TSQM scales. One might hypothesize that patients with more severe conditions may have been willing to tolerate higher side effects in order to affect a cure. However, this does not easily explain the lower EFFECT scores also reported by persons with poor health ratings. It is most likely that less satisfaction with the effectiveness of treatment is associated with treatment resistant illness conditions and/or fewer effective treatment alternatives.

Our results suggest that non-normal distribution of cross-sectional satisfaction scores should not be quickly dismissed as an artifact of systematic respondent bias, but rather understood as the result of a complex interaction between clinical-selection, the availability and effectiveness of treatment alternatives, and respondents' health status over time. Unfortunately, the cross-sectional design of our current study does not permit a meaningful characterization of the cumulative effects of such characteristics on TSQM score distributions over time, and is only suggestive of a need for longitudinal research to more fully address this phenomenon.

Treatment satisfaction and the cost of care

From a disease-management perspective, it is likely that assessment of TS-M will become increasingly important in the future; in part due to the increasing prevalence of chronic disease in our aging population and the increasing number of patients being asked to persist with long courses of pharmaceutical treatments. With the exception of certain areas of medicine where patient compliance is particularly problematic, relatively little is known about the influence of patients' satisfaction on medication adherence behavior. It is particularly pressing, given rising costs of health care, to identified and address the causes of non-adherence; since such behavior increases the use of medical resources to manage treatment failure. While the health care costs resulting from non-compliance are fairly well characterized for many conditions, such costing studies less frequently include patient preference data. Longitudinal economic research is required to explore these important causal relationships.

The pharmaceutical industry may also be an interested stakeholder. Evidence of the growing importance of patients' satisfaction can be found at most levels of our health service delivery systems. For example, the Health and Human Services' Agency for Healthcare Research and Quality and the American Hospitals' Association has recently begun to require that patient satisfaction data be published by hospitals to aid patients in their selection of hospital services, and possibly informing financial reimbursement schedules [76]. Such developments may foreshadow the role of TS-M within the pharmaceutical sector, in that TS-M outcomes directly influence the degree of market success enjoyed by new therapeutic technologies and medications. For example, TS-M assessment may play an expanded role in formulary access decisions.

Relevance to the delivery of clinical care

Patients' dissatisfaction with treatment may act as an early warning of threats to the clinical effectiveness and efficiency of medical care. Patients who perceive their medication to be ineffective, laden with side effects, or very inconvenient to use are less likely to either fill prescriptions or take their medication as prescribed. This in turn can impact the effectiveness of treatment and may result in service inefficiencies associated with treatment failure. TSQM provides a unique opportunity to compare various medications used to treat a particular illness on the primary dimensions of treatment satisfaction. Routine assessment of patients' level of TS-M provides a way for clinicians to screen individuals whose current medication experiences may increase the risk of poor medication adherence. If collected from many patients, such information could foster a deeper consideration of patients' perspectives when evaluating the merits and drawbacks of various treatment alternatives.

As partial compensation for the potential drain that individualized assessments can place on already burdened clinical staff, the dimensionality of TSQM offers a set of convenient reference points to quickly focus patient-caregiver discussion on potential problem areas, thereby facilitating a corrective engagement process. For example, a better appreciation of patients' experiences with a particular medication's side effects or inconveniences may lead physicians to optimize dosing or review administration instructions with their patients. Alternately, if patients' dissatisfaction with the effectiveness of their medication does not seem to be clinically warranted, some patient education may be in order. This later point is particularly important when the therapeutic actions of a medication are not physically or mentally discernable by most patients (e.g., preventive treatments of hyperlipidemia or hypertension). In general, clinicians who show an active interest in patients' experiences are more likely to be seen as possessing good clinical skills and a genuine concern for patients' well-being [77].