Skip to main content

Methodological and interpretive concerns about Beemster et al.’s article ‘The interpretation of change score of the pain disability index after vocational rehabilitation is baseline dependent’: a letter to the editor

Abstract

This is a critique of Beemster et al.’s article ‘The interpretation of change score of the pain disability index after vocational rehabilitation is baseline dependent’ (2018). The methodological issues in question include the choices of anchor to determine the minimal important change, and the intraclass correlation coefficient on which the calculation of the standard error of measurement was based. We believe these undermine the authors’ interpretation.

Main text

In their article ‘The interpretation of change score of the pain disability index after vocational rehabilitation is baseline dependent’, Beemster et al. [1] aimed to assess the interpretation of the change score of the Pain Disability Index (PDI) in patients with chronic musculoskeletal pain (CMP) attending vocational rehabilitation. The authors reported that, on the individual level, the smallest detectable change (SDC) was smaller than the minimal important change (MIC), indicating that the PDI can detect a real change in pain-related disability [1]. However, we question this conclusion, given two significant methodological concerns.

First, in our opinion, the choice of the anchor to determine the MIC is unhelpful. A patient-based global perceived effect (GPE) item, i.e. global rating scale (GRS), is generally acceptable to determine the MIC of a patient-reported outcome using an anchor-based method. The GPE consists of a single Likert-scale question by which patients rate, at follow-up, the perceived change in their health status since baseline. A crucial prerequisite is that the GPE assesses the same construct as does the questionnaire under study [2]. While the PDI measures pain-related disability, the GPE item used in this particular study asks patients “to indicate how much their pain has changed since baseline” [1]. The constructs ‘pain’ and ‘pain-related disability’ are very different, especially in patients undergoing chronic pain rehabilitation. Crucially, since pain complaints are often intractable, these interventions specifically aim to decrease pain-related disability. This also applies to the vocational rehabilitation intervention that the study sample underwent [3]. As a result, pain relief is far less common as an intervention success than decreased pain-related disability is. Accordingly, our hypothesis is that the currently used GPE item ‘pain’ might have resulted in less participants who classified themselves as improved (i.e. responders) than if a GPE item ‘pain-related disability’ would have been used. Additionally, it can be assumed that participants who indicated improvements in pain scores perceived a larger reduction of disability than those who did not experience pain relief. Therefore, a more valid GPE (i.e. measuring pain-related disability) would probably have resulted in a smaller change score on the PDI in the group responders and, in turn, have led to a lower MIC.

The second problem with the study is the way the standard error of measurement (SEM) has been calculated. The authors use a formula of debatable validity, SEM = SD √(1-ICC) [1] (where SD is the standard deviation and ICC is the intraclass correlation coefficient). Since they did not perform their own test-retest analysis, their ICC was derived from a study of Soer et al. [4], who extensively validated the PDI in three groups of patients with musculoskeletal pain. Beemster et al. state that this study’s sample was similar to their own [1]. However, comparison of the descriptive statistics does not bear this out. The ICC in the earlier study was calculated in patients with either acute, subacute, or chronic back pain, or chronic widespread pain of the musculoskeletal system [4]. This mixed population (n = 845) had a mean baseline PDI score of 37.6 with a SD of 14.6 (personal communication with Soer R.). In contrast, Beemster et al. found a baseline PDI score of 34.7 with a SD of 11.7 in their population of patients with CMP attending vocational rehabilitation [1]. These SDs show the former population to be more heterogeneous than the latter. However, the ICC is highly dependent on heterogeneity [2] and consequently, as de Vet et al. [2] specifically stress, applying a previously reported ICC to a more homogeneous population will lead to SEMs which are too small and therefore misleading. Since the SDCs at the levels of the individual and the group were obtained from these dubious SEMs, we consider these values invalid as well.

In summary, we think that the MIC may be overestimated, whereas the SDC is likely underestimated. Based on the above arguments, we believe that the conclusion that the SDC is smaller than the MIC is questionable. It is quite possible for the SDC and the MIC to be the same or even to be reversed in order of size, as is the case for many outcome measures. If the MIC of the study is in fact smaller than the SDC, change values between the MIC and SDC might be considered clinically important, but not statistically significant, and thus indistinguishable from measurement error. We suggest that a study similar to this one be carried out in the light of the methodological issues raised in this letter, in order to provide higher quality evidence. Without this, the conclusions of the study of Beemster et al. must be approached with caution. Specifically, we would discourage the use of the cutoff scores for clinically relevant changes provided. Aside from the implications for researchers performing responder analyses, health insurance companies, and policy makers, this is particularly relevant to health care professionals seeking to interpret individual change scores in clinical practice.

Availability of data and materials

Not applicable.

Abbreviations

CMP:

Chronic musculoskeletal pain

GPE:

Global perceived effect

GRS:

Global rating scale

ICC:

Intraclass correlation coefficient

MIC:

Minimal important change

PDI:

Pain Disability Index

SD:

Standard deviation

SDC:

Smallest detectable change

SEM:

Standard error of measurement

References

  1. 1.

    Beemster T, van Bennekom C, van Velzen J, Reneman M, Frings-Dresen M. The interpretation of change score of the pain disability index after vocational rehabilitation is baseline dependent. Health Qual Life Outcomes. 2018;16(1):182.

    CAS  Article  Google Scholar 

  2. 2.

    de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011.

    Google Scholar 

  3. 3.

    Beemster TT, van Velzen JM, van Bennekom CA, Frings-Dresen MH, Reneman MF. Cost-effectiveness of 40-hour versus 100-hour vocational rehabilitation on work participation for workers on sick leave due to subacute or chronic musculoskeletal pain: study protocol for a randomized controlled trial. Trials. 2015;16:317.

    Article  Google Scholar 

  4. 4.

    Soer R, Köke AJ, Vroomen PC, Stegeman P, Smeets RJ, Coppes MH, et al. Extensive validation of the pain disability index in 3 groups of patients with musculoskeletal pain. Spine. 2013;38(9):E562–8.

    Article  Google Scholar 

Download references

Acknowledgements

We thank Dr. Remko Soer for providing additional information on his study population.

Funding

Not applicable.

Author information

Affiliations

Authors

Contributions

LB and RS discussed the original article and wrote the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Laura Wilma Maria Elisabeth Beckers.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Beckers, L.W.M.E., Smeets, R.J.E.M. Methodological and interpretive concerns about Beemster et al.’s article ‘The interpretation of change score of the pain disability index after vocational rehabilitation is baseline dependent’: a letter to the editor. Health Qual Life Outcomes 18, 301 (2020). https://doi.org/10.1186/s12955-020-01555-1

Download citation

Keywords

  • Chronic musculoskeletal pain
  • Pain Disability Index
  • Minimal important change
  • Standard error of measurement
  • Smallest detectable change