Validity of instruments to measure physical activity may be questionable due to a lack of conceptual frameworks: a systematic review

Patient-reported outcome (PRO) instruments have always been an important tool in epidemiological and clinical research. Recently, interest in these instruments has increased with their use as outcome measures in randomized trials of pharmacological and non-pharmacological interventions. Regulatory agencies, namely the United States Food and Drug Administration (FDA) and the European Medicines Agency (EMA), have developed guidance documents concerning the appropriate development, validation, and use of PRO instruments in clinical trials [1, 2]. Particular emphasis has been placed on their validity, that is, the ability of a PRO to measure the concept that it is intended to measure. To this end, the use of conceptual frameworks is advised [3–6]. The conceptual framework explicitly defines the concepts measured by the instrument in a diagram that represents the relationships between the main concept (e.g., health-related quality of life), the domains (e.g., symptoms), the sub-domains (e.g., dyspnea), and the items measured as well as the scores obtained from a PRO instrument [2, 7]. An absent or inadequate conceptual framework is likely to lead to inadequate development and validation of a PRO [3–6], which in turn, may create confusion about what is actually being measured [7].

The proportion of PROs for which a conceptual framework formed the basis for the development and validation process is currently unknown. For regulatory agencies and stakeholders such as patients and physicians, it is only possible to understand the meaning of the effects of health care interventions on PROs if the underlying concepts to be measured are clearly outlined. Because PROs represent a very broad group of outcomes, it would be overly ambitious to assess all types of PROs that have been developed. Therefore, we focused on PROs that capture aspects of physical activity as the main concept and chronic respiratory diseases as the main study subjects. Physical activity is a key concept in public health because reduced physical activity is a well-known risk factor for many chronic diseases and disorders [8], and sedentary lifestyles are common around the world [9]. Despite the importance of physical activity, it is challenging to define what physical activity actually means and how to capture the important aspects of physical activity. Thus, a conceptual framework of physical activity is particularly important for instruments that intend to measure this parameter. We focused on chronic respiratory diseases for two reasons: they are a leading cause of morbidity and mortality worldwide [9], and respiratory health is not included in most physical activity recommendations, despite the epidemiological and clinical evidence that regular physical activity may reduce the incidence and improve the prognosis of chronic respiratory diseases [10–12]. The current gap between research and public health needs may be partly due to the absence of a universally accepted definition of physical activity in studies of patients with chronic respiratory (and similar) diseases.

The aims of this study were (i) to identify available conceptual frameworks of physical activity in chronic respiratory patients or similar populations, and (ii) to determine whether the development and validation of currently used PRO instruments to measure physical activity in these populations were based on a conceptual framework of physical activity.

This study was part of the European Union-funded PROactive project (http://www.proactivecopd.com), which aims to develop, validate, and apply patient-reported outcome instruments to capture the dimensions of physical activity in daily life relevant to patients with chronic obstructive pulmonary disease (COPD). The PROactive consortium is multidisciplinary and includes academic partners, patient organizations, and pharmaceutical companies.

We utilized standard systematic review methodology following the handbooks of the Centre for Reviews and Dissemination [13] and the Cochrane Collaboration [14]. The manuscript follows the PRISMA [15] statement for reporting of systematic reviews and meta-analyses. All methods were specified in advance, documented in a protocol, and approved by the PROactive consortium. This manuscript includes data from two systematic reviews performed as part of the PROactive project. First, a systematic literature search, detailed below, was conducted to identify conceptual frameworks of physical activity. Second, we performed a secondary analysis from another systematic review of the PROactive consortium [unpublished observation] that identified existing PRO instruments for measuring physical activity.

This review identified only 2 conceptual frameworks for physical activity in COPD patients, whereas no conceptual frameworks seem to exist for patients with chronic heart disease, or elderly people. Furthermore, none of the available instruments for measuring dimensions of physical activity or related constructs in these populations was based on a conceptual framework for physical activity. These results may reflect the incomplete understanding of what physical activity means in chronic respiratory disease patients and similar populations (i.e., chronic heart disease patients or the elderly).

A potential limitation of this review is that some conceptual frameworks for physical activity may have been missed, despite a rigorous database search followed by a comprehensive hand-search and communication with expert. The indexing of this type of publication is not standardized, thus creating challenges in identifying relevant publications. Another potential limitation is that the FDA guidance for the PRO measures was published in 2006, whereas most of the PRO instruments included in our reviews were developed prior to that date. However, as early as 1985, previous guidelines for developing questionnaires included the requirement of a conceptual framework [3–6], even if this was labeled differently. The strengths of our review are the inclusion of chronic respiratory disease patients, patients with chronic cardiac diseases and elderly people, and the use of the same population criteria and concept definitions within two reviews.

A challenge of this review was understanding the definitions and concepts presented in the articles included in the full-text assessment because of the inconsistent use of terminology and the overlap between domains, sub-domains, and items within each conceptual framework. To resolve this issue, the authors strictly applied the definition of a conceptual framework (a concept to be measured by domains, sub-domains, and, ultimately, items). It was apparent that most articles considered for full-text assessment did not focus on physical activity as the concept to be measured or did not present a conceptual framework. Similarly, identifying a conceptual framework from the manuscripts developing PRO instruments was complicated because most articles provided the main concept, but the identification of their conceptual framework was much more difficult. A general recommendation from our review is that manuscripts should maintain consistency in the labeling of main concepts and domains, and in their definitions. Additionally, a discrepancy may be observed between the number of conceptual frameworks identified in the first (n = 2) and second reviews (n = 22). It is important to emphasize that none of the conceptual frameworks identified in the second review had physical activity as the main concept; consequently, they cannot be considered to represent conceptual frameworks of physical activity.

The requirement that PRO instrument development should be based on a conceptual framework has been long established and acknowledged in several guides and standards, such as the Medical Outcomes Trust [4, 5], Health Measurements Scales [6] and the American Psychological Association (APA) guidelines [3]. In particular, the Standards for Educational and Psychological Testing of the APA state that "the construct of interest for a particular test should be embedded in a conceptual framework, no matter how imperfect that framework may be" [3]. Our finding of a lack of conceptual frameworks for physical activity is in agreement with the lack of conceptual frameworks identified by O'Brien et al. for the comprehensive evaluation and treatment of people living with HIV [23]. Unfortunately, no other papers similar to this paper have been identified, which supports the view that there is little awareness and knowledge of this topic. Our second finding, that less than half of the PROs for physical activity or related constructs were based on a conceptual framework, is consistent with the finding of Pollard et al. that none of the PRO instruments for health outcome measures in osteoarthritis were based on a conceptual framework [24]. Of additional interest, we noted that only 3 questionnaires out of 22 refined their domains after the analysis, despite the need to develop PRO instruments through an iterative process, per FDA recommendations [25]. All things considered, our study suggests that there is limited knowledge of the complete process of developing and validating a PRO instrument.

It is debatable whether physical activity is a construct that requires a conceptual framework for its definition given the current trend in activity monitoring. The authors' opinion, based on the current state-of-the-art, is that physical activity is a multifaceted construct that goes well beyond the amount or frequency of physical activity. The concept of physical activity could also include, for example, the ability to perform activities of daily living and symptoms such as dyspnea or pain associated with physical activity, which are not captured by accelerometers or pedometers. Irrespective of the number of domains included in the construct of physical activity, it is critical that the use of questionnaires to measure it should be based on a previous conceptual framework.

The lack of conceptual frameworks for physical activity by individuals with chronic diseases such as COPD also has public health implications. In the meeting report entitled Implementation of the World Health Organization (WHO) Strategy for Prevention and Control of Chronic Respiratory Diseases, increasing physical activity levels was included in the recommendations for the development of policies in the area of chronic respiratory diseases [25]. The lack of a conceptual framework seems to preclude the success of potential interventions because there is no clear concept based upon which to intervene. One could question whether this lack of definition of physical activity is responsible, at least in part, for the lack of effectiveness of most physical activity interventions in chronic-disease patients at the population level [26]. A second potentially negative effect of the lack of conceptual frameworks is the difficulty of measuring the effectiveness of the interventions because instruments may not exactly measure what they claim to measure. Furthermore, using a PRO that is not based on a conceptual framework may lead to measurement error (information bias), which is a challenge to detect intervention effects. Therefore, we argue that (new) conceptual frameworks for physical activity for the elderly and for populations with chronic respiratory diseases and chronic heart diseases must be derived. These frameworks should precede the development of new PRO instruments for physical activity in these populations.

We identified only 2 conceptual frameworks for physical activity in COPD. We found that none of the currently available PROs that aim to measure physical activity in chronic respiratory disease patients or similar populations (chronic heart disease patients or the elderly) are based on a conceptual framework with physical activity as the main concept. Our findings contrast sharply with the FDA's guidance on the development and validation of PROs, raising the question of how the large body of evidence from trials that use these instruments should be evaluated by health care providers, guideline developers, and regulatory bodies.