Skip to main content

Validity and reliability of the (adjusted) Impact on Participation and Autonomy questionnaire for social-support populations

Abstract

Background

Decentralisation of social support and budget cuts spurred interest in outcome-oriented payment systems in the Netherlands. Hence, measurement of relevant outcomes, such as self-reliance and participation, is needed. The Impact on Participation and Autonomy (IPA) questionnaire for rehabilitation clients was adapted for social support, called IPA-MO, and its validity and reliability were tested among social support clients in eight municipalities in 2014 and 2016.

Methods

The total research sample comprised of 4120 persons. Homogeneous subgroups were distinguished based on 1) disability and 2) age. Exploratory factor analysis (PCF) was used to identify domains for IPA-MO. Model fit was assessed with Confirmatory Factor Analysis (CFA) using structural equation modelling.

Results

PCF revealed that the IPA-MO model consists of all five original IPA domains: Autonomy indoors, Family role, Autonomy outdoors, Social life and relations, and Work and education. As a result of new items added, a new, sixth domain was developed: Financial autonomy. Due to high non-response on Work & education, construct validity was first tested for a five-domain IPA-MO model. The composition of the IPA-MO domains showed slight differences: the item on ‘mobility indoor’ shifted from Autonomy indoors (IPA) to Family Role in IPA-MO. The item on reciprocity shifted from Social Life and relations (IPA) to Autonomy outdoors (IPA-MO). Internal reliability was confirmed for all domains (Cronbach’s alpha >.80). CFA showed acceptable construct validity of the five-domain IPA-MO model for the social support population (CFI .936, TLI .925, SRMR .051), all age groups and most disability-based groups. Construct validity including Work & education was tested for 234 participants. Then, PCF revealed six domains and the model fit was acceptable (CFI .915, TLI .903, SRMR .067).

Conclusions

IPA-MO is a valid and reliable instrument to assess outcomes of social support. Further research on the domain Financial autonomy is needed. Social-support clients are numerous and dispersed and, in spite of the best intentions, hard to involve widely in policy processes. A valid outcome measure offers municipalities possibilities to gain insight in social costs and benefits of new policies. Outcome measurement also allows to contract bundled-services of providers, thereby changing the incentives for providers from increasing production to much needed innovation. Taking the perceptions of autonomy and participation of social-support clients as the acid test for the effectiveness of social support policies, may prove a serious game-changer in politics.

Introduction

Under the Dutch Social Support Act (“Wet maatschappelijke ondersteuning – Wmo”), public social-support services have progressively been transferred to local governments. Since 2006, municipalities manage a wide range of non-medical facilities to support adult inhabitants living with a disability in the field of domestic tasks, mobility and social life. In 2015, facilities for individual and group guidance also became part of Wmo [1]. Simultaneously, the Participation Act and Youth Act enhanced the municipalities’ mandate in the field of employment and youth care [2, 3].

The decentralisation of responsibilities in the ‘social domain’ is related to national policies that aim to reform the welfare state, whereby people living with disabilities should participate in ‘normal life’ and be as self-reliant as possible. Government policies should foster an enabling environment, and their support should strengthen people’s ability to participate. This is a major change from the past, when government policy was oriented towards taking over the care for these people, based on the assumption that inactivity is a logical consequence of impairments [4]. This change in national policies is in line with the paradigm change introduced by the WHO in 2001 with the ratification of the International Classification of Functioning, Disability and Health (ICF) [5]. The new emphasis on social functioning of people living with a disability is expected to increase their quality of life and contain the rise in public care costs [1].

Along with the decentralisation of social support, budget cuts were imposed as municipalities were expected to organise services in a more efficient way. This resulted in a renewed interest in integrated service delivery and outcome-based payment (instead of the current production-related reimbursement systems) [6]. In a study commissioned by the Dutch Ministry of Health, the possibilities for outcome-driven social-support systems were explored [7]. Based on Brickley et al. [8] three key elements for outcome-driven systems were identified: adequate organisational design, well-aligned incentives in payment models and reliable assessment of outcomes. The Impact on Participation and Autonomy (IPA) questionnaire was identified as a promising instrument to assess relevant outcomes [7]. Originally IPA was developed in the Netherlands in order to identify interventions with rehabilitation clients from their own perspective [9]. Rehabilitation science and practice have a long-standing tradition in enhancing the participation in society of people with disabling conditions, which is also the newly adopted focus in social support. The body of knowledge and experience developed in rehabilitation science may therefore be useful in developing new social-support strategies oriented towards self-reliance and participation.

Specific reasons for selecting the IPA instrument in this study were fourfold:

  1. 1)

    IPA offers a broad overarching scope that matches target groups and responsibilities of municipalities under the Social Support and Participation Acts: it is a generic instrument (designed for adults with any disabling condition) and covers all relevant life domains (domestic tasks, mobility, social life and relationships, self-care, income, work, education and leisure);

  2. 2)

    IPA is a validated questionnaire that assesses the self-perceived degree of participation of individuals. A distinctive quality is the inclusion of autonomy - the extent to which an individual has control over the way he lives – as inseparably linked to participation [9]. Autonomy is closely linked to the self-reliance that the new social-support policies in the Netherlands seek to reinforce [10];

  3. 3)

    IPA measures participation in terms of ‘performance’, described as ‘what an individual does in his or her current environment’. The current environment is understood to include assistive devices, personal assistance and public facilities used by the individual [5]. In other words, performance reflects how people function in everyday life with the available support. As such, it may serve as an indicator of the effectiveness of social support policies;

  4. 4)

    IPA is a self-reported questionnaire. Subjective measures are increasingly seen as preferable to assess ‘needs met’ for qualitative goals such as quality of life [11] and participation [12]. Whiteneck [13] even argues that participation, by its very nature, can only be evaluated by self-report. The Social Support Act (Wmo 2015) itself also stresses the importance of clients’ perception in quality assessment of social support [10].

The objective of this study is to assess the reliability and validity of the IPA instrument to measure participation of social-support clients on a population level.

The remainder of the article is set up as follows. In the Methods section first, the original IPA questionnaire and its validation for rehabilitation clients are presented. Second, the adaptations made in the questionnaire for use among social-support clients are discussed. The third section covers a description of the data collection and the factor analysis used. In the Results section, the analyses are presented. The article ends with conclusions and a discussion of our findings.

Methods

Between August 2014 and October 2016, eight municipalities wanted baseline information before the Social Support 2015 and Participation Act came into force. Using the IPA questionnaire surveys were implemented among large cohorts of social support clients with physical, mental and intellectual disabilities as well as persons who are unable to earn a living and depend on income support. All participants live in the community (not in institutional homes).

The Impact on Participation and Autonomy instrument: the original questionnaire

The Impact on Participation and Autonomy questionnaire assesses two different aspects of participation: perceived participation and perceived problem-experience [14]. Perceived problem-experience is scored independently and the results are used to establish goals in individual rehabilitation programmes that reflect personal preferences. In this article the focus is on perceived participation.

Perceived participation is assessed by 32 items, grouped together in 9 sections: mobility (4; 1a-1d), self-care (5; 6a-6e), domestic tasks and role (6, 2a-2f), income (1; 3d), leisure (1; 4), social contacts and relations (7; 5a-5f,5 h), helping others (1; 5 g), work (5; 7a-7e), education (1; 8) and a final item on overall autonomy and participation as perceived by the participant (1; 9a). The items are phrased in a way that emphasises control over tasks and activities (decisional autonomy) rather than whether they can be implemented with or without support (executional autonomy). As an illustration, an item on personal care is (6a or 6b): “my chances to decide when I get washed and dressed are …” Items are scored on a five-point Likert scale, ranging from very good, good, fair, poor to very poor.

Cardol [14] found that 26 of the 32 perceived-participation items load onto four domains called ‘participation domains’: Autonomy indoors (AI), Family role (FR), Autonomy outdoors (AO), and Social life and relations (SOC). She assumed a fifth participation domain, Work and education (WORK) but could not confirm it since few participants in her study had employment. The five latent domains contribute to the overarching concept of participation. Cardol et al. [9] validated the IPA questionnaire extensively. Psychometric properties such as internal reliability and test-retest reliability proved to be good on a domain level, though some items were psychometrically weak. Responsiveness to change, one of the aspects Cardol addressed, was good for three domains (WORK, AO, FR) while moderate to no responsiveness was found for two domains (AI, SOC). Convergent and divergent validity were tested with instruments such as London Handicap Scale, Sickness Impact Profile and Short Form-36 and were generally confirmed [14].

The IPA questionnaire has since been validated and adopted in many countries both in and outside Europe [15,16,17,18,19,20,21,22]. Most studies focused on rehabilitation clients with well-defined, specific physical impairments. Construct validity was tested with various methods: exploratory factor analysis, the Rasch methodology, Principal Component Analysis and Confirmatory Factor Analysis. Only the UK study [15] validated the Work and education domain. Most studies confirm the model of Cardol, with the exception of an Iranian study that found participation domains were clustered into two dimensions: performance-based and social-based participation [20]. Wilkie et al. [23] conclude that the instrument has good face validity and its measurement of participation is comprehensive and relevant according to patients. He recommends further testing of construct validity and responsiveness since study populations have not been very large and Confirmatory Factor Analysis has hardly been used in validity studies.

Adapting IPA for use in social-support populations

Feasibility of the IPA questionnaire for social-support clients was tested in a pilot among some 500 participants in 2013. This resulted in the following adaptations to the questionnaire:

  1. a)

    the section on self-care was positioned more towards the end of the questionnaire, in order to avoid participants feeling addressed as ‘patients’;

  2. b)

    three new items were added in the section on income (3a-3c) at the request of municipalities, who are responsible for debt prevention and relief. The new items were derived from a validated instrument for assessment of financial capacities of individuals developed by Mesis [24];

  3. c)

    the item on intimate relationships (5f) was rephrased into ‘my chances to give and receive love and affection’, thus avoiding the impression that the item refers to sexual relationships only and broadening the scope to all affective bonds;

  4. d)

    the items on perceived problem-experience were left out;

  5. e)

    an extensive inventory of support available to participants (informal care, assistive devices, public facilities, privately acquired services and personal assistance) was included.

The adapted IPA questionnaire contains 35 items on perceived participation and is used in the data collection among social-support populations.

Data and the process of validation

This section starts with defining the social-support population by disability type. Next, the data collection, the assessment of clients by disability type and a description of the data are presented. In the last part we describe the process of validating the IPA questionnaire and the methods used.

Defining the social-support population

The survey population comprised of people who receive individual or collective guidance (the ‘new’ client groups for municipalities under the Wmo 2015), as well as persons with mild physical disabilities (‘traditional’ Wmo clients). Also, people who have received income support for a long period (> 1 year) were included based on the assumption that their participation might be hampered due to physical or mental problems [25].

Wmo-clients were divided into four groups: people with mild physical disabilities – mainly elderly – used ‘light’ municipal support such as assistive devices, special transport facilities or domestic help. Participants with severe physical disabilities suffered from diminished physical or mental vitality as a result of severe motoric handicaps, dementia, chronic and progressive illness (e.g. rheumatism, ALS) or acquired brain injury. They received more intensive support such as individual and/or collective guidance, in order to enable them to live in the community. The ‘mild intellectual disabilities’ group included people who were socially vulnerable as a result of restricted intellectual capacities. Participants with mental-health problems dealt with addictions, psychotic disorders, anxiety attacks or behavioural problems such as autism or ADHD [26]. The latter two groups received individual guidance and guided group work so as to support their daily functioning. Participants recruited among the Participation Act clients form a fifth group.

Data collection

Between August 2014 and October 2016, surveys with the adapted IPA questionnaire were implemented among large cohorts of social support clients in eight municipalities that wanted baseline information before the Social Support Act 2015 and the Participation Act came into force. Three of them implemented a follow-up survey after 2 years. Municipalities belong to population categories 20,000–50,000 (6), 100,000–150,000 (1) and 150,000–200,000 (1) inhabitants. Four municipalities are mainly urban and four have a rural character.

Participants from categories without readily-available registration (guidance clients in 2014/15, clients with mild physical disabilities in all years) were recruited through care providers. For income-support clients and guidance clients in the follow up surveys, participants were recruited by random samples. Municipalities approached all income-support clients who had received this support for 1 year or longer. For guidance clients systematic samples were drawn from municipal registrations in 2016. All participants received a questionnaire (hard copy) accompanied by a letter of invitation signed by the alderman of their municipality. The majority of questionnaires were self-administered. People with (mild) intellectual disabilities got support from family members or care providers, who received instructions stressing that client perspective should prevail. Though interference of the helpers’ view is a risk of this procedure, we considered it worthwhile to try and capture the perspective of this group too.

All data from the baseline and follow-up surveys (n = 4660) were combined in a single database. Records with 4 or more sections with invalid responseFootnote 1 (n = 75) as well as those of respondents under the age of 18 (n = 9) were removed. Of the remaining 4576 records, income-support clients of one baseline survey (n = 341) were excluded since this group was not limited to long-term income-support. Wmo-respondents that did not report use of formal support (n = 110) or whose type of disability could not be assessed (n = 5) were also removed.

Assessing disability type

In 2014/2015, information on the type of disability of participants was deduced from the specialisation of their provider organisation. Participants in the follow-up surveys in 2016 were scored on disability based on available data (support provided, provider organisation, age, personal remarks). Each respondent was included in one disability category. However, it should be noted that these categories are not completely mutually exclusive. People living with severe physical disabilities may also have mental problems (such as depression), income-support clients sometimes have intellectual disabilities, people living with ADHD or autistic disorders are sometimes treated as having intellectual disabilities and on other occasions as having mental problems.

In Table 1 the distribution of the research sample by regional characteristics and by disability type or income-support is shown.

Table 1 Origin of the respondents by regional characteristics and by disability type or income-support

Given the size and variety in geographical origin of the samples, we assume that the subgroups in our study population are representative for the respective disability or age category. Our total study population, however, is not representative for either the national level nor any of the local communities where the surveys were held. Compared to the national level, elderly Wmo-clients and the group on income support are underrepresented (Table 2). At the municipal level, composition of social-support populations varies substantially due to local factors such as labour market, age distribution and presence of care institutions [27].

Table 2 National social-support clients (2015) and research population [38, 39]

Creating subgroups: by disability type and by age

The study population is highly heterogeneous, as it covers a variety of disabilities. Moreover, it covers a wide age spectrum (18–103 yrs) whereas Cardol [14] restricted participation in her sample to the age of 18–75. We chose to include (very) old people since they form an important segment of the population that receive assistance under the Social Support ActFootnote 2 Besides, the validity and relevance of the IPA questionnaire for elderly people (up to 99 yrs) is confirmed by Ottenval Hammar [28].

In order to analyse more homogeneous groups, two approaches were taken. The first was to distinguish participants by the type of disability, as described above. A second approach distinguished participants by age. The life cycle theory suggests that people face different needs and challenges in subsequent stages of their lives [29]. As a consequence, intensity and domains of participation vary in different phases of life [30]. Additional pragmatic advantages are objectivity and mutual exclusiveness of age categories, and the fact that no information on the type of disability is needed. We distinguished five age-groups: young adults (18–35), 2 groups for grown adults (36–50 and 51–66) and 2 groups for late adults (67–80 and 81+).

People with mild physical disabilities are the most numerous group; in terms of age, the group aged 51–66 years is largest. Mental problems and intellectual disability prevail in younger age groups whereas in older groups, physical disabilities are predominant (Table 3).

Table 3 Cross tabulation disabilities and age groups in research population

Some characteristics of the data are given in Table 4. Elderly people often use a range of state and private services to facilitate their daily functioning. Many of them also get informal help from friends and family on a regular basis. For the younger groups – many of them with cognitive or mental impairments -, guidance is the main form of state support. Informal help is most common among the 18–35 year-olds and decreases among older adults (36–66 years). Labour participation for participants in the working age is low: 15–23% have a salaried job, 30–40% of participants have unpaid activities, either in guided workshops or voluntary work. Almost half the participants (49%) under the age of 67 have no working activities at all.

Table 4 Descriptive statistics of the social-support IPA-MO database

The validation process

In order to validate the adapted IPA questionnaire we proceeded as follows.

First, we tested domain reliability and validity of Cardol’s model [14] with the original IPA items.

  1. 1)

    validation of the original IPA for 4 domains: Autonomy indoors, Family role, Autonomy outdoors and Social life and relations (internal reliability, factor structure and loadings and goodness of fit) Validation is based on the entire research sample;

  2. 2)

    validation of the original IPA for 5 domains, including Work and education (internal reliability, factor structure and loadings, goodness of fit). This analysis is based on the (small) group of participants that responded to all items on Work and education (n = 234);

Next, the adjusted IPA questionnaire with the new items on income (3a-c) was tested. We expected the new items to form a new domain together with the original item on income (3d). We excluded the final item “my possibilities to live the life I want” (item 9) from the model as we consider it to be distinct in nature: as a concluding item, it is meant and phrased as an overall appreciation rather than a predictor of perceived participation. Reliability of the IPA domains is analysed and Principal Component Factoring (PCF) is used to explore the new model structure. Results are tested on goodness of fit by applying CFA using structural equation modelling (Sem).

  1. 3)

    validation of the adapted IPA model including the new items on income and excluding the final item: exploratory analysis of factor structure and loadings, internal reliability of adapted/new domains, goodness of fit. This analysis is done for the whole research population, excluding the items on work and education;

  2. 4)

    validation of the adapted IPA model including the Work and education domain, the new items on income and excluding the final item: exploratory analysis of factor structure and loadings, internal reliability, goodness of fit. This analysis was done for the same participants as in 2).

Lastly, based on the resulting version of the IPA instrument, factors are constructed for its domains: for the whole research sample, the disability groups as well as for the five age categories.

Construct validity was analysed by Principal Component Factoring (PCF) and by Comparative Factor Analysis fitting a structural equation model (CFA by Sem). Goodness of fit was assessed with the usual statistics such as the Comparative Fit Index (CFI), the Tucker-Lewis Index (TLI) and the standardised root mean squared residual (SRMR). In line with Hu and Bentler [31], SRMR< .08 in combination with CFI or TLI > = .95 is used as a cut-off criteria for a good model fit, SRMR <.08 in combination with CFI/TLI > = .90 for acceptable fit.

Stata 15 was used for all analyses.

Results

In this section we present the results of our analysis as follows. First we validate the original IPA with 4 domains (excluding Work and education) and 5 domains (including Work and education) based on respectively 26 and 32 items of the original IPA questionnaire. For the 4-domain model the whole research sample is used. For the 5-domain model the analysis is based on the (small) group of participants that responded to all items on Work and education (N = 234).

Next we validate the adapted IPA model including the new items on financial issues and excluding the final item (see Appendix). Using explorative factor analysis, as expected a new domain emerged - Financial autonomy. Construct validity of a model with and without the Work and Education domain was tested using the whole research sample. Having investigated and shown the validity of the adapted IPA model this section concludes with a short description of the predicted domain scores for the whole research sample, and the subgroups based on disability and on age.

Reliability and validity of the original IPA model for social-support population

4 domain IPA model: autonomy indoors, family role, autonomy outdoors, and social life and relations

Explorative factor analysis using PCF was used to test validity of the 4-domain IPA model. Following Cardol, we used orthogonal rotation. In Table 5 the results are presented. Results by and large sustained the IPA model. Variance explained is 66%, comparable to that found (67%) by Cardol [14]. Misfits occurred with both items on mobility indoors of the AI domain and the item on financial independence in the FR domain. The latter was found psychometrically weak in Cardol’s study as well [14]. The remaining 23 items showed the highest rotated loadings (ranging from .44 to .85.) on the expected domain. Five of them had high loadings (> = .40) on other domains as well.

Table 5 Rotated factor loadings original IPA model for social-support clients. Principal component factoring, Varimax rotation n = 3301

The four IPA domains showed good internal reliability, with Cronbach’s alpha between .85 and .91. Item-test correlations ranged from .51 to .87. Item-rest correlations was low (<.50) for item 3d “spending money as wished” in FR. Average inter-item covariance was high (> 50) in AI and very high (>.8) in FR and AO (Table 5). This was an indication that items were very similar to each other and possibly redundant [32].

CFA showed poor fit indices for the 4-domain IPA model (CFI .791, TLI .768, SRMR .091). CFA for individual domains revealed poor indices for AI and SOC. High error covariances between pairs of items were found in all domains: 1a-1b and 6a-6b in AI, 2c-2d in FR, 1c-1d and 4 and 9 in AO, 5a-5b, 5b-5c and 5d-5e in SOC. In five pairs of items on an identical subject and sometimes even phrased using the same words,Footnote 3 one of the items was eliminated (1a, 2c, 5b, 5e, 6b). The items 1c-1d (visits and trips) and 4 and 9 (leisure time and overall quality of life) were considered to cover distinct concepts and therefore maintained. Instead, we opted for adjusting the error covariance between item 1c-1d. These modifications resulted in good fit for all domains (Table 6; first panel).

Table 6 Goodness of fit, including the comparison of domains based on the original (26-items), and adapted (21-items) IPA questionnaire

The modified domains were tested on internal reliability. Cronbach’s alpha is good (> .80) for all domains, though slightly lower than in the original domain. Item-rest correlation remained low for the financial item (3d, FR), average inter-item covariance was still high for FR (.80) (Table 6, second panel). As mean scores of the short-version and original domains did not differ much and were highly correlated (.98–1.00; Table 6, third panel), we conclude that the short version is a reliable and conceptually similar alternative for the original IPA. The overall model fit for IPA-short version improved but was still below levels of acceptability (26-items: CFI = .79, TLI = .77, SRMR = .09;21-items: CFI = .89, TLI = .88, SRMR = −.07).

Including the 5th domain, Work and education, internal reliability was confirmed by Cronbach’s alpha (.81; n = 234). Item-test correlations ranged from .57 to .83. Sibley [15] found Cronbach’s alpha equal to .90 and item-total correlations .52–.77. Item-rest correlations were low (< 0.50) for two items (7c and 8).

CFA for the 5th domain, Work and education, gave close to acceptable fit indices: CFI .929, TLI .882 and SRMR .049. A high error covariance was found between items 7a “chances to get a job” and 7f “chances to find another job or employer” (MI 24.62, EPC.33). Therefore, item 7a was excluded from the model. The 5-item version of the Work and education domain showed lower but acceptable domain reliability (Cronbach alpha 0.77) and excellent model fit (CFI .998, TLI .996 and SRMR .024).

Factor analysis (PCF) with the complete IPA (adapted version, 26 items) revealed a 5-factor model in which all work-related items form a separate domain (factor loadings .46–.88). Contact with colleagues loaded high on SOC as well. Education loaded on AO. However, CFA showed a poor model fit (CFI .875, TLI .859, SRMR .073) (the results are available upon request from the first author).

Concluding

Validation of the original IPA domains for social-support clients showed a need to delete 6 items. The resulting 26-item IPA revealed a more or less similar classification of the items into the domains but lacks sufficient model fit. In the next section we proceed by adapting the questionnaire even further.

Towards a participation model for the social-support population: IPA-MO

5 domain IPA-MO model: Autonomy indoors, family role, autonomy outdoors, social life & relations, and financial autonomy

For the construction of an adequate model for social-support clients, called IPA-MO, we proceeded with the short version IPA, including the three new items on financial issues and excluding the final item for reasons previously explained (see also Appendix). Principal component factoring with oblique rotation was used to determine the factor structure. For the total research population, a four-factor structure was found in which the IPA domains (AI, FR, SOC and AO) are visible with slight modifications: 1b (mobility indoors) moves from AI to FR, 5g (supporting other people) from SOC to AO. The financial items form a new domain that we call Financial Autonomy (FIN). FR and AO load on the same factor. The results are presented in Table 7.

Table 7 Rotated factor loadings IPA-MO model for social-support clients (principal component factoring, oblique rotation, n = 3301)

All domains showed good reliability with Cronbach’s alpha’s ranging from .80 and .93. High inter-item covariances are observed in domains AO, FR and FIN. In the new domain on Financial autonomy, item 3a has a low correlation (.49) with the other items in this domain. Excluding this item gives a slightly higher alpha but very high inter-item covariance (.97) as well.

CFA showed best model fit for a 5-domain model (AI, FR, AO, FIN, SOC) leaving out item 3a from the FIN domain (CFI .920, TLI .907, SRMR .053). High covariance errors between items still hampered model fit. Adding a path between the measurement errors of two pairs of items (1b-1c, 2d-2e) improves model fit to acceptable levels (CFI .936, TLI .925, SRMR .051). The path diagram of the model is represented in Fig. 1.

Fig. 1
figure1

The IPA-MO model – perceived participation of social-support clients

6 domain IPA-MO model: adding work and education

Including the Work and education domain in the IPA-MO model, factor analysis (PCF) reveals a six-factor model in which each domain forms a separate factor. Item 7c ‘Contact with colleagues’ best fits in the SOC domain, leaving the Work and education domain with four items (7b, 7d, 7e, 8). Domain reliability is still acceptable (Cronbach’s alpha .78, item-test correlation .80–.90, item-rest correlation .37–.67, average inter-item covariance .65). Item 5 g ‘supporting others’ moved from AO to SOC, as in the original IPA. Domain composition and factor loadings are shown in Table 8. Model fit was on the verge of acceptability (CFI .906, TLI .892, SRMR .072). Including an additional path to correct for the highest error covariance (3b-3c) improved goodness of fit to acceptable values: CFI .915, TLI .903, SRMR .067.

Table 8 Domain composition and rotated factor loadings (PCF, Promax) for IPA-MO including work (n = 237)

Model fit for subgroups

Using CFA, the 5-domain IPA-MO model was tested on goodness of fit for each one of the disability-based and age groups. Acceptable fit was confirmed for all of them. Differences between groups are small, only the group with severe physical disabilities clearly has lower fit indices than the others (Table 9).

Table 9 Model fit of IPA-MO model (5 domains)

Exploratory factor analysis (PCF) revealed differences between groups in factor structure. For most groups, a 4-factor model was found with domains AO and FR located on the same factor. The items 5 h ‘frequency of social contacts’ and 5 g ‘supporting others’ often showed highest loadings on ‘SOC’. However, adapting the model structure accordingly did not improve model fit.

For the 81+ and the ‘mild physical disabilities’ group, all domains loaded on separate factors in a 5-factor model. Items 5 h and 5 g had significant loadings on AO only.

Table 10 summarises weighted scores for the IPA-MO participation domains for the total research population as well as individual client groups. All groups show highest scores on their autonomy at home (AI). The average score (3.88) indicates that people living with a disability generally manage well at home with the support they get. Lowest average scores are reported on the domains of Work and education (2.88), Autonomy outdoors (3.04) and Financial autonomy (3.06), indicating that interaction with society is more complicated. The average score for the possibility to live as one wants is just above ‘fair’ (3.12). The instrument reveals differences between subgroups. People with severe physical disabilities suffer most deficiencies in their autonomy and participation. Autonomy indoors, Autonomy outdoors and Family role diminish with age, while scores for Social life and relations, as well as living the way one wants, decline till the age group of 51–66 years and then go up again.

Table 10 Results of the IPA-MO questionnaire for the research population (n = 4120)

Discussion

The present study examined the reliability and validity of the IPA instrument adapted for social-support clients, called IPA-MO. We used data from large cohorts of social support populations (n = 4120) from various parts of the Netherlands. The research sample comprised the full range of client types that municipalities have to support: elderly people with mild and severe physical disabilities, people with mild cognitive and mental conditions, and inhabitants dependent on income support. In order to create more homogeneous groups, both impairment-based and age-based groups were distinguished and analysed in this study.

We found that IPA-MO is a reliable and valid instrument for this population and for each one of the homogeneous client groups. Five participation domains are identified: Autonomy indoors, Family role, Autonomy outdoors, Social life and relations (as in IPA), and Financial autonomy as a new domain. CFA showed acceptable fit of this model for the whole research population and for all subgroups separately, though fit was marginally acceptable for people with severe physical disabilities.

Six items of the original IPA questionnaire were eliminated in the IPA-MO model since they showed high correlation with other items. Whereas nearly identical items may be useful when the questionnaire is used on an individual basis – as in rehabilitation practice –, in our surveys these items hardly added any information and hurt validity of the instrument by causing high error covariances. The shortened domains showed good reliability, high conceptual similarity with the original domains and resulted in acceptable fit for the overall model. Elimination of highly correlated items is often applied when screening instruments from medical practice are used for research purposes, also for efficiency purposes [33].

All participation domains of the original IPA model proved valid for social-support clients, though minor changes occur. One of the most salient changes is that the item on reciprocity (‘supporting other people’) is related to Autonomy outdoors instead of to Social life and relations. In studies among elderly people, Haak [34] and Sixsmith [35] found that ‘doing things for others’ is an important basis for participation of aged people and strengthens their personal identity. Elderly people form a minority in our research population so this may also be true for younger persons that do not participate in the labour process. The fact that supporting other people did not contribute to Autonomy outdoors for participants with work, further supports this assumption. Since reciprocity is an important feature of a participation society, its potential to improve participation deserves further research.

Financial autonomy is a new participation domain as a result of two new items added to the original single item on this subject. The new items are derived from an institutional screening instrument. Whereas this domain is statistically reliable and valid, its items are few and conceptually almost identical (extremely high inter-item covariance). Further research is recommended to investigate if clients’ perception of financial autonomy is sufficiently covered.

Validation of the participation domain Work and education was based on a small group of participants due to high non-response. Whereas good domain reliability was found, validity needs further testing. In our sample, education proved to correlate weakly with work items, and contact with colleagues loaded onto Social life and relations.

Based on the analysis in this study, both disability-related groups and age groups are statistically valid options to identify homogeneous subgroups within the social-support population. Disability-related subgroups have higher discriminatory power as they reveal larger differences, both in terms of model fit and scores on participation domains and perceived well-being.

The domains in our model are all interrelated. Strongest interrelation is found for Autonomy outdoors, Family role and Autonomy indoors. An IPA study among rehabilitation clients in Iran showed similar findings, to the extent that Autonomy outdoors, Family role and Autonomy indoors were combined into one domain [19]. The new domain Financial autonomy shows the highest interrelation with Autonomy outdoors and Social life and relations. For our study population, Autonomy outdoors and Family role load onto the same factor for most groups, yet model fit is better when they are treated as separate domains.

By its very nature, Autonomy outdoors – meeting and supporting people, spending leisure time, paying visits, making trips – represents the way people participate in society. The near-symbiosis with Family role suggests that for social-support clients, Autonomy outdoors is closely related to home-based activities. As most of them do not work and many live alone, their connection with society seems weak. This is probably (part of) the reason why these people receive social support. A relevant question for the aspired transformation towards a participation society is whether social support can reinforce connections with and improve participation in society. The IPA-MO instrument can be helpful to monitor this development.

Reliable outcome measurement is a key element for outcome-oriented policies (Brickley et al. [8]). Most local governments in the Netherlands are reorganising their social support interventions so as to achieve more efficiency and avoid budget deficits: e.g. family members are expected to contribute more to supporting their relatives and general facilities are being created to replace individual support. Whereas this decreases expenditure, the social impact of these policy measures remains largely unknown. The IPA-MO instrument provides an option for local governments to measure the social outcomes of their policies in a reliable way. For this purpose, they can conduct periodic surveys among a representative sample of inhabitants living with disabilities. As the IPA-MO amounts to an overall score representing perceived autonomy and participation and weights for domain scores can be established (further research), the instrument can be used for social cost-benefit analyses [36] which allows insight into the overall impact of social policies.

Having a valid instrument at their disposal to measure outcomes, also offers municipalities possibilities to contract integrated care or care-service packages, and replace the fee-for-service with value-based reimbursement systems. This will shift the incentive for providers from production to more meaningful results and it is expected to encourage innovation and reduce the administrative burden in social support.

Finally, outcome measurement by IPA-MO puts clients’ perspectives at the centre of policy making. Clients are numerous and dispersed and, in spite of the best intentions, hard to involve widely in policy processes. Taking their perceptions of autonomy and participation as the acid test for the effectiveness of social support policies, may prove a serious game-changer in politics.

Conclusions

This study demonstrates that the IPA-MO is a valid and reliable instrument to assess outcomes of social support in terms of self-reliance and participation among heterogeneous populations, including people that live with physical, intellectual or mental disabilities as well as people dependent on income support. The instrument was built on the existing and validated IPA instrument for rehabilitation clients and adapted for the purpose of impact measurement of social support policies among the target population of local governments. In comparison with the original IPA, several items were eliminated and a new domain Financial autonomy was included. Further research is needed on face validity and responsiveness of this new domain. The IPA-MO instrument allows for the distinction of subgroups based on age or nature of the disability, which provides local policy makers with more specific information on autonomy and participation so as to target policies to the potential and needs of different groups.

Notes

  1. 1.

    Following the instruction manual, sections of the questionnaire are invalid when less than 75% of the items have been answered

  2. 2.

    Up to the age of 95 years old, the majority of Dutch inhabitants live independently. Under the Social Support Act (Wmo), local governments have the responsibility to assist them with technical aids and ambulant support if needed. Given the ageing Dutch population and the fact that government policies have restricted entrance to intramural care, the number of elderly Wmo clients is likely to increase [37]

  3. 3.

    E.g. 6a is “Getting washed and dressed the way I want” and 6b is “getting washed and dressed when I want”

Abbreviations

ADHD:

Attention deficit hyperactivity disorder

AI:

Autonomy indoors

ALS:

Amyotrophic lateral sclerosis

AO:

Autonomy outdoors

CFA:

Confirmatory Factor Analysis

CFI:

Comparative Fit Index

EFA:

Exploratory Factor Analysis

FIN:

Financial autonomy

FR:

Family role

ICF:

International Classification of Functioning, Disability and Health

IPA:

Impact on Participation and Autonomy questionnaire

IPA-MO:

Impact on Participation and Autonomy questionnaire of Social Support

PCF:

Principal Component Factoring

Sem:

Structural equation modelling

SOC:

Social life and relations

SRMR:

Standardised roots mean squared residual

TLI:

Tucker Lewis Index

WMO:

Social Support Act (Wet Maatschappelijke Ondersteuning)

WORK:

Work and education

References

  1. 1.

    Government Gazette of the Netherlands. Uitvoeringsbesluit Wmo 2015. Staatsblad van het Koninkrijk der Nederlanden, 2014, nr 420. 2014.

    Google Scholar 

  2. 2.

    Government Gazette of the Netherlands. Invoeringswet Participatiewet.: Staatsblad van het Koninkrijk der Nederlanden, 2014, nr 270. 2014.

    Google Scholar 

  3. 3.

    Government Gazette of the Netherlands. Jeugdwet Staatsblad van het Koninkrijk der Nederlanden, 2014, nr 105. 2014.

    Google Scholar 

  4. 4.

    Raad voor Maatschappelijke Ontwikkeling. Terugtreden is vooruitzien. The Hague: Advies 54; 2014.

    Google Scholar 

  5. 5.

    WHO. Towards a common language for functioning, disability and health: ICF – the International Classification of Functioning, Disability and Health. Geneva: World Health Organisation; 2002.

  6. 6.

    Ministry of Health/Transition Bureau Introduction Wmo. Bekostigingsmodellen – modellen, afwegingen en uitdagingen voor gemeenten. Den Haag: Nyfer. 2014. https://www.nyfer.nl/publicaties.html

  7. 7.

    Berenschot L, van der Geest L. Handreiking ‘Sturen op uitkomsten in de Wmo – investeren in maatschappelijke participatie’. Utrecht: Nyfer.

  8. 8.

    Brickley JA, Smith CW, Zimmerman JL. Managerial economics and organizational architecture. Boston: McGraw-Hill; 2014.

  9. 9.

    Cardol M. Beyond disability. Assessing participation and autonomy in medical rehabilitation. PhD dissertation: University of Amsterdam, Faculty of Medicine; 2001.

  10. 10.

    Ministry of Health. Memorie van Toelichting van Wet Maatschappelijke Ondersteuning. Amsterdam; 2014. https://zoek.officielebekendmakingen.nl/kst-33841-3.pdf.

  11. 11.

    Costanza R, Fisher B, Ali S, Beer C, Bond L, Boumans R, Danigelis NL, Dickinson J, Elliott C, Farley J, Elliott Gayer D, MacDonald Glenn L, Hudspeth T, Mahoney D, McCahill L, McIntosh B, Reed B, Abu Turab S, Rizvi DM, Rizzo T, Simpatico RS. Quality of life: an approach integrating opportunities, human needs and subjective well-being. Ecol Econ. 2007;61:267–76.

    Article  Google Scholar 

  12. 12.

    Häggström A, Larsson M. The complexity of participation in daily life: a qualitative study of the experiences of persons with acquired brain injury. J Rehabil Med. 2008;40:89–95.

    Article  Google Scholar 

  13. 13.

    Whiteneck G, Dijkers MP. Difficult to measure constructs: conceptual and methodological issues concerning participation and environmental factors. Arch Phys Med Rehabil. 2009;90(11):S22–35.

    Article  Google Scholar 

  14. 14.

    Cardol MM, de Haan RJ, de Jong BA, van den Bos GAM, de Groot IJM. Psychometric properties of the ‘Impact on Participation and Autonomy’ (IPA) questionnaire. Arch Phys Med Rehabil. 2001;82:210–6.

    CAS  Article  Google Scholar 

  15. 15.

    Sibley A, Kersten P, White WB, Mehta R, George S. Measuring autonomy in disabled people: validation of a new domain in a UK population. Clin Rehabil. 2006;20:793–803.

    CAS  Article  Google Scholar 

  16. 16.

    Lund L, Maria AGF, Lexell J, Bernspång B. Impact on Participation and Autonomy questionnaire: internal domain validity of the Swedish version for use in people with spinal cord injury. J Rehabil Med. 2007;39:156–62.

    Article  Google Scholar 

  17. 17.

    Franchignoni F, Ferriero G, Giordano A, Guglielmi V, Picco D. Rasch psychometric validation of the Impact on Participation and Autonomy questionnaire in people with Parkinson’s disease. Eura Medicophys. 2007;43(4):451–61.

    CAS  PubMed  Google Scholar 

  18. 18.

    Poulin V, Desrosiers J. Validation française de questionnaires de participation sociale auprès d’aînés en situation d’incapacités. Can J Occup Ther. 2010;77(3):159–66.

    Article  Google Scholar 

  19. 19.

    Fallahpour M, Tham K, Joghataei MT, Jonsson H. Perceived participation and autonomy: aspects of functioning and contextual factors predicting participation after stroke. J Rehabil Med. 2011;43:388–97.

    Article  Google Scholar 

  20. 20.

    Fallahpour M, Jonsson H, Taghi Joghata M, Kottorp A. Impact on Participation and Autonomy (IPA): psychometric evaluation of the Persian version to use for persons with stroke. Scand J Occup Ther. 2011;18:59–71.

    Article  Google Scholar 

  21. 21.

    Sutiwong J, Vongsirinavarat M, Vachalathiti R, Chaiyawat P. Impact on Participation and Autonomy questionnaire: psychometric properties of the Thai version. J Phys Ther Sci. 2013;25:769–74.

    Article  Google Scholar 

  22. 22.

    Mueller M, Strobl R, Jahn K, Linkohr B, Ladwig KH, Mielck A, Grill E. Impact of vertigo and dizziness on self-perceived participation and autonomy in older adults: results from the KORA-age study. Qual Life Res. 2014;23:2301–8.

    Article  Google Scholar 

  23. 23.

    Wilkie R, Jordan JL, Muller S, Nicholls E, Healey EL, van der Windt DA. Measures of social function and participation in musculoskeletal populations. Arthritis Care Res. 2011;63(S11):S325–36.

    Article  Google Scholar 

  24. 24.

    Divosa/Regioplan. Mesis – methodisch screeningsinstrument schulddienst-verlening. Utrecht; 2012.

  25. 25.

    Oomens S, T de Jong, RWB Blonk. Gezondheid en functioneren van moeilijk bemiddelbare bijstandsgerechtigden in een gesubsidieerde baan. Gedrag & Organisatie, 2016, Jrg 22. 2009. Nr 4.

    Google Scholar 

  26. 26.

    HHM. Cliëntgroepen extramurale begeleiding. Transitiebureau Invoering Wmo. The Hague; 2014.

  27. 27.

    Central Bureau of Statistics. Gemeentelijke monitor sociaal domein. https://www.cbs.nl/nl-nl/onze-diensten/methoden/onderzoeksomschrijvingen/korte-onderzoeksbeschrijvingen/gemeentelijke-monitor-sociaal-domein-wet-maatschappelijke-ondersteuning. Accessed 6 Sept 2017

  28. 28.

    Ottenvall Hammar I, Ekelund C, Wilhelmson K, Eklund K. Impact on Participation and Autonomy: test of validity and reliability for older persons. Health Psychol Res. 2014;2:1825.

    Article  Google Scholar 

  29. 29.

    Mönks FJ. Ontwikkelingspsychologie. Inleiding tot de verschillende deelgebieden: Van Gorcum; 2015.

  30. 30.

    van Houtum L, Cardol M, de Jong JD. Ondanks ‘keuzebiografie’ blijft de keuze beperkt: participatie naar levensfase. Netherlands: TSG: Tijdschrift voor Gezondheidswetenschappen. 2012;90(4):245–52.

  31. 31.

    Hu L, Bentler PM. Cut off criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J. 1999;6(1):1–55.

    Article  Google Scholar 

  32. 32.

    Cross Validated.. Average inter-item correlations as internal consistency indicator. 2016. https://stats.stackexchange.com/questions/198506/average-inter-item-correlations-as-internal-consistency-indicator. Accessed 30 Sept 2016.

  33. 33.

    Shen J, Gao S. A solution to separation and multicollinearity in multiple logistic regression. J Data Sci. 2008;6(4):515–31.

    PubMed  PubMed Central  Google Scholar 

  34. 34.

    Haak M, Dahlin Ivanoff S, Fänge A, Sixsmith J, Iwarsson S. Home as the locus and origin for participation: experiences among very old Swedish people. OJTR: Occupation, Participation and Health. Summer. 2007;27(3):95–103.

    Google Scholar 

  35. 35.

    Sixsmith J, Boneham M. Older men’s participation in community life: notions of social capital, health and empowerment. Ageing Int. 2003;28(4):372–88.

    Article  Google Scholar 

  36. 36.

    SEO. Werkwijzer voor kosten-batenanalyse in het sociale domein. (Guide for cost-benefit analysis in social policies). Rotterdam; 2016.

  37. 37.

    van Campen C, Iedema J, van Groenou MB, Deeg D. Langer zelfstandig – ouder worden met hulpbronnen, ondersteuning en zorg. SCP. Den Haag; 2017.

  38. 38.

    Pommer E, Boelhouwer J, van den Berg E, den Draak M. Rondom de transitie - Overall rapportage sociaal domein 2015. Sociaal Cultureel Planbureau. The Hage; 2016.

  39. 39.

    Central Bureau of Statistics. Statline/ Sociale zekerheid/ personen met bijstand/duur van de uitkering. http://www.statline.cbs.nl.

  40. 40.

    Costello AB, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from our analysis. Pract Assess Res Eval. 2005;10(7):1.

    Google Scholar 

Download references

Acknowledgements

We would like to thank the municipalities that participated in the surveys and all the people that filled out the questionnaire, often with additional illustrations, for the insight they allow us into their perceptions and priorities regarding self-reliance and participation.

Funding

The research for this article was funded by the authors’ organisations: Berenschot Onderzoek en Advies and University of Utrecht School of Economics.

Availability of data and materials

Available upon request.

Author information

Affiliations

Authors

Contributions

LB contributed with general context and analysis. YG contributed as methodological guide and was responsible for quality control. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Yolanda Grift.

Ethics declarations

Authors’ information

Yolanda Grift has a long career in econometrics, both as a teacher and in research. She is a dedicated academic and worked in multiple partnerships with consultancy and executive agencies so as to foster interaction between science and practice. Lucienne Berenschot has been working as a consultant on health and social support for 15+ years. She has a keen interest in modernisation of the health system, taking the perspective of the target population as a starting point and reorganising service provision in a technological and attitudinal up-to-date way.

Ethics approval and consent to participate

Participation in all surveys was on a strictly voluntary basis. Participants were asked to contribute in order to enhance insight into daily reality from their perspective, with the guarantee of anonymity.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 11 The IPA-MO questionnaire used for social-support clients

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Berenschot, L., Grift, Y. Validity and reliability of the (adjusted) Impact on Participation and Autonomy questionnaire for social-support populations. Health Qual Life Outcomes 17, 41 (2019). https://doi.org/10.1186/s12955-019-1106-0

Download citation

Keywords

  • Social support
  • Participation
  • Autonomy
  • IPA
  • Impairment
  • Validation
  • Factor analysis