The AMC Linear Disability Score project in a population requiring residential care: psychometric properties
© Holman et al; licensee BioMed Central Ltd. 2004
Received: 22 June 2004
Accepted: 03 August 2004
Published: 03 August 2004
Currently there is a lot of interest in the flexible framework offered by item banks for measuring patient relevant outcomes, including functional status. However, there are few item banks, which have been developed to quantify functional status, as expressed by the ability to perform activities of daily life.
This paper examines the psychometric properties of the AMC Linear Disability Score (ALDS) project item bank using an item response theory model and full information factor analysis. Data were collected from 555 respondents on a total of 160 items.
Following the analysis, 79 items remained in the item bank. The remaining 81 items were excluded because of: difficulties in presentation (1 item); low levels of variation in response pattern (28 items); significant differences in measurement characteristics for males and females or for respondents under or over 85 years old (26 items); or lack of model fit to the data at item level (26 items).
It is conceivable that the item bank will have different measurement characteristics for other patient or demographic populations. However, these results indicate that the ALDS item bank has sound psychometric properties for respondents in residential care settings and could form a stable base for measuring functional status in a range of situations, including the implementation of computerised adaptive testing of functional status.
It is now widely accepted that examining quality of life is an important aspect in the treatment and evaluation of many conditions. Functional status is seen as an important determinant of quality of life. A wide variety of instruments have been developed to quantify functional status . These instruments tend to have a fixed length and all items are administered to the whole group of patients under scrutiny. However, currently interest is moving towards the more flexible framework offered by item banks. An item bank is a collection of items, for which the measurement properties of each item are known [2, 3]. When using an item bank, it is not essential for all respondents to be examined using all items. This enables the burden of testing to be considerably reduced for both patients and researchers. It is even possible to select the 'best' items for individual patients using computerised adaptive testing algorithms . Furthermore, results from studies using different selections of items from an item bank can be directly compared. Item banks, measuring concepts such as quality of life [2, 5], the impact of headaches  or functional status [7, 8], have been developed.
The AMC Linear Disability Score (ALDS) project item bank was developed to quantify functional status [7, 9]. The ALDS item bank covers a large number of activities, which are suitable for assessing respondents with a very wide range of functional status and many types of chronic condition. The item bank is particularly suitable for use in the Netherlands. The ALDS items were obtained from a systematic review of generic and disease specific functional health instruments . Five psychometric aspects of the ALDS item bank need to be considered before it can be implemented. These are: (a) there needs to be enough variation in the response categories used for each item ; (b) estimates of the item response theory model parameters should not depend on patient characteristics such as age or gender [10, 11]; (c) estimates of the item response theory model parameters, which are stable across different subsets of items from the instrument and based on a sufficiently large sample  of respondents, should be available ; (d) an examination of the extent to which the ALDS items represent a single construct; and (e) testing whether a simpler item response theory model is suitable for the set of items.
This paper examines these five aspects of the ALDS item bank using the responses given by residents of supported housing schemes, residential care and nursing homes in and around Amsterdam, the Netherlands. This, mainly elderly, population has been chosen because they generally experience some level of functional restriction and consume a large amount of health care services.
This paper considers 160 items, which were considered to be applicable in a residential care setting. Each item has two response categories: 'I could carry out the activity' and 'I could not carry out the activity'. If a respondent had never had the opportunity to experience an activity 'not applicable' was recorded. In the analysis, responses in the category 'not applicable' were treated as if the individual items had not been presented to the individual respondents . It was felt that presenting all 160 items to each respondent would place an unnecessary and unacceptable burden on those responding to the items. Therefore, the data described in this paper were collected using an incomplete, anchored calibration design [7, 9, 14, 15] with four sets of 80 items. Item sets A and B have half their items in common, as do item sets B and C, item sets C and D and item sets A and D. The items in common between two sets of items are known as 'anchors' and allow all items and patients to be calibrated on the same scale. The patterns of missing data in this type of design are, in statistical terms, ignorable . The item sets were administered randomly to 150 respondents (item set A), 143 respondents (item set B), 138 respondents (item set C) and 124 respondents (item set D).
A total of 555 residents of supported housing, residential care and nursing homes were interviewed. The median age was 84 years (range 37 to 101 years), while 444 (80%) were female. Since the respondents were interviewed 'at home', accurate data on medical conditions were not available. All respondents gave informed consent. The study was approved by the medical ethics committee in our hospital.
The item response theory models
In this paper the data were analysed using the two-parameter logistic item response theory model [7, 9, 17, 18]. In this model, the probability, P ik , that patient k responds to item i in the category 'can' is modelled using
where θ k denotes the ability of patient k to perform activities of daily life. The discrimination parameter (α i ) and difficulty parameter (β i ) describe the measurement characteristics of item i. The larger the value of β i , the more difficult item i is. In addition, the larger the value of α i , the better an item is a distinguishing between abilities above and below β i . If the values of α i are constrained to be equal for all items, the model in equation 1 becomes the one-parameter logistic item response theory model . The model in equation 1 can be extended to test whether the values of β i for, say males and females, are significantly different. If the values of β i for different groups of respondents are significantly different, then there is evidence of differential item functioning. Full-information factor analysis also uses an extension of the model in equation 1. These approaches are described in mathematical terms in the Appendix. In this paper, estimates of α i and β i were obtained using a marginal maximum likelihood based procedure . This method assumes that the ability parameters (θ k ) follow a Normal distribution and can account for incomplete designs, as described in the Appendix. Expected a posteriori methods were used to estimate θ k .
To achieve the objectives of this study, there were five steps in the statistical analysis. In step (a), the amount of variation in the response categories used for each item  was considered and items demonstrating too little variation were removed. Items were excluded from further analysis if fewer than 10% or more than 90% of the patients responded in the category 'cannot'. In step (b), the items were examined to investigate whether the value of the item difficulty parameter (β i ) was similar for male and female patients and for patients younger than 85 years and those aged 85 or older. The model is described in depth in the Appendix. Items were excluded from further analysis if the value of the item difficulty parameter was significantly different (1% level) between gender or aged based groups. In this step, the fit of the model to the data from each item was not assessed. In step (c), estimates of the item parameters (α i and β i ) were obtained. The fit of the model to the data from each item was assessed using G2 statistics . Items, for which the fit statistic had a p-value of less than 0.01, were excluded from the item bank. In addition, the stability of the estimates of the item parameters over different sets of items was examined using the model from step (b). Items were excluded from further analysis if the value of the item difficulty parameter was significantly different (1% level) between item sets A and B, B and C, C and D or A and D. Furthermore, a Kolmogornov-Smirnov test was carried out to examine whether the ability parameters (θ k ) were Normally distributed. In step (d), the dimensionality of the item bank was examined using item response theory based full information factor analysis [18, 22, 23]. The number of latent roots greater than 1 is regarded as an indicator of the number of factors in the data set. This method is described in more depth in the Appendix. Four exploratory factor analyses were carried out, one on each of the anchors between item sets A and B (293 respondents), B and C (281 respondents), C and D (262 respondents) or A and D (274 respondents). A fifth, confirmatory, factor analysis was carried out on the whole data set (555 respondents). In addition, Cronbach's coefficient alpha was calculated for each anchor and the whole data set [24, 25]. In step (e) the one-parameter logistic item response theory model was fitted to the remaining items. The differences between the -2log likelihoods of this model and the two-parameter model fitted in step (c) was tested using a χ2 test. The analysis in steps (a), (b), (c) and (e) was carried out in Bilog, version 3.0 . The analysis in step (d) was carried out using TESTFACT, version 4.0 .
The number of items proceeding to each step of the analysis The number of and examples of items removed at each stage of the psychometric analysis.
Stage of analysis
Number of items removed
Reason for removal
Concerns about the way the item was presented
< 10% or > 90% of responses in 'cannot'
Reaching for a cup and taking a sip of water
Combing hair at a sink
Cycling on a heavily laden bicycle
Significant difference between M and F and/or under and over 85 years
Washing up (easier for older respondents)
Crossing the street (easier for younger respondents
Preparing a warm meal (easier for female respondents)
Item fit p-value < 0.01 or estimates of β i not stable
Taking oral medication
Getting money out of the bank using an ATM
In item bank
See Table 2
The 79 items remaining in the calibrated item bank. The items remaining in the calibrated item bank. The number of respondents, to whom the item was offered (Offered to), the number responding in the category 'not applicable' (NA), the number responding in the category 'can' (can) and the number responding in the category 'cannot' (cannot) are given. The discrimination (β) and difficulty (β) parameters are given along with their standard errors in parentheses.
Description of item content
Item response category
Location parameter (β)
Discrimination parameter (α)
Walking up stairs with a bag
Mopping a flight of stairs
Cleaning the top of a high cupboard
Cleaning a bathroom
Going for a walk in the woods
Fetching groceries for 3–4 days
Mopping the floor
Caring for plants on a balcony
Travelling by bus or tram
Walking up two flights of stairs
Cleaning a fridge
Going to a restaurant
Carrying a tray
Going for a long walk (15+ minutes)
Going to the dentist
Sweeping the floor
Cutting toe nails
Walking up a hill or bridge
Walking up one flight of stairs
Going to a concert
Going to the pharmacist
Hanging a load of washing out
Going to the post office or bank
Going to a party
Filling an official form in
Using a washing machine
Visiting an outpatients' clinic
Taking bottles to the bottle bank
Short walk (less than 15 minutes)
Putting a rubbish bag outside
Reaching into a high cupboard
Using a dustpan and brush
Opening and closing a high window
Fetching groceries for one day
Using a public toilet
Putting flowers in a vase
Frying an egg
Warming up a tin of soup
Cleaning a toilet
Putting socks and lace up shoes on
Changing the bulb in a table light
Cleaning a bathroom sink
Cutting finger nails
Rubbing lotion into whole body
Reaching into a low cupboard
Picking something up off the floor
Getting in and out of a car
Shaking a tablecloth out
Making a bed
Preparing breakfast or lunch
Using the lift in a public building
Putting an alarm clock right
Pulling a blanket up
Visiting the neighbours
Travelling as a passenger in a car
Shaving face or applying make up
Watering a house plant
Opening and closing a window
Putting trousers on
Making coffee or tea
Peeling an apple
Making a bowl of cereal
Eating a meal at the table
Hanging clothes up in a cupboard
Opening and closing curtains
Moving between two dining chairs
Putting a scarf and gloves on
Making a cheese sandwich
Moving to sit on the edge of a bed
Putting a coat on
Putting a shirt or blouse on
Washing upper body at a sink
Answering the front door
Getting out of bed into a chair
Washing lower body at sink
Putting a T-shirt on
Locking a door
In this study, the psychometric properties of the item bank have been examined using a sample of 555 respondents and an incomplete calibration design. Each item was presented to between 262 and 293 respondents. These figures are above the minimum, of 200 respondents, regarded as necessary to implement the models used in this paper . It could be argued that it would have been desirable for all respondents to be presented with all items, but this would have placed an unacceptable burden on the, often frail, population in this study. Incomplete calibration designs are regularly implemented in the development and maintenance of item banks used in educational testing [4, 14] and have gained some recognition in health related applications . Developments in psychometric theory mean that it is now possible to perform the same types of analysis on data resulting from incomplete designs, as is performed on data from complete calibration designs [22, 23, 25]. The number of items in the anchors following the analysis, indicate that the design was still amply linked .
One of the major assumptions underlying the use of the item response theory models described in this paper is that the items reflect a single latent trait (θ). This has been examined using item response theory based full-information factor analysis. Part of the full-information factor analysis was performed on sub-sets of the data, as exploratory analyses on incomplete designs may lead to instable results. However, the confirmatory factor analysis was performed on all data. The results, together with the high level of internal consistency, as measured by Cronbach's alpha, and the acceptable fit of the two-parameter logistic item response theory model to the data indicate that the items presented in this paper probably represent a unidimensional construct in a population of respondents requiring residential care.
Another important assumption when using item response theory models in conjunction with marginal maximum likelihood estimation procedures is that the values of the latent trait (θ) follow a pre-specified, usually Normal, distribution. In this study, there was no evidence that these values did not follow a Normal distribution. This is in contrast to many previously published studies into health and quality of life outcomes, where a strongly skewed distribution was found. The authors feel that there are two reasons for this contrast. Firstly, in this study, the respondents all had some level of restriction in their ability to perform activities of daily life. Secondly, the item bank includes items well above and well below the level of functional status enjoyed by the respondents. This means that the item bank did not have a ceiling or floor effect with respect this this population.
In this study, 81 (51%) of 160 items were removed from the item bank because they did not conform to the psychometric standards required of the item bank. This is a much higher level than would be expected in the calibration of an item bank for use in educational measurement. However, when the results are examined more carefully, 28 items were removed because they were too difficult or too easy for the population in this study. In addition, 26 items were removed because they had different item parameters for different groups of respondents. These problems would have been identified much earlier in an educational item bank. Hence, only 26 (25%) of 106 items were removed due to item misfit. The number of items retained in the item bank may have been higher if a more flexible model, based on, for example, non-parametric smoothing techniques had been used . However, this type of model is less suitable as a base for implementing modern testing algorithms, such as computerised adaptive testing. In addition, it is possible that more items could be made available if the items demonstrating differential item functioning were included in the item bank with different item location parameters (β i ) for males and females or for younger and older respondents. This may seem complicated, but is straightforward in the framework of a computerised item bank.
This paper has concentrated on the two-parameter logistic item response theory model. However, the one-parameter logistic item response theory model was also fitted to the 79 items remaining in the item bank. This model fitted the data significantly less well than the two-parameter model, even after 3 items demonstrating misfit at the item level were removed. This confirms the choice of the two-parameter model. This model was chosen because it allows the probability of responding in the category 'can' to be modelled more flexibly than when the one-parameter logistic model is used. This enables a more realistic model for the data to be built than when the more restrictive approach associated with the one-parameter model is chosen .
This paper has examined the calibration of the ALDS item bank in a population requiring residential care. It has been shown that the item bank has sound psychometric properties and could form a stable base for a wide range of applications. However, it is possible that the items will have different measurement characteristics for patients requiring treatment for specific chronic conditions or in other countries. Hence, it is important that the ALDS item bank is tested carefully before it is used to assess the functional status in other groups of respondents or in other countries.
Now that the measurement properties of the ALDS item bank have been examined carefully, the item bank can be used as a foundation for quantifying functional status. If modern algorithms, such as computerised adaptive testing, are implemented, it will be possible to obtain accurate measurements, whilst keeping the burden of testing on respondents and interviewers to a minimum. Items can be selected for use in further research, for allocation individuals to appropriate care settings and for calculating institutional funding based on the actual care load. It is hoped that the ALDS item bank will play an important part in the implementation of computerised adaptive testing of functional status.
Differential item functioning
The model in equation 1 can be extended to test whether different groups of respondents to have different values of β i . This is known as differential item functioning. For instance, if interest is in possible differences in β i between males and females, then the probability, P ik , that patient k responds to item i in the category 'can' is written as
where β iM is the item difficulty for male respondents, β iF-M is the difference between the item difficulty for males and for females and I k is an indicator variable taking the value 0 if respondent k is male and the value 1 if respondent k is female. The hypothesis H0 : β iF-M = 0 can be tested to examine whether item i has the same measurement characteristics for males and for females.
Item parameter estimation in incomplete designs
In this study, the item parameters (α i and β i ) were estimated using marginal maximum likelihood methods. The likelihood, L, over n items and K (K = 555) respondents can be written as
where I ik is an indicator variable taking the value 1 if respondent k was offered item i and the value 0 otherwise and where J ik is an indicator variable taking the value 1 if respondent k responded to item i in the category 'can' and the value 0 otherwise. Furthermore, the probability, P ik , that respondent k responded to item i in the category 'can' is as in equation 1, or, where appropriate, as in equation 2 or 4. In the estimation process, the values of θ k or θ km were assumed to follow a Normal distribution with mean equal to 0 and unknown variance, σ2, and were integrated out of the likelihood to obtain the marginal likelihood. The marginal likelihood was maximised using an EM algorithm .
Full information factor analysis
Full information factor analysis is a technique based on multidimensional item response theory models where the ability is represented by M variables, denoted θ km where m = 1, 2,..., M [22, 23]. The model, in equation 1, for the probability, P ik , that person k responds to item i in the category 'can' can be extended to
where θ km denotes the value of the latent variable θ m associated with person k and α im denotes the discrimination parameter for item i with respect to the latent variable θ m . Furthermore, δ i is a difficulty type parameter. The loading, a im of item i on factor m can be calculated using
The value of the standard difficulty parameter, (β i ), can be calculated using
Generally, the parameters α im and δ i are estimated using marginal maximum likelihood methods.
RH and RL were supported by a grant from the Anton Meelmeijer fonds, a charity supporting innovative research in the Academic Medical Center, Amsterdam, The Netherlands.
RL conceived the study and supervised the data collection. RH prepared the first draft and carried out the analyses. RL, MV and RJdH critically reviewed the manuscript. RH prepared the final version.
AMC Linear Disability Score
- Lindeboom R, Vermeulen M, Holman R, de Haan RJ: Activities of daily living instruments in clinical neurology. Optimizing scales for neurologic assessments. Neurology 2003, 60: 738–742.PubMedView ArticleGoogle Scholar
- Bode RK, Lai JS, Cella D, Heinemann AW: Issues in the development of an item bank. Arch Phys Med Rehabil 2003,84(4 Suppl 2):S52–60. 10.1053/apmr.2003.50247PubMedView ArticleGoogle Scholar
- McHorney CA: Ten recommendations for advancing patient-centered outcomes measurement for older persons. Ann Intern Med 2003,139(5 Pt 2):403–409.PubMedView ArticleGoogle Scholar
- van der Linden WJ, Glas CAW: Computerized Adaptive Testing. Theory and Practice Kluwer Academic Publishers, Dordrecht, the Netherlands 2000.View ArticleGoogle Scholar
- Accessed 8th July 2004, [http://www.amihealthy.com/static/dynamicsf36info.asp]
- Accessed 8th July 2004, [http://www.headachetest.com/]
- Holman R, Lindeboom R, Vermeulen M, Glas CAW, de Haan RJ: The Amsterdam Linear Disability Score (ALDS) project. The calibration of an item bank to measure functional status using item response theory. Quality of Life Newsletter 2001, 27: 4–5.Google Scholar
- Webster K, Cella D, Yost K: The functional assessment of chronic illness therapy (facit) measurement system: properties, applications, and interpretation. Health Qual Life Outcomes 2003, 1: 79. 10.1186/1477-7525-1-79PubMed CentralPubMedView ArticleGoogle Scholar
- Holman R, Lindeboom R, Glas CAW, Vermeulen M, de Haan RJ: Constructing an item bank using item response theory: the AMC Linear Disability Score project. Health Services and Outcomes Research Methodology 2003, 4: 19–33. 10.1023/A:1025824810390View ArticleGoogle Scholar
- Holman R, Lindeboom R, Vermeulen M, Glas CAW, de Haan RJ: The Amsterdam Linear Disability Score (ALDS) project. Differential item functioning with regard to gender. Quality of Life Newsletter 2002, 29: 13–14.Google Scholar
- Holman R, Lindeboom R, de Haan RJ: Gender and age based differential item functioning in the AMC Linear Disability Score project. Quality of Life Newsletter 2004, 32: 1–4.Google Scholar
- Zimowski MF, Mukari E, Mislevy RJ, Bock RD: BILOG-MG. Multiple group IRT analysis and test maintenance for binary items Scientific Software International, Inc. Chicago, IL 1996. [http://www.ssicentral.com/irt.htm]Google Scholar
- Holman R, Glas CAW, Zwinderman AH, de Haan RJ: Practical methods for dealing with 'not applicable' item responses in the AMC Linear Disability Score project. Health Qual Life Outcomes 2004, 2: 29. 10.1186/1477-7525-2-29PubMed CentralPubMedView ArticleGoogle Scholar
- Kolen MJ, Brennan RL: Test Equating Springer, New York 1995.View ArticleGoogle Scholar
- van Buuren S, Hopman-Rock M: Revision of the ICIDH severity of disabilities scale by data linking and item response theory. Stat Med 2001, 20: 1061–1076. 10.1002/sim.723PubMedView ArticleGoogle Scholar
- Mislevy RJ, Chang H: Does addaptive testing violate local independence? Psychometrika 2000, 65: 149–156.View ArticleGoogle Scholar
- Birnbaum A: Some latent trait models and their use in inferring an examinee's ability. Chapter in Lord FM, Novick MR. Statistical theories of mental test scores Addison-Wesley, Reading, MA 1968.Google Scholar
- Thissen D, Wainer H: Test Scoring LEA, Mahwah, NJ 2001.Google Scholar
- Rasch G: Probabilistic models for some intelligence and attainment tests Danish Institute for Educational Research, Copenhagen, Denmark 1960.Google Scholar
- Bock RD, Aitkin M: Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika 1981, 46: 443–459.View ArticleGoogle Scholar
- Bock RD, Mislevy RJ: Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement 1982, 6: 431–444.View ArticleGoogle Scholar
- du Toit M, editor: IRT from SSI: Bilog-MG, Multilog, Parscale, Testfact Scientific Software International, Inc, Lincolnwood, IL 2003.Google Scholar
- Bock RD, Gibbons RD, Muraki E: Full information item factor analysis. Applied Psychological Measurement 1988, 12: 261–280.View ArticleGoogle Scholar
- Cronbach LJ: Coefficient alpha and the internal structure of tests. Psychometrika 1951, 16: 297–334.View ArticleGoogle Scholar
- Harvey WR: Estimation of variance and covariance components in the mixed model. Biometrics 1970, 26: 485–504.View ArticleGoogle Scholar
- Ramsay JO: Kernal smoothing approaches to non-parametric item characteristic curve estimation. Psychometrika 1991, 56: 611–630.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.