RESNA 27th International Annual Confence
Psychometric and Administrative Properties of Measures Used in Assistive Technology Device Outcomes Research
This paper summarizes the measurement domains and properties of tools used in 82 assistive technology device outcome studies published between 1980 and 2001. Few authors offered adequate evidence supporting the reliability or validity of the tools used to collect data. Most offered no information regarding the preparation time required for learning the tools, administering them to research participants, or scoring them.
outcomes, measurement, research, psychometrics, domains
The properties of reliability, validity, and administrative burden associated with a particular measurement instrument support the credibility of data gathered using the tool [1, 2]. These properties are considered essential components of formal research reporting [3]. Although there have been numerous calls for increasing the quantity and quality of assistive technology (AT) outcomes research [4-6], no one has analyzed the nature of data that the field has been accumulating. This paper summarizes our evaluation of 82 outcome studies, published between 1980 and 2001, addressing AT devices (ATDs).
The objective was to identify all research studies published from 1980 to 2001 that involved a follow-up study of ATD users and that measured one or more domains of ATD impact. Minimally, a sub-set of the study participants must have been users of one or more devices in any of the following categories: mobility aids (including crutches, canes, walkers, and wheeled mobility devices), seating and postural support devices for wheelchair users; augmentative and alternative communication, computer-based ATDs, or environmental control systems. Articles excluded from consideration were case studies, ethnographic studies, review articles, meta-analyses, and conference papers. A comprehensive literature search, restricted to English language articles involving AT device outcomes research, was conducted using several bibliographic databases (CINAHL, Medline, EMBASE, ERIC, and PsychInfo).
The coding syllabus was adapted from the format used by Dijkers and colleagues for evaluating the medical rehabilitation literature [2]. Outcome variables (i.e., dependent variables) and their corresponding measurement tools were identified from the Methods sections of the articles.
Initially, five papers were coded in parallel by the first three authors in order to evaluate the coding instructions, uncover limitations in usability of the coding form, and refine the coding categories. Five additional papers were coded by the same three individuals in order to reach consensus regarding interpretation and coding of idiosyncratic scenarios. The first author coded the remaining papers.
The inclusion criteria were met by 82 articles involving 212 outcome variables. The majority of studies (56%) were conducted in the United States, with the rest based in Canada, Europe, and Australia.
AT Device |
# of articles (%) |
Environmental control units |
10 (12%) |
Computer access |
10 (12%) |
Augmentative communication |
8 (10%) |
Self-care |
8 (10%) |
Seat cushion |
5 (6%) |
Manual wheelchair |
4 (5%) |
Mobility aid, non-wheeled |
4 (5%) |
Power wheelchair |
2 (2%) |
Unable to determine |
31 (38%) |
TOTAL |
82 (100%) |
The study samples in each article were evaluated in terms of disabling condition, type of ATD used, and age group. Nine categories of disabling condition were used to classify the studies' participants. The largest category by far was neurologic impairments. The dominant impairment category could not be determined for 30% of the articles due to imprecise descriptions of the proportions of disabling conditions present in sample populations. Nine categories were used to categorize the ATDs that were studied. Table 1 summarizes the number of articles for which various ATDs were the dominant device category. The dominant device category could not be determined for 38% of the studies because insufficient information was presented to make that determination. Three categories were used to record the mix of age groups present in each study: children (birth to age 17), adults (ages 18 to 65), and elders (greater than age 65). For 21 studies (26%), the modal age category could not be determined because insufficient information was presented. In summary, very few (~10%) studies included participants whose disability and age fell into a single category and who used a single category of ATD. Almost one-third (25 of 82) of the studies used a sample that included multiple categories of disabling condition and age who used multiple categories of ATDs.
| Domain | # of outcome variables (%) |
|---|---|
Usability |
71 (34%) |
Use |
49 (23%) |
User Satisfaction |
24 (11%) |
Functional Level |
24 (11%) |
Quality of Life |
17 (8%) |
Role Participation |
16 (8%) |
Cost |
11 (5%) |
Total |
212 (100%) |
The 212 outcome variables were categorized into one of seven different domains, with ATD usability, usage, user satisfaction, and functional level comprising approximately 80% of the variables identified (Table 2). Of 212 reported outcome variables, 168 (79%) were measured using study-specific instruments, i.e., tools developed to serve the purposes of the particular study. Forty-four (21%) variables were measured using previously published measurement tools.
One hundred forty-nine (70%) outcome variables were obtained via participant self-report, and 28 (13%) were based on clinician observation and rating. Almost two-thirds of the variables were measured with either nominal or ordinal data. Approximately 12% of the variables were not defined well enough to identify the data level.
Few authors offered adequate evidence supporting the reliability or validity of the tools used to collect data (Table 3). Most offered no information regarding the preparation time required for learning the tools, administering them to research participants, or scoring them.
Type of Psychometric or Administrative Evidence |
No report |
Some report, questionable or incomplete applicability |
Satisfactory presentation of evidence & applicability |
|---|---|---|---|
Test-retest reliability |
195 (92%) |
10 (5%) |
7 (3%) |
Inter-rater reliability |
187 (88%) |
11 (5%) |
14 (7%) |
Content validity |
191 (90%) |
7 (3%) |
14 (7%) |
Criterion validity |
196 (93%) |
6 (3%) |
10 (5%) |
Construct validity |
196 (92%) |
7 (3%) |
9 (4%) |
Training required to learn tool |
207 (98%) |
1 (<1%) |
4 (2%) |
Time to administer the tool |
181 (85%) |
17 (8%) |
14 (7%) |
Time to score the tool |
211 (99%) |
1 (<1%) |
0 (0%) |
The studies' sample populations exhibited substantial heterogeneity, characterized by merging a number of distinguishable subgroups (e.g., involving age, disabling condition, or type of ATD used) and not differentiating the findings in terms of those groups. Sample heterogeneity is desirable in studies hypothesizing that an intervention is robust across a range of subgroups. However, it poses hazards for exploratory studies that lack such hypotheses, because it allows the results to be confounded by factors that have neither been identified nor controlled. The effect is to muddle interpretation of the
findings and limit their contribution to a cohesive body of evidence-based literature [7].
A preponderance of the studies' measurement tools had not been published independently. In many cases the immaturity of the AT outcomes research area has forced researchers to develop their own measures. Nonetheless, serious questions arise from using psychometrically unproven instruments. Genuine treatment effects may go undetected if the reliability of measures is actually weak. Tools lacking validity can result in systematic under-estimation, over-estimation, or misrepresentation of treatment effects. Neglect in reporting psychometric background data is not unique to the AT field. Dijkers et al. found similar results in their evaluation of medical rehabilitation research literature [2].
Based on our findings, we offer several recommendations in order to elevate the quality of ATD outcomes research reporting:
Funding was provided in part by grant H133A010401 from the National Institute of Disability and Rehabilitation Research. The first author extends heartfelt appreciation to Crystal Yalch for gathering most of the articles reviewed here.
Jim Lenker,
Department of Rehabilitation Science,
School of Public Health and Health Professions,
University at Buffalo, 14214-3079.
(716) 829-3141, x109;
lenker@buffalo.edu