RESNA 27th International Annual Confence
Satisfaction with Assistive Technology: What are we measuring?
As part of its comprehensive needs assessment of the state of the science in AT outcomes, the ATOMS Project (Assistive Technology Outcomes Measurement System) performed a literature review of satisfaction in health care, and specifically, satisfaction with AT (assistive technology) devices and services. This paper presents part of this investigation: the analysis of 44 published studies that specifically assessed AT consumer satisfaction. This study identified the scope and gaps identified the use of satisfaction variables. The focus of the review is on the methodological issues inherent in the attempt to quantify satisfaction with AT devices and services. The 44 studies demonstrate that there are no widely agreed upon protocols for measuring this construct or even a common definition. The field of AT is in need of standardized, validated outcome measures. However, to date, the construct of satisfaction has remained too elusive to reliably quantify.
Satisfaction, Assistive technology
Consumer satisfaction is engendering immense popularity in the field of health care. As the field of assistive technology struggles with quantifying the outcomes of its products and services multiple authors recommend client satisfaction as an important outcome measure for assistive technology (1-5). However, issues relating to defining and measuring satisfaction are muddled. As part of its comprehensive needs assessment, the ATOMS Project performed a literature review of satisfaction in health care, and specifically, satisfaction with AT devices and services. This paper presents part of this investigation: an analysis of 44 published studies that specifically examined AT consumer satisfaction.
We searched eight databases (CINAHL, Medline, ERIC, Psych Info, ABI Inform, Business Source Elite, Academic Search, and Humanities Full Text) with combinations of the key words, “assistive technology”, “technology”, ‘self-help devices” “consumer”, “equipment”, “satisfaction”, “satisfaction survey”, “benefit”, and “disadvantage.” The search resulted in 1193 articles.
We selected articles based on the following criteria: 1) the articles measured patient, product or consumer satisfaction, 2) they discussed advantages and disadvantages of measuring satisfaction, or, 3) they focused on measurement issues related to satisfaction. The articles were then sorted by three specific categories (domain, product type, and purpose of article) with fifteen subcategories. These categories best reflected the wide domain of articles selected from fields of medicine, rehabilitation, nursing, business, and management and are listed below.
This paper details the analysis of the 44 research studies that focused specifically on satisfaction with AT devices and/or services. They are classified by methodology.
Seven of the studies employed the QUEST (Quebec User Evaluation of Satisfaction with Assistive Technology) an established 12-item outcome measure that assesses user satisfaction with devices and services (6). The authors define satisfaction “as a person's critical evaluation of several aspects of a device” (p.101) with two underlying dimensions: devices (8 characteristics) and services (4 characteristics). These papers used different versions of the instrument as it was under development. Important to note here is that in three of these studies the researchers report that they added patient report responses to elucidate ratings.
Twenty of the studies were easy to classify by methodology or scaling. Two studies were qualitative, nine studies scored satisfaction on a dichotomous scale, and 10 studies used ranked (e g, Likert-like or visual analog) scaling. In these studies the satisfaction variable/s were usually defined in one of two ways. Commonly, device (or service) characteristics were listed with respondents either ranking the characteristic on degree of satisfaction or they were specifically asked, “are you satisfied with…” for a yes/no response.
| Author | Article & confusing language | |
|---|---|---|
| Neilson, et al. (2001) | Measuring the effects of seating on people with profound and multiple disabilities - a preliminary study | |
| Considered 8 "Satisfaction dimensions" where "overall satisfaction" and "overal outcome" are two of these 8 variables. Used a visual analog scale. | ||
| Additionally, whether or not they would undergo the procedure again was considered a measure of satisfaction. | ||
| Suzuki & Lockette (2000) | Client satisfaction survey of a wheelchair seating clinic | |
| Considered 12 variables for satisfaction, two of which were "satisfied with comfort/mobility of equipment" "satisfied w/ cost-value of equipment" | ||
| The other 10 items of their "Satisfaction Survey" did not use the word satisfaction but Likert responses to all 12 questions were, "from 1 = not satisfied or not happy to 5 = very satisfied or very happy" | ||
| Used 5 pt, scale plus open-ended | ||
| Wuolle et al. (1999) | Satisfaction with & usage of a hand neuroprosthesis | |
| 1) The authors measured 6 "general satisfaction" variables where "I am satisfied with my NP" was one of these. | ||
| 2) The others variables were: "recommend to others", "reliable", "have again", "met my expectations", "I would pay for if I had the money" were considered satisfaction measures. | ||
| 3) 5 pt scale, strongly agree to strongly disagree | ||
Beyond this, the measurement of satisfaction in the remaining 17 studies (39%) becomes very muddled and demonstrates the significant problem that AT researchers have in defining satisfaction to measure. Three of these studies (see Table 1) demonstrate an internal confusion as they recognize satisfaction as a multi-variable construct. In these instances the researchers created or used items that specifically ask questions with “satisfaction” in the wording but then use other variables that do not use the word.
| Author | Article and description of satisfaction variable(s) | |
|---|---|---|
| Chamberlain et al. (2001) | Equipment: is it the answer? An audit of equipment provision | |
| Asked "If someone you know had similar problems with everyday tasks, would you recommend our service to them?" for measure of satisfaction. | ||
| Chen LP et al. (1998) | An evaluation of reachers for use by older persons with disabilities | |
| Concluded satisfaction by "ranking the reachers on basis of experience and needs." Plus, asked to comment. | ||
| Chen T, et al. 2000 | Caregiver involvement in the use of assistive devices by frail older persons | |
| Reports a relationship between caregiver involvement and the use of satisfaction with assistive devices, but their survey does not have a specific “satisfaction” variable. | ||
| Dunn et al. (1998) | Follow-up assessment of standing mobility device users | |
| Asked quality of life questions plus open-ended questions, concluded concerns about lack of follow-up about satisfaction. | ||
| Feys et al (2001) | Assistive technology to improve PC interraction for people with intention tremor | |
| Satisfaction questions included two regarding ease of use, one about fatigue, and one about comfort | ||
| Finlayson & Havixbec (1992) | A post-discharge study on the use of assistive devices | |
| Satisfaction was measured by asking clients to comment on the adequacy of the instruction provided them. | ||
| Garber (2002) | Wheelchair utilization and satisfaction following cerebral vascular accident | |
| Rated six wheelchair characteristics where 1=poor and 5=excellent, and averaged the six scores to represent overall satisfaction, | ||
| Grishbrooke (2003) | Living with Lifts: a study of users' experiences | |
| Concluded satisfaction when, "all participants were satisfied that that the lift did what they wanted it to do". | ||
| Kohn & LeBlanc (1994) | Measuring quality and performance of assistive technology: results of a prospective monitoting program | |
| Concluded "a high level of performance and perceived satisfaction" with 93-95% device use at 7 months. | ||
| Mann et al. (1994) | Design of hand-held remotes for older persons with impairments | |
| "Subjects were asked which of the four devices they liked best for 10 features" (p.143) as measure of satisfaction. | ||
| McGrath et al. (1985) | Assistive devices: utilization by children | |
| Concluded "satisfaction with devices high" from utilization data. | ||
| Steele et al. (1989) | Reading aid technology for blind persons: Responses to a questionnaire of experienced users | |
| In abstract, mentioned one research question as satisfaction and then the word did not appear again in the article. Reported using a 160 item questionnaire. | ||
| Taylor et al. (2001) | Pattern of use & user satisfaction of Neuro Control Freehandsystem | |
| Authors considered that satisfaction consisted of: quick to use, easy to use, comfort and fit. | ||
| van Roosmalen et al. (2002) | Preliminary evaluation of wheelchair occupant restraint system usage in motor vehicles | |
| Authors considered that there were 3 satisfaction variables: comfort, fit and use | ||
Fourteen studies, however, have more significant terminology conflicts (See Table 2). These studies purport to quantify satisfaction but do not mention the word in their measures. For example, Taylor, Esnouf and Hobby (2001) report patient responses to a questionnaire in their article “Pattern of use and user satisfaction of Neuro Control Freehand system.” Their concern was with use and acceptability of the implanted functional electric stimulation device and they asked recipients of the system about reliability, system acceptability, confidence, and "general" questions: were they glad that they had the system, would they recommend the system, were they more independence, did they feel less of a burden, more confident, and whether they felt that the implant had improved their quality of life. The word “satisfaction” never appears after the title(7). The authors conclude that the system provides increased function that is considered to be of benefit. Similarly, Steele et al surveyed blind users of reading aid technology with the purpose of the questionnaire to discover the “ range of current application of the devices and degree of satisfaction with them” (1989, p. 23) (italics added). They measured application, specific likes and dislikes, attitudes toward ancillary services, and the user's perceptions of the impact of their reading aids on their professional and personal lives (8). They never mention satisfaction again. They table summarizes the similar problems in all 14 studies.
The medical literature is replete with discussions of methodological confounds as the medical field has attempted to measure patient satisfaction for their various purposes. A brief list of some of these salient concerns follows.
In the 44 specific AT satisfaction studies reviewed, 2 were qualitative, and of the remaining 42, 24 or 57% reported adding open-ended, comments or patient report responses to their quantifiable questions. Most AT researchers are clinicians and it is argued that the intuitive component of clinical practice affects our judgment as we struggle to quantify patient satisfaction as an AT outcome variable.
While the methods of science are fairly rigorous and standardized, the application of the scientific method is not. Each researcher must decide subjectively what he or she is going to study, what variables are relevant and therefore controlled, what instruments should be used to measure a phenomenon, and how to interpret the results. In this area of study that examines the sub-domain of AT satisfaction there are almost as many definitions of AT satisfaction as there are studies. . As such, we have to be extremely careful how we use and interpret these studies. Each must be examined for its exact perspective as there are very few common phenomena.
This project is supported in part by the National Institute on Disability and Rehabilitation Research, grant number H133A010403. The opinions contained in this paper are those of the grantee and do not necessarily reflect those of the NIDRR and the U.S. Department of Education.