ABSTRACT

The MED-AUDIT (Medical Equipment Device – Accessibility and Universal Design Information Tool) prototype demonstrates the potentials of this measurement approach to assess the accessibility of medical devices.  This study investigated the inter-rater reliability of the MED-AUDIT. It was hypothesized that MED-AUDIT would demonstrate good to excellent interrater reliability across five different models of blood pressure monitors, one manual inflate arm monitor, two automatic arm monitors, and two automatic wrist monitors when scored on the MED-AUDIT by 3 different raters. Intraclass correlation coefficients were calculated across the three raters for 1) the raw scores of the top two parent levels of questions in the taxonomy and 2) summary accessibility scores for 13 impairments. Results showed the MED-AUDIT demonstrated good to excellent interrater reliability for all devices, except that it was moderate for raw parent questions for the automatic arm monitor and was poor for the 13 impairment categories for the manual blood pressure monitor. This study demonstrates that the MED-AUDIT evaluation methodology is feasible and reliable, however requires further testing.

INTRODUCTION

In the past few decades there has been a dramatic increase in the survival rate of individuals with severe disabilities and an increased demand for medical devices to be used in patient's homes, with the patient as the end user [1-5]. However, tremendous disparities in health care access prevent these populations from receiving treatment in a timely manner, primarily due to inaccessible medical technologies [6-8]. For example, to monitor their blood pressure, individuals must be able use their upper extremities to get the cuff on themselves and to see or hear the display; or individuals must be able to stand to get a mammogram; or must have adequate strength to climb and balance on an examination table. This translates into hundreds of millions of dollars spent by clinical professionals, caregivers, and lay people to purchase medical devices that may not work for a large majority of the patients they are intended for, thereby creating a large disparity in healthcare access [9-11]. 

Three factors exacerbate the problem and highlight the push towards accessible medical devices: (1) the rapid aging of America, (2) increased survival rate for people with disabilities (PWD), and (3) increase in use of home health equipment used and purchased directly by patients [12-14]. However, there is no assessment that measures the accessibility of medical devices. The MED-AUDIT is being developed to meet this need. The MEDAUDIT is a software-based assessment that performs a task analysis, assesses design features, addresses information for specific impairments, and calculates an integrated accessibility score for medical devices. The questions on task analysis and design features comprise the taxonomy of the MED-AUDIT. It includes two background matrices that represent relationships between impairments and device features and device features and tasks which create a database map to produce accessibility scores for thirteen different impairments.

Past studies have reported on aspects of the development, and satisfactory usability, feasibility, and preliminary psychometric properties of the MED-AUDIT [15]. It has already been established that MED-AUDIT distinguishes between accessibility levels for different devices [16], however the inter-rater reliability of the MED-AUDIT across real devices across multiple raters has not been investigated. Therefore, the research question explored in this paper is, "Does the MED-AUDIT taxonomy demonstrate inter-rater reliability across five different models of blood pressure monitors across three raters"?

METHODS

This study was designed to determine the inter-rater reliability of the MED-AUDIT taxonomy across five different models of blood pressure monitors (BPM), when scored individually by 3 trained raters (See Figure 1). The five BPM rated in this study consisted of a manual inflate blood pressure monitor, two automatic arm monitor, and two automatic wrist monitors. The BPM were chosen to represent the range of blood pressure monitoring devices available as well as to represent different levels of accessibility.

All five BPM were scored on the same version of the MED-AUDIT, which includes about 1150 distinct questions arranged in a hierarchal outline broken down into five or six levels for the taxonomy. The outline structure provides the branching options from level to level. The branching enables irrelevant parts of the taxonomy for a certain device to be bypassed, making questions targeted and requiring less expertise (See Figure 2). The taxonomy includes 193 questions related to tasks required to use the medical device and are scored on a three-point scale (0-does not require, 1somewhat requires, and 2-requires). It also includes 957 questions related to features included in the device and are scored on a 3-point scale (0-does not include, 1somewhat includes, and 2-includes).

Three raters unpackaged and interacted with each device prior to scoring it. Each device was scored independently before moving onto the next device. On completion of scoring, the data from the software was exported to an Excel sheet which provided a raw score and a percentage score for each question. We also obtained summary accessibility scores across 13 impairment categories: hard of hearing, deaf, low vision, blind, expressive communication, comprehensive disorders, other cognitive disorders, behavioral impairment, sensitivity impairment, lower limb impairment, upper limb impairment, head/neck/trunk impairment, and systemic body impairment. Intraclass correlation coefficients were calculated across the three raters for each device for 1) the raw scores of the top two parent level questions in the taxonomy and 2) summary accessibility scores for the 13 impairments. ICC is interpreted as >0.90 = excellent reliability, 0.75-0.9 = good reliability, 0.5-0.75 = moderate reliability, and <0.5 = poor reliability [17]. In addition, Gwet's AC1 was used to assess agreement on the smallest child level questions for a more robust evaluation of interrater reliability.

RESULTS

Table 1: Intraclass Correlation Coefficients Across Tasks and Device Features

Manual Blood Pressure Monitor	0.844
Automatic Arm Blood Pressure Monitor	0.565
Automatic Wrist Blood Pressure Monitor	0.957
Deluxe Arm Blood Pressure Monitor	0.922
Deluxe Wrist Blood Pressure Monitor	0.825

The raters took between 45 minutes to an hour to score each of the five devices, including unpackaging the device, reviewing instructions, interacting with the device, and rating the MEDAUDIT for the device. The rater reported no difficulties using the MED-AUDIT. Table 1 highlights the results for intraclass correlation coefficients for the raw scores across the top two parent levels of questions of the MED-AUDIT taxonomy for all five blood pressure monitors. The scores for the Automatic wrist (ICC = 0.957) and Deluxe arm monitor (ICC = 0.922) were excellent, for the Manual (ICC = 0.844) and Deluxe wrist monitor (ICC = 0.957) were good, and for the Automatic arm monitor was moderate (ICC = 0.565). 

Table 2: Intraclass Correlation Coefficients Across 13 Impairment Categories

Manual Blood Pressure Monitor	0.311
Automatic Arm Blood Pressure Monitor	0.928
Automatic Wrist Blood Pressure Monitor	0.978
Deluxe Arm Blood Pressure Monitor	0.949
Deluxe Wrist Blood Pressure Monitor	0.966

Table 2 demonstrates the results for the ICC across the three raters across all 13 impairments for the five blood pressure monitors. The ICC's were excellent for the Automatic arm (ICC = 0.928), the Automatic wrist (ICC = 0.978), the Deluxe arm (ICC = 0.949), and the Deluxe wrist (ICC = 0.966) monitors but was poor for the Manual monitor (ICC = 0.311). Figure 3 presents line graphs of the three raters scores for the five devices across the 13 impairments. The graphs highlight that ratings for the manual blood pressure monitor were varied across the 3 raters compared to the other four monitors.

Table 3 shows Gwet's AC1 agreement coefficients at the level of the lowest level of child questions. The responses to these questions are directly provided by the raters. AC1 coefficients were good for Automatic Arm Blood Pressure Monitor (AC1 = 0.816), Automatic Wrist Blood Pressure Monitor (AC1 = 0.848), Deluxe Arm Blood Pressure Monitor (AC1 = 0.822), and Deluxe Wrist Blood Pressure Monitor (AC1 = 0.860). This coefficient shows poor reliability for the Manual Blood Pressure Monitor (AC1 = 0.254).

DISCUSSION

Results from the ICC and AC1 analyses across three raters show that the coefficients were good to excellent for all devices, except that it was moderate for the tasks and device features for the Automatic arm monitor and poor across accessibility scores for the 13 impairment categories as well as for child level analyses for the manual blood pressure monitor. In looking at the graphs and scores across raters, although a majority of analysis demonstrate good to excellent reliability, we would like to establish excellent reliability irrespective of the complexity of the device. The results highlight a need to conduct a further detailed analysis of the data at the question level to determine which areas led to differences between rater scores. We will also evaluate the question descriptions to determine if they need additional detail to clarify the concepts. Last, we will investigate and evaluate the matrices and algorithms that generate accessibility scores to determine if they need revision.

Table 3: Gwet's AC1 Agreement Coefficients Across Smallest Child Level Questions

Manual Blood Pressure Monitor	0.254
Automatic Arm Blood Pressure Monitor	0.816
Automatic Wrist Blood Pressure Monitor	0.848
Deluxe Arm Blood Pressure Monitor	0.822
Deluxe Wrist Blood Pressure Monitor	0.860

This study suggests some future steps for continuing to establish the reliability and validity of the MED-AUDIT: 1) test different types of medical devices that cover the spectrum of diagnostic, monitoring, positioning, and maintenance devices, and 2) re-evaluate the questions and complete a detailed analysis of the appropriateness and accuracy of questions towards measuring medical device accessibility. 

Establishing the reliability and validity of the MED-AUDIT scores will provide insights into how to provide this information to designers as well as consumers of medical devices and may have labeling implications for medical devices. Providing consumers with accessibility information will improve medical device design and healthcare for consumers with disabilities.