Expert Validation of the MED-AUDIT (Medical Equipment Device-Accessibility & Universal Design Information Tool)

Rochelle Mendonca, M.S., OT, Roger O. Smith, Ph.D., OT, Melissa Lemke, MS, Jack Winters, Ph.D.

ABSTRACT

This paper reports on two expert studies run to establish key validity components of the MED-AUDIT “Novice User Version” (NUV). Early exploratory analysis aimed to determine aspects of reliability and validity by examining the ability of the MED-AUDIT to establish a hierarchy of accessibility for medical devices, content validity, inter-rater reliability and validity of the scoring algorithm. Results revealed that MED-AUDIT scores match experts’ subjective perceptions of accessibility establishing a key aspect of validity. Additionally, content validity indices showed that questions were comprehensive, however, the wording of some questions and definitions required revision.  Inter-rater reliability was strong. Expert groups focusing on the scoring structure and impairment categories determined that the MED-AUDIT has developed a novel approach toward measuring accessibility quantitatively and the instrument generates scores that are valid. All feedback from the field experts is currently being applied to refine the MED-AUDIT and in the design of future studies.

Keywords

Measurement, experts, accessibility, MED-AUDIT, validity

Background/Statement of problem/research question

Large proportions of individuals with functional limitations are significantly challenged by many medical devices in their roles as recipients or providers of healthcare.1 For example, to obtain proper diagnoses, individuals often must stand (e.g. mammography); or to obtain complete medical exams clients must have adequate strength to climb and balance on an exam table.2  These issues emphasize the need for accessible medical devices for individuals with disabilities. However, there is no assessment that measures the accessibility of medical devices. The MED-AUDIT (Medical Equipment Device-Accessibility and Universal Design Information Tool) is being developed to meet this need. The MED-AUDIT is a software based assessment that performs a task analysis, assesses design features, addresses information for specific impairments, and calculates an integrated accessibility score for medical devices.

Two versions of the MED-AUDIT have been developed to quantitatively assess the accessibility of medical devices: the Novice User Version (NUV) and the Expert User Version (EUV). The NUV targets designers who may have extensive knowledge about medical device features and tasks, but may not know what features improve end-user accessibility. The NUV includes two background matrices that contain relationships between impairments and device features and device features and tasks which create a database map to produce accessibility scores for thirteen different impairments. The EUV targets experts in accessibility and design who have expertise to identify how device features and tasks impact individuals with functional limitations, and therefore does not use the pre-mapped database.3 Past studies have reported on aspects of the development, usability and psychometric properties of both versions of the MED-AUDIT. Preliminary usability, reliability and validity have been satisfactorily demonstrated for both versions4, but the use of experts and their contributions to NUV development has not been reported.

An integral part of the development of any measurement instrument is review and feedback from experts. Expert contribution toward the development of assessment instruments range from determining items to include in the assessment, validation of items included, validation of scoring procedures, and testing the assessment.5,6 This paper reports on two expert studies designed to help establish the reliability and validity of the NUV. The NUV was designed for individuals with limited knowledge in accessibility and disability. Therefore, the success of the NUV depends on embedding expert knowledge into its quantification scheme. Experts in the fields of accessibility, disability, and ergonomics provided feedback on the content of the instrument, the impairment categories, and accessibility scores generated for two different models of a blood pressure monitor. The main questions explored in this paper are:

  1. Ability of the NUV to predict the hierarchy of accessibility for two models of a blood pressure monitor based on experts’ scores,
  2. Comprehensiveness and clarity of content,
  3. Inter-rater reliability of scores obtained from the NUV,
  4. Delineation of impairment categories and
  5. Effectiveness of the scoring algorithm.

Methods/Approach/solutions considered

Two expert validation studies (Phase I & II) were conducted two years apart during the development of the MED-AUDIT. In Phase I, four experts in the fields of accessibility, universal design and disability were recruited. The experts in this study did not use the MED-AUDIT software, but examined the NUV taxonomy on MS Excel spreadsheets. Each expert rated the accessibility of two models of a blood pressure monitor: a manual and an automatic for thirteen functional impairments. On completion of the ratings, their scores were compiled and they participated in a phone conference to reach a consensus accessibility score for each of the impairments for both monitors. Experts provided feedback regarding the impairment categories. Specific discussion addressed the impairment taxonomy delineation.

In Phase II, four experts were recruited. Two of the experts provided continuity to the group and were the same as in Phase I.  Backgrounds of the experts met the same criteria as the earlier phase. The experts assessed the accessibility of two models of a blood pressure monitor; a manual and an automatic on the NUV.  Prior to scoring the NUV, each expert answered a pre-test questionnaire including demographic and universal design knowledge questions. Then they were provided instructions on the use of the software and time to interact with the NUV and the devices. Following this introduction to the task, each expert rated the two devices on the NUV. Lastly, participants answered a post-test questionnaire, which included questions about the validity and usability of the version they scored, and two questions to assess their perceptions of accessibility of the two monitors for 13 disabilities which was similar to Phase I. In Phase II, experts provided feedback regarding the content, the matrices, the scoring structure and the validity of scores generated from the NUV in focus groups. This paper reports the reliability and validity of the MED-AUDIT in terms of content, scoring components, consistency of scoring between experts and between the two phases, and the reliability of MED-AUDIT scores between experts.

Results

Phase I and II elicited and compared expert judgment scores as a 0-100% scale for each of thirteen impairments and as a composite expert consensus score. Additionally, Phase II included NUV scores using the embedded matrices and integrating algorithm. Two blood pressure monitors were assessed in each phase. Table 1 shows the expert consensus scores for each of the 13 impairments for both monitors.

For each impairment, the number in parenthesis reflects the ranking of the device. The number 1 implies the device with the lower accessibility score while number 2 is the device with a higher accessibility score. The rankings show that the manual blood pressure monitor was consistently rated as being less accessible compared to the automatic blood pressure monitor. In addition to assigning accessibility scores, experts reviewed the two taxonomies and provided feedback regarding the impairment categories. The category of “Overall body Impairment” was changed to “Systemic Body Impairment”. The definition and examples for the impairments were revised to provide additional clarity. In Phase II, experts assigned 0-100% accessibility scores to the thirteen impairments for the two monitors. Their scores were averaged to generate a single accessibility score for each impairment as shown in table 2.

The rankings in table 2 reflect that the manual blood pressure monitor was perceived as being less accessible than the automatic blood pressure monitor. These results helped establish the gold standard hierarchy of accessibility for the two devices being scored. Upon assigning subjective accessibility scores, experts scored the NUV for both blood pressure monitors and accessibility scores were generated for the thirteen different impairments. The raw scores from the NUV were normalized to obtain scores on a 0-100% scale. Table 3 shows the average of the expert NUV scores for both blood pressure monitors.

Based on the rankings obtained, the manual monitor was rated as being less accessible compared to the automatic monitor. Thirty-nine comparisons conducted for each of the accessibility hierarchy scores from all three data sets showed that the two devices were consistently ranked with the manual monitor being less accessible, indicating 100% validity for the hierarchy rating. The ranking of scores from the NUV matched experts’ ranked perceptions of the accessibility of the two devices.

In Phase II, experts scored a post-test questionnaire that included questions related to the content of the NUV. A validity index was calculated for each of the questions related to the content of the NUV by dividing the number of favorable responses by the total responses as shown in Table 4.

Table 4: This table displays the content validity indices for the questions related to the content of the NUV of the MED-AUDIT
Questions Total favorable responses Validity Index Number of neutral responses
The MED-AUDIT covers all tasks needed to access a medical device.
3
75%
1
The MED-AUDIT covers all device features needed to access a medical device.
4
100%
0
All questions were clear
2
50%
0
All the definitions were clear
2
50%
0
Responses were appropriate
3
75%
1
Definitions were easy to understand
2
50%
2

The validity index ranged from 50-100%. The experts indicated that the MED-AUDIT seemed comprehensive in terms of device features and tasks with indices of 75-100%, however, the wording of questions, definitions and response sets needed revision with these areas obtaining 50% indices. 

Inter-rater reliability of the experts’ NUV scores was assessed with intraclass correlation coefficients. Reliability of scores was 0.784 for the manual monitor and 0.981 for the automatic monitor that demonstrates excellent reliability and helps validate the experts as a “gold standard”. The experts reviewed the scores obtained from the NUV in a focus group to determine if the scores were valid for the two devices and were satisfied with the scores. They also reviewed the impairment categories to determine if there was any confusion in the categories divisions. They concluded that there needed to be additional examples to clarify the impairment categories. The experts finally reviewed the background expert system that generates the accessibility scores and were satisfied with the results.

Discussion

These expert studies contributed significant findings to the development of the “NUV” of the MED-AUDIT. The expert subjective perception of the accessibility of the two blood pressure monitors matched gross ranking of the results obtained from the NUV showing that the automatic monitor was more accessible compared to the manual monitor. This demonstrates preliminary construct validity of the scores and the scoring structure. Also, the experts were satisfied with the questions and response sets as demonstrated by the high validity indices with the exception of some specific issues related to wording of items. In response to these findings, all content questions that achieved indices below 75% are currently being revised.  These include wording of questions, definitions, and description of impairment categories. However, the high intra-class correlation coefficients of experts’ scores indicate that the assessment can be consistently scored across users.

The MED-AUDIT prototype instrument has begun to establish that we can measure accessibility of medical devices. These expert studies provide evidence that experts can be consistent and with some external validity may serve as a “gold standard” for future MED-AUDIT validation. Current and future MED-AUDIT research and development will continuously integrate feedback from experts.

Several instrumentation development breakthroughs surrounding the MED-AUDIT seem to be citable.  The include: a) Eliciting accessibility data through the simplicity of measuring tasks required and device features that do not require expertise in accessibility or disability, b) The division and reporting of scores for thirteen different impairment categories which provides designers a rich source of feedback about their designs related to universal accessibility, c) Using a database of expert derived scores of relationships to generate accessibility scores for designers without disability or accessibility expertise. 

References

  1. Mace, R. L. (1998). Removing barriers to health care: A guide for health professionals.   Retrieved January 11, 2005, from http://www.fpg.unc.edu/%7Encodh/RB-HealthCare.pdf
  2. Nosek, M. A. (2000). Overcoming the odds: the health of women with physical disabilities in the United States. Archives of Physical Medicine and Rehabilitation, 81, 135-138.
  3. Smith, R. O., Barnekow, K., Lemke, M. R., Mendonca, R., Winter, M., Schwanke, T., et al. (2007). Development of the MED-AUDIT (Medical Equipment Device-Accessibility and Universal Design Tool). In J. M. Winters & M. F. Story (Eds.), MEDICAL INSTRUMENTATION: Accessibility and Usability Considerations (pp. 283-296). Boca Raton: CRC Press.
  4. Mendonca, R., & Smith, R. O. (2007, June 15-19). Validity analysis: MED-AUDIT (Medical Equipment Device-Accessibility and Universal Design Information Tool). Paper presented at the RESNA 30th International Conference on Technology and Disability: Research, Design, Practice and Policy, Phoenix, AZ.
  5. Liu, K. P. Y., Lee, T., Yan, A., Siu, C. W. M., Choy, F. W. Y., Leung, K. L. K., et al. (2007). Use of the Interact Short Form as a tool to evaluate emotion of people with profound intellectual disabilities. Journal of Intellectual Disability Research, 51(II), 884-891.
  6. Routhier, F., Vincent, C., Desrosiers, J., Nadeau, S., & Guerette, C. (2004). Development of an obstacle course assessment of wheelchair user performance (OCAWUP): A content validity study. Technology and disability, 16, 19-31.

ACKNOWLEDGEMENTS

This work is supported in part by the National Institute on Disability and Rehabilitation Research, grant numbers H133A010403 and H133E020729. The opinions contained in this paper are those of the grantee and do not necessarily reflect those of the NIDRR and U.S. Department of Education. 

Author Contact Information:

Rochelle Mendonca, MS, OT, Rehabilitation Research Design and Disability (R2D2) Center, 2400 E. Hartford Ave., Enderis Hall, Room No. 135, Milwaukee, WI, 53211, Office Phone: (414)412-6803. E-mail: chs-mendonca@uwm.edu