RESNA Annual Conference - 2020

Qussai M. Obiedat, Roger O. Smith

Rehabilitation Research Design & Disability (R2D2) Center, University of Wisconsin-Milwaukee


Accessibility to public buildings is of societal importance for all individuals, especially people with disabilities. Despite the availability of several accessibility assessment tools, there is still paucity in valid and reliable comprehensive assessments addressing accessibility needs of individuals with impairments besides physical impairments. Additionally, such assessments are lengthy and time consuming. AccessTools app is a comprehensive assessment tool designed to be used by trained assessors to identify, document, and measure the complete accessibility of different building elements. The app utilizes a branching question system using a Trichotomous Tailored, Sub-branching Scoring system to increase its efficiency.

This study introduces preliminary findings from testing inter-rater reliability (IRR) property between trained and not trained raters using the AccessTools assessment in nine restaurants. The analysis resulted in a poor reliability between raters (ICC (1,1) = 0.402) and a poor to fair agreement between raters, which was expected as the tool is intended for trained raters. Performing the ICC and Kappa agreement analysis on the results pointed out the discrepancies in using the "Not Applicable" score between raters, highlighting a need for better instruction and training and improving the overall content validity of the assessment.


Building accessibility is a concept that is commonly interpreted as the ability of people with physical disabilities to access their environment, and not often considered by "abled body" individuals. The creation of the Americans with Disabilities Act (ADA) in 1990 and update in 2010 (ADA-ABA) enabled thousands of people with disabilities (PWD) to gain access to formerly inaccessible public buildings [1]. However, the ADA Accessibility Guidelines (ADAAG) is still limited for the following reasons: a) the ADA provides only absolute minimal guidelines for building features; b) older buildings are exempt from ADAAG; and, c) accessibility is relative based on a person's functional capabilities [2,3].

In order to empower PWD to engage and participate within their communities, it is essential to comprehensively assess the accessibility of public buildings and address any identified barriers. Provide sufficient information regarding the accessibility details of the buildings that can be tailored to their impairments and functional capabilities is also imperative. A basic problem in research on accessibility is the paucity of reliable methods for assessing and analyzing accessibility problems [4]. Additionally, despite the fact that many assessment tools have been developed to provide and assess building accessibility, the majority of the available tools focus on the physical aspect of accessibility and do not consider accessibility requirements for individuals that may have cognitive, visual, or auditory impairments.  

The Access Ratings for Buildings (AR-B) project provides two core apps, AccessPlace and AccessTools, aiming to provide a comprehensive accessibility assessment tool and establish an informational resource for the accessibility of public buildings. AccessPlace is a web-based app which aims to elicit and document the experiences of PWD while using community buildings and provides Personalized Accessibility Information (PAI) by prioritizing comments from individuals with similar conditions. For example, a blind person would find information about building signage ranking higher than a comment about door width populated by a person using a wheelchair. AccessTools is an iOS iPad app that is designed to be used by trained assessors to document the details of building elements and helps measure the complete accessibility of different building elements.

Accessibility is an issue embracing all public buildings in society and is of vital importance to PWD' societal participation, this paper is delimited to restaurants. This study introduces preliminary findings that test the inter-rater reliability (IRR) psychometric property between trained and not trained student raters while using the AccessTools app.


AccessTools App

A screenshot of the main AccessTool assessment user interface is displayed in order to highlight the 10 main elements of the assessment. These elements include Parking & Exterior Signage, Main Entrance/Exterior Doorway(s), Other Entrance(s)/Emergency Exit(s), Reception & General Information, Indoor Routes, Stairs & Elevator(s), Seating, Restroom(s), Other Interior Doorway(s), and Specialty Features (Restaurants).
Figure 1. AccessTool app elements

The AccessTools app is designed to be used by professional raters to assess, document, and quantify the accessibility of different building elements. The assessment is comprised of 10 elements: Parking & Exterior Signage, Main Entrance/Exterior Doorway(s), Other Entrance(s)/Emergency Exit(s), Reception & General Information, Indoor Routes, Stairs & Elevator(s), Seating, Restroom(s), Other Interior Doorway(s), and Specialty Features (Restaurants). Figure 1 displays a screenshot of the current AccessTools app. Each of these basic building elements branches into dozens of sub-elements like audio levels, widths, heights, etc.

In total, the assessment contains 1180 questions. In order to make the assessment more efficient, the AccessTools software utilizes a branching question system to elicit accessibility scores of each element and sub-element. Presented as Trichotomous Tailored, Sub-branching Scoring (TTSS) [5], the software asks if a building feature is accessible ('[2] Fully Accessible / Yes', '[1] Somewhat Accessible', '[0] Not Accessible / No', '[X] Not Applicable') and uses skip logic to auto-advance through hundreds of questions about the building to optimize user efficiency. Figure 2 shows the expanding outline and scoring screen.  

The app provides three embedded mini-apps to assess specific building elements to reduce the time needed to perform the assessment. AccessSlope, AccessSound, and AccessRuler enable an assessor to use the iPad sensors to quickly measure inclines, decibels and distances in the context of accessibility, removing the need for separate tools, such as a tape measure, level, and clipboard. The laser ruler and included mini-apps upload the measurements right in context in the AccessTools app. Additionally, AccessTools enables assessors to take photos and videos of specific building elements.


In this pilot study, 32 second year master occupational therapy students participated in the data collection. Students were assigned to two groups, based on there previous exposure to using the AccessTools app. Group 1 received general accessibility training via an online module, and group 2 did not receive the training before data collection. The training module was created as a part of an ongoing study aiming to assess the effect of training on using the AccessTools app as a collaborative effort between University of Wisconsin-Milwaukee, Florida International University, Texas Woman's University, and Columbia University. Students were asked to work in pairs and each pair to assess two restaurants. One pair from each group were assigned to assess the same restaurant. Thus, we had a total of 16 restaurants and each restaurant was assessed twice by a pair from group 1 and a pair from group 2. In each assessment, the following four elements were assessed using the app: Main Entrance/Exterior Doorways, Indoor Routes, Seating, and Restrooms.


A screenshot of the main AccessTool assessment user interface is displayed in order to provide an example of the branching question system that is used in the app. It also displays the Trichotomous Tailored, Sub-branching Scoring (TTSS) and the four available answers for all questions (‘[2] Fully Accessible / Yes’, ‘[1] Somewhat Accessible’, ‘[0] Not Accessible / No’, [X] Not Applicable).
Figure 2. AccessTool app branching question system

The assessments from 7 restaurants were excluded from the analysis due to incompleteness. For the remaining 9 restaurants, the final score of each of the 4 elements was normalized and the interclass correlation coefficient (ICC) was calculated using SPSS (IBM Corp). The ICC calculation resulted in a poor reliability (ICC (1,1) = 0.402) [6]. To further investigate the agreement between raters, Cohen's kappa statistic (K) was calculated for each restaurant using Excel. The average K score from the nine restaurants resulted in a poor to fair agreement [7]. Table 1 displays the K statistic results. One of the differences that can explain the difference in the ICC and the K statistic is that the K statistic computation counts for the N/A answers, while the normalized scores at the main element level do not include these answers. Looking more closely at the data, there were several discrepancies between raters in using "N/A" vs. "Not Accessible / No" answers. Recalculating the K statistic after excluding the questions that one rater answered as "N/A" while the second rater answered as "Not Accessible / No", resulted in increasing the agreement between all raters (Table 1).


Table 1: Kappa statistic
Restaurant K with all Observations K without N/A
1 0.02 0.20
2 0.31 0.55
3 0.07 0.09
4 0.58 0.52
5 0.17 0.17
6 0.25 0.42
7 0.24 0.59
8 0.25 0.32
9 0.64 0.71
Average 0.28 0.40

Although the IRR analysis of the AccessTools app in this preliminary study resulted in a poor to fair agreement between raters, it shed some light on important findings. The poor agreements between raters were expected as this assessment tool is designed to be used by professional trained raters. However, analyzing the IRR between the raters highlighted a major discrepancy between the interpretation of "N/A" and "Not Accessible / No" score when a feature of the building was not present, which led to a considerable impact on the IRR score. For example, a question may ask about the accessibility of an elevator while there is no elevator in the building. A rater may select "N/A" because no elevator is present.  However, another rater may select "Not Accessible / No" because an elevator is required for the building to be accessible.

While the TTSS system provides an efficient scoring process, there is potential for erroneous scores. For instance, when scoring a parent element as "Fully Accessible / Yes", "Not Accessible / No", or "NA" all sub-branching questions under this parent is assigned with the same score. Without having a full understanding of the parent question, this limits the provision of more detailed information in the sub-branching questions. Thus, a rater may under- or overrate the accessibility of building elements.  

These findings highlight the importance of adding more clarity to the app scoring criteria and the training instructions. Although this study appears as if it was intended to establish the IRR of AccessTools, it identified some important aspects that need to be addressed to strengthen the content validity of the app.


  1. United States. 2010 ADA standards for accessible design. 2010.
  2. Hammel J, Magasi S, Heinemann A, Gray DB, Stark S, Kisala P, et al. Environmental barriers and supports to everyday participation: a qualitative insider perspective from people with disabilities. Arch Phys Med Rehabil 2015;96(4):578-588.
  3. Iwarsson S, Ståhl A. Accessibility, usability and universal design—positioning and definition of concepts describing person-environment relationships. Disabil Rehabil 2003;25(2):57-66.
  4. Steinfeld E, Danford GS, Danford G. Enabling environments: Measuring the impact of environment on disability and rehabilitation. : Springer Science & Business Media; 1999.
  5. Smith RO. OTFACT: Multi-level performance-oriented software with an assistive technology outcomes assessment protocol. Technology and Disability 2002;14:133-139.
  6. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine 2016;15(2):155-163.
  7. Portney L, Watkins M. Foundations of clinical research: application to practice. 3rd ed. Upper Saddle River, N.J: Prentice Hall; 2009.


The work of the Access Rating for Buildings project was supported by a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research, NIDILRR grant number (NIDIRR) H133G100211. NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this work do not necessarily represent the policy of NIDILRR, ACL, HHS, and you should not assume endorsement by the Federal Government.