Introduction

Building accessibility is a concept that is commonly interpreted as the ability of people with physical disabilities to access their environment, and not often considered by "abled body" individuals. The creation of the Americans with Disabilities Act (ADA) in 1990 and update in 2010 (ADA-ABA) enabled thousands of people with disabilities (PWD) to gain access to formerly inaccessible public buildings [1]. However, the ADA Accessibility Guidelines (ADAAG) is still limited for the following reasons: a) the ADA provides only absolute minimal guidelines for building features; b) older buildings are exempt from ADAAG; and, c) accessibility is relative based on a person's functional capabilities [2,3].

In order to empower PWD to engage and participate within their communities, it is essential to comprehensively assess the accessibility of public buildings and address any identified barriers. Provide sufficient information regarding the accessibility details of the buildings that can be tailored to their impairments and functional capabilities is also imperative. A basic problem in research on accessibility is the paucity of reliable methods for assessing and analyzing accessibility problems [4]. Additionally, despite the fact that many assessment tools have been developed to provide and assess building accessibility, the majority of the available tools focus on the physical aspect of accessibility and do not consider accessibility requirements for individuals that may have cognitive, visual, or auditory impairments.  

The Access Ratings for Buildings (AR-B) project provides two core apps, AccessPlace and AccessTools, aiming to provide a comprehensive accessibility assessment tool and establish an informational resource for the accessibility of public buildings. AccessPlace is a web-based app which aims to elicit and document the experiences of PWD while using community buildings and provides Personalized Accessibility Information (PAI) by prioritizing comments from individuals with similar conditions. For example, a blind person would find information about building signage ranking higher than a comment about door width populated by a person using a wheelchair. AccessTools is an iOS iPad app that is designed to be used by trained assessors to document the details of building elements and helps measure the complete accessibility of different building elements.

Accessibility is an issue embracing all public buildings in society and is of vital importance to PWD' societal participation, this paper is delimited to restaurants. This study introduces preliminary findings that test the inter-rater reliability (IRR) psychometric property between trained and not trained student raters while using the AccessTools app.

Methods

AccessTools App

The AccessTools app is designed to be used by professional raters to assess, document, and quantify the accessibility of different building elements. The assessment is comprised of 10 elements: Parking & Exterior Signage, Main Entrance/Exterior Doorway(s), Other Entrance(s)/Emergency Exit(s), Reception & General Information, Indoor Routes, Stairs & Elevator(s), Seating, Restroom(s), Other Interior Doorway(s), and Specialty Features (Restaurants). Figure 1 displays a screenshot of the current AccessTools app. Each of these basic building elements branches into dozens of sub-elements like audio levels, widths, heights, etc.

In total, the assessment contains 1180 questions. In order to make the assessment more efficient, the AccessTools software utilizes a branching question system to elicit accessibility scores of each element and sub-element. Presented as Trichotomous Tailored, Sub-branching Scoring (TTSS) [5], the software asks if a building feature is accessible ('[2] Fully Accessible / Yes', '[1] Somewhat Accessible', '[0] Not Accessible / No', '[X] Not Applicable') and uses skip logic to auto-advance through hundreds of questions about the building to optimize user efficiency. Figure 2 shows the expanding outline and scoring screen.  

The app provides three embedded mini-apps to assess specific building elements to reduce the time needed to perform the assessment. AccessSlope, AccessSound, and AccessRuler enable an assessor to use the iPad sensors to quickly measure inclines, decibels and distances in the context of accessibility, removing the need for separate tools, such as a tape measure, level, and clipboard. The laser ruler and included mini-apps upload the measurements right in context in the AccessTools app. Additionally, AccessTools enables assessors to take photos and videos of specific building elements.

Procedure

In this pilot study, 32 second year master occupational therapy students participated in the data collection. Students were assigned to two groups, based on there previous exposure to using the AccessTools app. Group 1 received general accessibility training via an online module, and group 2 did not receive the training before data collection. The training module was created as a part of an ongoing study aiming to assess the effect of training on using the AccessTools app as a collaborative effort between University of Wisconsin-Milwaukee, Florida International University, Texas Woman's University, and Columbia University. Students were asked to work in pairs and each pair to assess two restaurants. One pair from each group were assigned to assess the same restaurant. Thus, we had a total of 16 restaurants and each restaurant was assessed twice by a pair from group 1 and a pair from group 2. In each assessment, the following four elements were assessed using the app: Main Entrance/Exterior Doorways, Indoor Routes, Seating, and Restrooms.

Results

The assessments from 7 restaurants were excluded from the analysis due to incompleteness. For the remaining 9 restaurants, the final score of each of the 4 elements was normalized and the interclass correlation coefficient (ICC) was calculated using SPSS (IBM Corp). The ICC calculation resulted in a poor reliability (ICC (1,1) = 0.402) [6]. To further investigate the agreement between raters, Cohen's kappa statistic (K) was calculated for each restaurant using Excel. The average K score from the nine restaurants resulted in a poor to fair agreement [7]. Table 1 displays the K statistic results. One of the differences that can explain the difference in the ICC and the K statistic is that the K statistic computation counts for the N/A answers, while the normalized scores at the main element level do not include these answers. Looking more closely at the data, there were several discrepancies between raters in using "N/A" vs. "Not Accessible / No" answers. Recalculating the K statistic after excluding the questions that one rater answered as "N/A" while the second rater answered as "Not Accessible / No", resulted in increasing the agreement between all raters (Table 1).

Discussion

Table 1: Kappa statistic

Restaurant	K with all Observations	K without N/A
1	0.02	0.20
2	0.31	0.55
3	0.07	0.09
4	0.58	0.52
5	0.17	0.17
6	0.25	0.42
7	0.24	0.59
8	0.25	0.32
9	0.64	0.71
Average	0.28	0.40

Although the IRR analysis of the AccessTools app in this preliminary study resulted in a poor to fair agreement between raters, it shed some light on important findings. The poor agreements between raters were expected as this assessment tool is designed to be used by professional trained raters. However, analyzing the IRR between the raters highlighted a major discrepancy between the interpretation of "N/A" and "Not Accessible / No" score when a feature of the building was not present, which led to a considerable impact on the IRR score. For example, a question may ask about the accessibility of an elevator while there is no elevator in the building. A rater may select "N/A" because no elevator is present.  However, another rater may select "Not Accessible / No" because an elevator is required for the building to be accessible.

While the TTSS system provides an efficient scoring process, there is potential for erroneous scores. For instance, when scoring a parent element as "Fully Accessible / Yes", "Not Accessible / No", or "NA" all sub-branching questions under this parent is assigned with the same score. Without having a full understanding of the parent question, this limits the provision of more detailed information in the sub-branching questions. Thus, a rater may under- or overrate the accessibility of building elements.  

These findings highlight the importance of adding more clarity to the app scoring criteria and the training instructions. Although this study appears as if it was intended to establish the IRR of AccessTools, it identified some important aspects that need to be addressed to strengthen the content validity of the app.