Mobile Device Assessment Tool (MoDAT) to Capture Functional Capacity of Persons with Physical and Cognitive Disabilities
F. Indradhirmaya1, N. Monteiro2, L. Morris3, A. Fairman2, 3, A. Saptono1
1Department of Health Information Management, School of Health and Rehabilitation Science, University of Pittsburgh,
2Department of Occupational Therapy, College of Health and Wellness, Johnson and Wales University,
3Department of Rehabilitation Science and Technology, School of Health and Rehabilitation Science, University of Pittsburgh
INTRODUCTION
Smartphone devices have become more popular among persons with disabilities. In turn, one of the key challenges is to make the smartphone and its apps usable and accessible to a diverse range of users with various capabilities [1]. Even though there are universal usability issues that can affect any user, accessibility issues are often encountered by people with motor, cognitive, and/or sensory impairments [1]. Modern smartphones are equipped with a range of accessibility features that can be of great help to users with different needs and abilities. These features include auditory enhancements, visibility enhancements, and assistance with interaction and dexterity [2]. However, it's important to note that the specific accessibility options available on a smartphone can vary depending on the device and operating system in use.
Studies have demonstrated that assistive technology can enhance learning, increase accessibility for individuals with disabilities, and provide a means of addressing specific challenges related to disability [3]. Built-in accessibility features in smartphones have been designed to support individuals with disabilities [4, 5, 6]. Despite these features being available, people are often unaware of their existence and unlikely to discover them when they need them [2, 5, 6]. This lack of awareness could result in individuals with disabilities being excluded from using technology that could enhance their daily lives. Therefore, it is crucial to increase awareness of these built-in accessibility features and encourage their usage to improve accessibility and inclusion for people with disabilities.
There are tools that measure, give recommendations, and/or support the development and rehabilitation of various skills. For instance, Dexteria VMI [7] aims to assess and train visual-motor skills (e.g. visual figure-ground, visual-discrimination, copy-build) to improve handwriting among the younger population. Human Computer Interface (HCI) researchers have also developed tools that can assess a person's ability to perform isolated actions required for mobile device use, such as swipe, touch, and drag, and recommend appropriate accessibility settings based on their performance [4, 8]. Additionally, these researchers have created tools that can automatically enable or suggest specific accessibility settings based on a person's performance of everyday tasks on a mobile device [2, 9]. While these tools are helpful, it is worth mentioning that some of these tools were not designed for clinical use and are not publicly available.
Currently, no assessment tools on the market are available to measure the capacity of persons with disabilities to use a smartphone effectively and efficiently. Thus, determining what accessibility options can be recommended to improve individuals with disability's capacity to use smartphones is challenging because clinicians can only depend on their previous experience, which might be limited due to their background training. This highlights the need for a tool to measure and determine a person's ability, or lack of, in using a smartphone by having them do various tasks that mimic gestures, such as swiping, tapping, and dragging.
To bridge this gap, the Mobile Device Assessment Tool (MoDAT) is being developed as an assessment tool to evaluate a user's dexterity and cognitive ability while using a smartphone. This tool aims to replicate the smartphone environment and gather detailed data otherwise inaccessible to clinicians. MoDAT's assessments are also designed using the top-down approach, which is simulating a smartphone environment instead of developing multiple isolated tasks. Utilizing an occupation-based or top-down approach is more effective as it is more meaningful, relevant, and engages the user [10,11]. Additionally, this assessment tool adopts the mHealth model to assist both clinicians and users in discovering the optimal accessibility features for the users.
METHODOLOGY
Application
MoDAT is a software designed to display and measure individuals with disabilities' capacity while doing smartphone-related tasks, such as making a phone call and checking the weather from a weather app. The software aims to recommend accessibility features and modifications to improve the person's usability in using a smartphone. The software itself consists of the mobile app, portal, and backend.
MoDAT is designed to mimic how real smartphones operate closely. The developed tasks simulate the actual activity that the average smartphone user would perform. There are currently six tasks that are either developed or currently in development: Making a phone call, sending a text message, using an app to locate information, searching the internet, and downloading an app from the app store.
These tasks are carefully selected and designed to capture various gestures a smartphone user would perform. The tasks also range in difficulty and complexity. For instance, the phone call task only consists of single-tap actions. On the other hand, the internet search task demands multiple gestures for the user to complete. It also consists of more complex interfaces and cognitive load, such as typing on a keyboard, typing related search parameters, and selecting the best search results. By having tasks that range in complexity and require different gestures to complete, the data will show the extent to which a user can effectively and efficiently use a smartphone and identify any difficulties they may have encountered completing any simulated tasks.
System design
MoDAT is comprised of a mobile app for measuring individuals with disabilities' capacity, and a web portal for clinicians. The mobile app allows the assessment of individuals with disabilities capacities via simulated tasks and provides feedback through results review. The portal provides clinicians with an overview of participant progress and data visualization. The link between the app and the portal is supported by a backend platform that handles data storage, retrieval, processing, and communication between the mobile app and portal.
The mobile app is a mobile application installed on the user's smartphone. The app aims to track and simulate the typical smartphone user's various activities. The simulated activities are populated by the clinician or the researcher through the portal before the user's trials. The app can capture gestures such as taps, keyboard presses, and swipes. Furthermore, it can also capture various timestamps of actions, determine which component the user interacts with, track how long it takes to complete tasks, and capture the coordinates of various gestures. The user will be able to review their performance on all the simulations they performed with data presented in a client-friendly interface.
The portal is a web-based application that can be accessed from anywhere with an internet connection through most browsers. Clinicians and researchers can manage participants, add, and edit simulations assigned to a participant, and analyze results through an intuitive interface. The analysis page also consists of a visual for each task the participant performed, along with a mapping of where the participant performed any gestures. The data provided will aid the clinician in making recommendations to apply accessibility features or use of assistive technology to improve the users' performance.
The backend is a server-side component that connects the mobile app with the portal. It plays a crucial role in enabling the user to interact with the system and access the features and functionality that the system provides. This portion of the software is responsible for receiving, process, returning, validating, and storing various data from the mobile app and the portal.
Both front-end portion of the application is built with Angular, Typescript, HTML, and CSS. However, the mobile application uses Ionic, a cross-platform software development kit, to run the application natively on Android and iOS without having to write two different codebases. The backend, written in the Java language, is used to process and serve data generated from the two front-end applications. The system utilizes MySQL database. The participants have access to the mobile application, while the clinicians and researchers have access to the portal application. When the participants, clinicians, or researchers interact with either application, the data is sent to the backend to be processed and displayed on each application. The backend acts as a hub for consistent communication between the two applications.
RESULTS
One of the available assessments is the phone call assessment, in which the participant is given a phone number and must complete the task in the simulator by navigating through the interface to start the calling process and enter the given phone number as if they are making a call. Figure 2 shows the analytics page of the web portal, which shows the coordinates and visuals of where the participant taps on the screen. Any incorrect taps or missed buttons would be highlighted to show potential situations that can be addressed via accessibility options or strategy.
The research team has also conducted a focus group study with the consumer advisory group that was formed at the beginning of the research process to gather feedback early on. The group consists of a variety of people with a range of capacities in using a smartphone and specific needs. One major feedback that most people suggested would be to make the tasks compatible with voice accessibility features. This change may not affect the visual aspect of the mobile application very much; however, there were some significant changes to the semantics of the HTML that was written for each of the tasks to make sure they are accessible by the voice accessibility feature.
DISCUSSION
The research team has identified additional challenges that remain unaddressed in the current prototype of the MoDAT. A significant issue users report is the difficulty in using their upper dexterity to operate their smartphones. As a result, many participants resort to relying on the voice assistant feature to navigate and use their devices more efficiently. However, using voice assistants also presents challenges, such as misinterpretation of commands leading to unintended actions.
To improve the user experience for those who rely on voice assistants, it is essential to incorporate this feature into the MoDAT mobile application. Since Ionic is used to develop the application, properly identifying HTML semantics is necessary to ensure the voice assistant software locates and executes user commands accurately. By implementing this feature, developers can gain valuable insights into how to enhance the user experience for voice assistant users, making the MoDAT mobile application more accessible and user-friendly.
REFERENCES
[1] Badshah, S., Khan, A. A., Hussain, S., & Khan, B. (2021). What users really think about the usability of smartphone applications: diversity based empirical investigation. Multimedia Tools and Applications, 80, 9177-9207.
[2] Wu, J., Reyes, G., White, S. C., Zhang, X., & Bigham, J. P. (2020, October). Towards recommending accessibility features on mobile devices. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1-3).
[3] Igual, R., Plaza, I., MartÃn, L., Corbalan, M., & Medrano, C. (2013). Guidelines to design smartphone applications for people with intellectual disability: A practical experience. In Ambient Intelligence-Software and Applications: 4th International Symposium on Ambient Intelligence (ISAmI 2013 (pp. 65-69). Springer International Publishing.
[4] Peng, Y. H., Lin, M. T., Chen, Y., Chen, T., Ku, P. S., Taele, P., ... & Chen, M. Y. (2019, May). PersonalTouch: Improving touchscreen usability by personalizing accessibility settings based on individual user's touchscreen interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-11).
[5] Franz, R. L., Wobbrock, J. O., Cheng, Y., & Findlater, L. (2019, October). Perception and adoption of mobile accessibility features by older adults experiencing ability changes. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (pp. 267-278).
[6] Fairman, A.D., Morris, L. (2023, February 4). Development of the Mobile Device Assessment Tool (MoDAT) [Conference presentation]. Assistive Technology Industry Association Annual Conference, Orlando, FL, United States.
[7] The dexteria family of apps for Occupational Therapy and Education: Ipad apps for Fine Motor Skills, autism, and dyslexia. The Dexteria Family of Apps for Occupational Therapy and Education | iPad Apps for Fine Motor Skills, Autism, and Dyslexia. (n.d.). Retrieved February 22, 2023, from http://www.dexteria.net/
[8] Mott, M. E., Vatavu, R. D., Kane, S. K., & Wobbrock, J. O. (2016, May). Smart touch: Improving touch accuracy for people with motor impairments with template matching. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 1934-1946).
[9] Wobbrock, J. O. (2019, June). Situationally aware mobile devices for overcoming situational impairments. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (pp. 1-18).
[10] Douglas, A., Liu, L., Warren, S., & Hopper, T. (2007). Cognitive assessments for older adults: Which ones are used by Canadian therapists and why. Canadian Journal of Occupational Therapy, 74(5), 370-381.
[11] Morone, G., Spitoni, G. F., De Bartolo, D., Ghanbari Ghooshchy, S., Di Iulio, F., Paolucci, S., ... & Iosa, M. (2019). Rehabilitative devices for a top-down approach. Expert review of medical devices, 16(3), 187-195.