INTRODUCTION

Individuals on the Autism spectrum can face significant challenges in terms of learning every day skills. They may require significant time to acquire a skill to fluency and often have difficulty maintaining the skill.  Another major challenge for some individuals with Autism Spectrum Disorder (ASD) is executive functioning skills such as time management. They may also have sensory-based challenges that can provide significant distraction or produce negative sensory experiences. During these periods, they may require significant levels of support for even basic activities. Therefore, many individuals with ASD need constant prompts and visual support to assist their skill development and acquisition, as well as skill maintenance and generalization. Some individuals may also need a continuous level of prompts to help them stay focused when completing even familiar tasks. Having a scalable support that could provide information about the environment and guidance on current tasks could be incredibly helpful to individuals with ASD to increase independence.

These issues motivated us to develop a portable context aware assistive application that takes advantage of a variety of contextual information (e.g., location, time, personal schedule, user capability and preference) to provide intelligent support for individuals with ASD in their everyday life. The prompts could assist users to acquire, maintain, and generalize new skills. The prompts could also help users when they experience executive functioning, motor planning, and sensory processing challenges. In this paper, we report the design, implementation, and initial evaluation of a prototype of such an assistive application.

RELATED RESEARCH

Some individuals with ASD experience difficulty in learning every day skills. They may require substantially longer time to acquire a skill to fluency, especially for those that involve complex, multi-step tasks (Jensen and Spannagel, 2011). Once a skill has been acquired, they often have difficulty maintaining the skill or generalizing the skill in other environments (Palmen and Didden, 2012).  Therefore, they may need constant prompts and visual support to assist their skill development and acquisition, as well as skill maintenance and generalization.

Individuals with ASD can also have difficulty with executive functioning skills such as time management (Landa and Goldberg, 2005). These executive functioning challenges can also take the form of motor planning challenges, which make it more difficult for the individual to plan out how to move their body (Hughes, 1996). In those cases, they need immediately accessible supports that can help them move through the task and relieve them of motor planning (e.g., step-by-step guide to go through the process). In addition, individuals with ASD have sensory-based challenges that may result in debilitation. During these periods, they may require significant levels of support for even basic activities (Jensen and Spannagel, 2011; Tseng, 2011).

Limited research has investigated the idea of using context information to support people with cognitive disabilities. Most of the published work is in the conceptual, exploratory stage without actual implementation or user involvement (e.g., Nehmer et al. 2006; Mileo et al. 2008). Mihailidis and Fernie (2002) described a context aware device that used a camera to assist people with dementia in independent living tasks (e.g., using the bathroom). Chang et al. (2009) used location-based information to provide prompts to individuals with cognitive disabilities for catering tasks. Both studies only explored one type of context information. Bodine et al. (2011) and Melonis et al. (2012) examined a much broader range of context information (location and motion information collected through sensors, visual information collected through cameras) for the purpose of providing automated prompting for people with cognitive disabilities. However, their application only focused on workplace tasks and the use of cameras raises privacy concerns in the independent living environment. None of the reported work has implemented or evaluated haptic interaction or customizable interfaces.

Thus, this research aims to develop a context aware assistive application that helps individuals with ASD in their everyday life. We adopt a user centered design and development approach and individuals with ASD as well as their caregivers have been closely involved in every stage of the application development.

OBSERVATION SESSIONS

In order to fully understand the challenges that people with ASD and their caregivers experience in everyday tasks, we completed ten observation sessions of people with ASD conducting daily tasks in diverse domains. During all the sessions, a group of students with ASD were supervised and guided by their mentors. We initially focused on the following domains:

• food preparation and service in a food catering business in a  life skill training class
• handling money in a life skill training class in a retail setting
• book and document sorting and organization in a library setting

In addition to the above observation sessions, we also interviewed approximately ten caregivers and mentors of people with ASD, and two experts in the area to identify the major challenges and the existing approaches to address those challenges.

SYSTEM REQUIREMENTS

Based on comprehensive information collected during the observation sessions and the interviews, we specified the following system requirements that the assistive application has to meet:

• The application has to be easy to learn and use,
• The cost of the application has to be affordable,
• The level of prompts supported should be easily scalable to adapt to users with varied capabilities,
• The application should provide multiple interaction channels to accommodate unique communication needs,
• The application should support easy and smooth interaction between users with ASD and their care providers,
• The application should provide a customizable interface to accommodate diverse user preferences and capabilities.

SYSTEM ARCHITECTURE

The application is developed on the Android platform. The system comprises of five basic components, namely the context module, the task module, the user module, the interface module, and the decision module. The interaction amongst these modules enables and contributes towards the efficiency and precision of the application (figure 1).

Context Module

The context module is responsible for collecting and utilizing the context information to infer user’s current state. This module supports the collection and query of temporal, environmental, and behavioral information and sends the data to the decision module to determine appropriate promptings. Mobile devices such as Android smart phones are used both as a tool to collect context information and as the prompting interface since they are easy to carry and are equipped with built in sensors and communicating modes such as text-to-speech functionalities and camera options. We employ temporal and environmental parameters to identify useful context information. For example, tri-axial wireless accelerometers can collect useful cues regarding user status and behaviors (e.g., seated or walking).  Infrared sensors are used to track and identify location. Environmental sound is collected and analyzed to provide cues about both physical and social environment (e.g., a quite home or a noisy room). In addition, information such as schedule and user preferences from the user module are also utilized as additional features for intelligent prompting.

Task Module

Every user task (e.g., cooking, getting dressed, doing laundry) is represented as a task module. The application database constitutes of numerous task modules for the user (mentee) to select from based on his current needs.  The caregiver (mentor) could also invoke and pre-select a group of tasks as per the user requirement for any given time period (day/week/month/etc.). The system provides flexibility for both users and caregivers to modify the task group based on the appropriate user specification and user proficiency in the tasks. Each task constitutes of a sequence of sub-tasks with different levels of granularity to accommodate users with different capabilities and preferences. Figure 2 shows a partial task definition for preparing egg salad with 4 levels of granularity. When preparing egg salad, the high functioning users may only need higher level prompts such as boil egg, prepare green onion, etc. (as shown at the 2nd level in figure 2); whereas the low functioning users may need detailed prompts such as find a pot, turn on the faucet, etc. (as shown at the lowest level in figure 2).

User Module

To achieve the goal of a highly customizable design, the system enables individuals with ASD and their caregivers to incorporate relevant user information in the design. The user information includes (but not limited to) schedule, individual preferences (e.g. food, music), educational background, skills and capabilities (e.g., language, technology usage), past performance data (e.g., bottlenecks and difficulty for any specific task), and caregiver experience in regards to the user (e.g., notes and instructions developed by the mentor). Each real-time activity is constantly logged and analyzed by the system to provide a feedback mechanism to further fine-tune the user module in order to address appropriate user requirements.

Interface Module

The application interface must address the diversity in the target population by offering various forms of feedback and a highly customizable interface. Because individuals with ASD have unique communication needs, the prompting information will be provided through multiple devices in various forms, namely text, audio, image, video, and haptic feedback. The application will also allow the user to easily customize the interface to present the information in their preferred format (e.g., color mapping, font size, output). The application will also allow users to easily customize the level of desired prompts (e.g., sequential step-by-step instructions or instructions only for complicated tasks). The application will also allow caregivers to customize the interface to provide the type of support appropriate for each user.

Decision Module

The decision module works as a central processing unit that accepts data from the other four modules and provides real-time prompts to the interface module. The decision module mainly consists of two processes: activity recognition and decision logic. Activity recognition collects user specific data, task specific data, processed data and sensor data from the other modules. Decision logic aids to provide prompts to interface module after analyzing and matching the above input information.

We are using a combination of machine learning techniques to build this component, including Naïve Bayes, Support Vector Machine (SVM), and Fuzzy Logic (Das et al. 2012, Mihailidis and Fernie 2002, Intille et al. 2004). Both linear and non-linear task prompting approaches are investigated (Bodine 2011). Initially, the module includes semi-automatic decision inputs by caregivers (mentee) to help in specific user activity. After learning/training from previous user activity, the system will operate in an automatic mode to aid in everyday user activity. We are currently comparing various techniques to explore the suitable approach in the specific setting of our project in terms of both prediction accuracy and computational cost for the application.

USER INTERFACE

In order to provide scalable prompts for users with different capabilities, the interface must support smooth navigation between different task modules and different levels of granularity. Figure 3 shows two sample interfaces for two levels of prompts. The interface on the left corresponds to the third level in Figure 2, explaining the major steps to boil eggs. The user can click the button next to each step to find more details about that step. The interface on the right shows the bottom level page with the most detailed content.  This page explains the sub-step ‘Turn on faucet’ (1.1.1.3.1.1.2 in Figure 2). The prompt is delivered thorough three output channels: text instruction, audio instruction (either automatically played or played when the user clicks on the ‘Tell me what to do’ button), and an image. The bottom level page may also contain warnings, tips, and reminders. Warnings include information about potentially dangerous situation that needs caution (e.g., hot surface); tips provide useful information such as the location of a specific utensil; reminders notify users about situations that may induce sensory-based challenges (e.g., the noise of an oven fan).

INITIAL EVALUATION

We conducted a user study to evaluate the initial proof-of-concept prototype. Three high school students with ASD participated in the study. One is high functioning, the second is low functioning, and the other is in between. During the study, we demonstrated the application prototype to the participants, then the participants navigated through the cooking task modules. The feedback from the participants is highly positive. All three participants stated that the application would be very helpful in their daily routine. They commended multiple features of the application, such as the scalable level of prompts, multiple output channels, and the smooth navigation design. All participants expressed interest in using the fully-functional application in the next stage of the project development.

CONCLUSION

This research aims to develop a portable context aware assistive application to help individuals on the Autism spectrum live a more independent life.  We believe that this solution has the potential to alleviate the workload of the caregivers of people with ASD. It will be particularly helpful when the caregivers are not available. We are currently working on the following tasks:

• implementing and testing the decision module,
• testing and fine-tuning the context and the interface modules,
• collecting and developing materials for the task and the user modules
• preparing to conduct the first comprehensive user evaluation in early spring 2014.