INTRODUCTION

People with dementia want to continue their everyday activities but struggle due to cognitive impairment. Sequencing of the steps in an activity is a major problem. Previous work, for example that of (Mihailidis, Boger, Craig, & Hoey, 2008; Pigot, Mayers, & Giroux, 2003) suggests that there is huge potential for machine-based prompting to support people with dementia through the sequences required to complete activities.

There is a need for scalable, affordable and adaptable systems that can be easily adopted and deployed. MRTs have an immense scope to generate these clues in response to the behaviour and actions of people with dementia. For example, Microsoft’s Hololens is capable of detecting the gaze of people and can be programmed to generate clues in response to gaze. This could be used to detect when people have become de-railed during an activity and are searching for cues to get back on track.

On a continuum of physical virtual devices, MRTs are anything in between (Desai, Blackler, & Popovic, 2016). MRTs or Augmented systems, can either consist of augmenting the real physical world with virtual objects, as in Augmented Reality (Azuma et al., 2001) or the virtual world augmented with real physical objects, as in Augmented Virtuality (Regenbrecht et al., 2004). MRTs such as Microsoft HoloLens(Microsoft, 2018), XBOX Kinect (Microsoft, 2010)and Osmo(Tangible Play, 2014) could offer new technological solutions for people with dementia to remain in their own home and communities by providing prompting through every day activities.

For MRTs to be able to deliver cues and prompts in task sequencing, firstly requires an understanding of interaction modalities that work for people with dementia in a mixed reality setting. Forms of prompts that are perceived and understood by people with dementia needs investigation. Effectiveness of these technologies to support people with dementia in everyday activities is governed by the level of ‘presence’ that people with dementia experience (Egges, Papagiannakis, & Magnenat-Thalmann, 2007; Schaik, Turnbull, Wersch, & Drummond, 2004; Wagner et al., 2009). 

‘Presence’ is an experience with technologically mediated perceptions that generates a feeling of being there and the illusion of non-mediation (Heeter, 1992; Lombard & Ditton, 1997; Wagner et al., 2009). The existing frameworks for presence have been established in the context of Virtual Reality (VR) that have limited use in the design and assessment of Mixed Reality environments. The real world in MRTs need to be taken into account along with the virtual world in understanding presence in MRTs. The cognitive impairment experienced by people with dementia adds another layer of complexity to our understanding of presence in relation to mediated real and virtual environments. This study examined what constitutes an experience of presence for people with dementia when they interact with Mixed Reality technologies. This study thus has examined the interactions of people with dementia with MRTs to identify barriers and facilitators to their use.

METHODS

Research Design

This research was an observational study conducted in participant homes and at Memory and Company, a memory health club for seniors with dementia. Game play was used as a probe to elicit natural behaviour in the participants. The Research Ethics Board of Ontario Shores Centre for Mental Health and Sciences approved the study. Informed written consent was obtained from all participants.

Participants

Nine people with dementia (MoCA = 13 to 25, mean MoCA = 21, Age= 63 to 90 years mean age = 79) participated in the study. Participants were recruited from Alzheimer’s Society of Durham Region (ASDR) and Memory and Company. 

Materials

We used two technologies in this study – Hololens from Microsoft and Osmo from Tangible Play. Participants played Tangram on Osmo and Young Conker on HoloLens.

Osmo and Tangram

Osmo, an Augmented Virtuality (AV) technology, with physical and virtual environments distinct for users to interact with separately. (Error! Reference source not found.), allows physical play with a virtual environment (tablet or smartphone). It comes with a reflector, a stand to place the tablet/smartphone on, and games. The camera on the tablet/smartphone captures any physical activity and manipulations performed in front of the tablet/smartphone. The captured information is fed back into the virtual world on the tablet/smartphone, and integrated with the added virtual elements in the app.

Tangram is played with seven flat shapes, called tans, which are put together to form shapes (Error! Reference source not found. (a)). The objective of the game is to form a specific shape using all seven pieces, without overlapping each other. A shape is presented to the player on the tablet/smartphone screen through the Tangram app. The player is expected to arrange the seven flat shapes in physical space to match the shape on the tablet/smartphone screen. When the correct flat shape is placed in the right place in the physical space, the corresponding flat shape in the app on the screen is filled with the corresponding colour. This is an indication that the placement of the shape in the physical space is correct.

The app provides visual prompts to the player in the form of coloured flat shapes briefly popping up in appropriate place in the shape on the screen. Another visual prompt is provided through a pair of hands emerging from the bottom of the screen, showing sequence of steps to follow to complete the puzzle. People with dementia are familiar with solving puzzles in their day programs and in their homes and thus tangram was an ideal choice amongst the games available with Osmo.

HoloLens and Young Conker

HoloLens is representative of Augmented Reality technology with overlapped physical and virtual spaces. It is a holographic computer in the form of a headset. HoloLens uses the real physical world of the user to overlay virtual elements (holograms) for the user (who wears the headset) to interact with them, see and hear them within their environment such as workspace, living room, kitchen, etc.

Young Conker transforms a real world setting such as a living room, kitchen etc. with all the physical world objects in it such as furniture, utensils, floors, walls, and so on into a platform to go on a mystery adventure.

 

The game starts with a scan of the environment through object identification, shape detection and spatial mapping algorithms built into the game. Players guide Conker, a holographic squirrel character in the environment through their gaze movements. Conker interacts with the users through speech and gestures/animations. Prompts in the form of text, graphics and animations are presented to the player to accomplish a set mission in the game

Procedure

People with dementia were observed playing games on two MRTs: Osmo and HoloLens. The participants played Tangram on Osmo and Young Conker on HoloLens. The study was conducted in two or more sessions. Cognitive impairment of the participant was recorded using MoCA assessment tool in the first session. The game play on the two technologies was carried in the second session. Participants with dementia in advanced stages (MoCA < 15) played the two games in two sessions, which means that they participated in the study in three sessions.

Participants were explained how each technology works and how to play the games on the MRT. The order in which the technologies were administered to the participants was alternated between participants to avoid carryover effects of a within subject design. The game play was video recorded for analysis. The data were analysed to identify elements of participant interactions with the two technologies (Osmo and HoloLens) that promote an experience of presence. The interactions of the participants in the virtual space, the physical space, and their attention crossover between the two spaces were coded in Noldus Observer XT 14.1 to identify themes representative of participant experiencing a feeling of being part of the combined real and virtual worlds with mediated elements. Recruitment, informed consent and MoCA screening took place before the game play sessions.

Analysis

The video recordings of the game play were coded in Noldus Observer XT 14.1 to identify themes representative of participants experiencing a feeling of being part of the real world with mediated elements around them. The video recordings were viewed several times, before a code was decided for an event. Three areas of the recordings were coded: the interactions with the physical world, interactions with the virtual world and the crossover between the two worlds.

RESULTS

Three main themes emerged from the analysis of the video recordings of the game play – affordances & perceptual elements, degree of realism and social aspect. The themes are described in Table 1. The two technologies presented prompts and clues in the form of sound/speech, textual, graphics and animations. These perceptual elements showed varied effectiveness in terms of their ability to be understood and perceived by people with dementia. All forms of visual prompts presented in game play often went un-noticed. However, speech or verbal prompts were successful in getting the attention of people with dementia. In terms of initiating an interaction in the game play, all participants always performed gestures and material interactions first. It is only when these forms of interactions did not work (people did not receive feedback from the technology that can be perceived by them), participants resorted to speech.

Clear and realistic situations that are a part of their surroundings (through spatial mapping of the surroundings for example) and embodied in their activities (such as puzzle solving) allowed participants to transcend across the physical virtual couplings. Some participants responded to Conker, the squirrel’s questions and comments in the Young Conker game with HoloLens. This resulted in participants not realising that they are actually interacting with two worlds.

People often started talking to the researcher or caregiver while interacting with the technologies. This ability of the technologies to allow social interaction with people around them is an indication that the technology mediation is invisible to the people.

Table 1.  Factors affecting Presence in interactions with Mixed Reality systems

Factors	Heuristics
Affordances and perceptual elements	The visual, auditory and material properties of the physical and virtual elements provide clues on what actions are to be performed. Clues presented to people with dementia need to be perceived to make appropriate decisions on the actions to be performed
Degree of realism	Realistic and clear representation of information in various forms of sensory modalities. This information is perceived by people with dementia to decide on the actions to be taken
Social aspect	Interactions of people with dementia with Mixed reality technologies involves social interaction with other people present around them.

DISCUSSION

The ability of Mixed reality technology to sustain continuous perception action loops in an interaction (Hinton, 2014) indicates the level of presence experienced by person with dementia. For this to happen, people with dementia should perceive the prompts presented to them and respond with correct responses to the prompts.  The study has found that people with dementia perceive verbal prompts more effectively than visual prompts and respond through embodied activities and gestures more effectively than other interaction modalities. Dementia can affect parts of the brain that control language (Bayles, 1982) which explains effectiveness of non-verbal forms of responses. Effectiveness of verbal prompts especially ones in human voice could be either because of familiarity to verbal instructions in people with dementia. Dementia affects attention, working memory and visuoperceptual ability to varying extents depending on the type of dementia (Calderon et al., 2001) which could explain participants’ attention to verbal prompts. People with certain types of dementia are also prone to visual hallucinations. Realistic and clear features (visual and audio) govern the feeling of presence.

CONCLUSIONS

The experience of presence in people with dementia interacting with Mixed reality technologies is governed by affordances and perceptual elements, degree of realism and social aspect. These findings have design implications for development of technologies to support people with dementia in their everyday activities.