Person-Centred Approaches To Evaluation Of Assistive Technology Products And Outcomes: An Australian Policy And Practice Perspective
Natasha Layton 1,2, Libby Callaway 1,2,3
1 Australian Rehabilitation and Assistive Technology Association, Victoria, Australia
2 Rehabilitation, Ageing and Independent Living Research Centre, Monash University, Victoria, Australia
3 Occupational Therapy Department, Monash University, Victoria, Australia
ABSTRACT
Australia is currently undergoing monumental reform. This includes the introduction of a $22B no-fault National Disability Insurance Scheme for Australian residents who experience significant and permanent disability and are aged under 65 years at the time of Scheme entry, and a community-based ageing in place funding stream for those aged over 65, called My Aged Care. Both of these policy initiatives may provide government funding for assistive technology and other supports, based on a person's participation goals and associated support needs. These new funding streams have opened up person-centred approaches to assistive technology (AT) product selection, and the opportunity to examine both AT outcomes and impact achieved.
This presentation will provide an overview of current work of Australia's national peak body for AT stakeholders, the Australian Rehabilitation and Assistive Technology Association (ARATA). Members of the Board of ARATA will present two frameworks in development by ARATA members and their collaborators. One is designed to evaluate and guide selection of AT options (the Performance Enhancing Technology Evaluation Framework), and the other focuses on impact achieved from the perspective of the AT user (My Outcomes Framework). Both have been codesigned with AT users, practitioners, researchers and funders.
In the presentation, research underway to test and operationalize these two frameworks will also be discussed. Practical application of the frameworks will be presented. Knowledge translation resources that are in development to assist AT users, practitioners, developers, suppliers and funders to select, trial, and evaluate both outcomes and impact achieved over time using these frameworks, will also be presented.
Relevance for the international session
This presentation will contribute evidence of approaches to the assessment and evaluation of AT outcomes and impact, and the relationship between the two focus areas. A perspective will be offered from Australia, but the dialogue will be relevant and applicable to international AT practice, the Global Alliance of Assistive Technology Organizations, and government and other policymaking bodies.
INTRODUCTION
Assistive technology is a key facilitator of human functioning and participation [1]. Effective assistive technology provision involves matching assistive products to person, to task and environment [2,3]. Assistive technology therefore includes both assistive products and services, delivered within an evidence-based set of service delivery steps [4,5]. There is substantial global interest in realizing the potential of assistive technology to deliver on public health [6] and human rights [7] agendas. Specific challenges include foregrounding ethics [8] and rights [9] in the use of emerging technologies, and enacting policy and service design which reconciles a cost effectiveness agenda with person-centered perspectives [10]. Articulating and measuring the process of assistive product selection is also challenging as this 'tacit' knowledge of allied health professionals and related workers is currently poorly evidenced [11].
The person-centered approaches described below endeavor to address these challenges, offering co-designed and internationally translatable ways to evaluate assistive technology products and outcomes. First, we present an evaluation framework for products. This framework may be used to augment, or at times replace, human input to address cognitive support needs a person may experience when completing daily tasks, and thus has been termed a 'Performance Enhancing Technology Framework'. Following, we will outline a global outcomesframework, called 'My Outcomes Framework', which integrates international classification systems, research evidence, and service delivery frameworks into eight tools to describe the impact of assistive technology from the AT user's perspective.
Performance Enhancing Technology (PET) evaluation framework
The past decade has seen the development and use of mainstream technologies grow exponentially, and in contrast to the more specialized and costly AT solutions that were previously used.
Electronic assistive technologies, including smartphones, tablets and their mobile applications, home automation systems, and wearable devices, may be considered essential components of everyday life [12]. One of the challenges of AT practice in the area of emerging technologies is that product development is occurring at a much faster rate than research to inform evidence-based approaches. For these reasons, AT users, their supporters, and those providing AT services may look to good practice steps to guide assistive technology provision [5]. The need for a framework for evaluation and selection of performance enhancing technologies that may be used for cognitive support has also been identified. It is in this context that the Performance Enhancing Technology Framework has been developed (Figure 1).
This framework can be applied by the technology user and practitioners in collaboration, when considering emerging technology interventions based on a person's goals and support needs. The framework design has been informed by an international scoping review, and grey literature and website scan on performance enhancing technologies. This desktop work was coupled with qualitative research methods, using indepth semi- structured interviews with AT users, practitioners, suppliers, funders and developers (n=18), to draft a set of domains. A final total of nine key domains, determined through thematic analysis and research participant and investigator consensus, have been identified. These domains guide consideration when exploring performance enhancing technology (see Table 1).
| Effectiveness | The extent to which the functioning of the PET improves the user's living situation, as evaluated by the practitioner and as perceived by the user, including whether it enhances functional capacity and/or independence, improves safety and/or enables participation in meaningful activities. |
| Design and Functionality | The ease of learning to set up, operate and continue to use the PET through its functions and controls, together with the physical design, aesthetics, interface and capacity for personalisation. |
| Reliability | The extent to which the PET operates with repeatable and predictable levels of accuracy under all conditions of reasonable use1 over an extended period of time. |
| Value for Money | Whether the investment of money, time or other resources to purchase, use, maintain and service the PET over time represents good value for money |
| Technical Specifications | The key technical specifications of the PET and how compatible it is with other PETs on the market |
| Sustainability | The currency of the PET and the sustainability of the technology over time (including operating system & network). |
| Service Delivery | The quality, timeliness, expertise and reliability of the professional services or PET supplier in providing customer support, repairs and servicing. |
| Privacy and Security | The usage, privacy, security and storage of data collected through regular use of the PET. Consider also regulatory and legal compliance. |
| Risk | The likelihood of device malfunction relative to the level of consequence to the user and/or their supporters. |
Applications
The PET Framework provides a structure, definition and set of questions for use during product exploration. It also offers a list of questions the AT user or practitioner can seek answers to when making AT choices, and comparing products in specific technology domains (e.g. medical reminder devices). The Framework is provided in three formats: an interactive PDF, a Word document and a fillable Excel spreadsheet. The spreadsheet version converts question responses (yes, no, unsure) into a ranking of products across the nine domains, based on the answers provided. It also identifies where more information is required. This Framework, associated tools, a glossary of terms, and digital and written instructions to guide use are currently undergoing final user testing for publication. Interest to apply the Framework is welcomed.
My AT Outcomes Framework
Global thinking on AT is progressing towards a systems view which encompasses AT people (users), policy, personnel, products and provision. A summary of AT outcome measures according to these system dimensions was published as background to the upcoming Global Report on Assistive Technology [13]. This summary discussed the range of challenges in synthesizing practice, research and policy in AT outcome measurement. It is in this context that the My AT Outcomes Framework (Figure 3) has been developed, empowering AT users to manage their own data and providing a flexible structure for people tell their stories in a systematic way. Needs analysis conducted by a consortia of stakeholders across disability and aged care sectors [14] identified the need for n-of-1 data capture formats to enable AT users to collate evidence regarding their support needs according to established criteria, and communicate this to AT practitioners and funders. Consensus was reached on the inclusion of six dimensions (see Table 2). An accessible PDF summary report of data entered is then generated. Before and after evaluation is recorded via rating of satisfaction, difficulty and time-use by the
user, and measurement of participation by the AT practitioner, before and after the AT bundle is deployed. These measures are superimposed to gain a dynamic picture of the impact of AT bundles and their funding. From a utility perspective, the framework needed to be robustly accessible, to generate a useable report, to capture change with different scenarios, and to enable users to opt to aggregate their data to address the critical lack of person-defined outcomes data which identifies and evidences the impact of AT provision upon outcomes.
| My Supports | Identify my bundle of assistive technology products and services, along with personal support and environmental modifications, using ISO 9999 [15] |
| My Outcomes | Articulate the full range of activity and participation chapters which are impacted by the AT bundle [16] |
| My Costs | An economic pathway analysis tool to identify costs and benefits for my AT bundle. This includes costing the AT bundle, and identify any cost offsets (direct cost savings and likely downstream cost savings) as well as social returns on investment |
| My Service Delivery | Capture the service delivery experience through the steps of initiation, assessment, trial and solution selection, procurement, implementation (delivery, setup, trial) follow up, review [17] |
| My Customer Experience | Identify whether, as a customer, the AT bundle meets expectations. This tool operationalizes nine values identified by AT users including flexible bundling; sufficient resourcing, lifestyle and lifespan considerations; support and resources; active involvement in decision making, and consideration of personal preferences and identity [18]. |
| My Rights | Using the Articles of the UN CRPD [19] which specify AT, people can identify whether their rights are realized, or not yet realized, with their AT bundle [20]. |
Applications
My AT Outcomes Framework provides a structure for key outcome dimensions, expressed as practical tools in person-first language. The dimensions and questions illuminate 'what is' and 'what can be expected', enabling a robust and methodical capture of data points with optional qualitative narrative, at one or more points of time (see example in Figure 4). The tool topics are, we propose, universally applicable to AT users and their practitioners. As such, these can be used as 'key questions' to raise awareness of the full impact of AT. They may be used to educate stakeholders as to AT rights, outcomes, cost implications and measurement dimensions. They may also be used to hold up a mirror as to the effectiveness of AT policy, against international benchmarks and frameworks. Interest has been expressed by international research consortia in developing and piloting the Framework, and is welcomed by the authors.
CONCLUSION
Australia's policy reforms have offered opportunity to re-think person-centred AT practice. This paper has presented co-designed tools to evaluate and guide selection of AT options and to document the impact achieved from the perspective of the AT user. Collaboration is welcomed with our global colleagues as we work towards equitable and measurable AT outcomes for all.
- Khasnabis, C., Mirza, Z., & MacLachlan, M. (2015). Opening the GATE to inclusion for people with disabilities. The Lancet, 386(10010), 2229-2230. doi:10.1016/S0140-6736(15)01093-4
- Cook, A. M., & Polgar, J. M. (2014). Assistive technologies: principles and practice. St. Louis, USA: Mosby.
- Scherer, M. J. (2005). Living in the State of Stuck: How Assistive Technology Impacts on the lives of People with Disabilities (4th Ed ed.). USA: Brookline Books.
- Andrich, R., Mathiassen, N-E., Hoogerwerf, E-J., & Gelderblom, GJ. (2013). Service delivery systems for assistive technology in Europe: An AAATE/EASTIN position paper. Technology and Disability, 25(3):127-146.
- Australian Rehabilitation and Assistive Technology Association (ARATA) (2016). Statement of Good Practice in Assistive Technology Provision in Australia. Retrieved from www.arata.org.au
- Tebbutt, E., Brodmann, R., Borg, J., MacLachlan, M., Khasnabis, C., & Horvath, R. (2016). Assistive products and the Sustainable Development Goals (SDGs). Globalization and Health, 12, 1-6.
- Borg, J., Lindstrom, A., & Larsson, S. (2009). Assistive technology in developing countries: national and international responsibilities to implement the Convention on the Rights of Persons with Disabilities. The Lancet, 374, 1863-1865.
- Dawson, D., Schleiger, E., Horton, J., McLaughlin, J., Robinson, C., Quezada, G., Scowcroft, J., & Hajkowicz, S. (2019). Artificial Intelligence: Australia's Ethics Framework. Australia: Data61 CSIRO.
- Australian Human Rights Commission (2019). Human rights and technology: discussion paper. Sydney, Australia: Author.
- Albala, S., Holloway, C., MacLachlan, M., Banes, D., Bandukda, R., Walker, J., . . . Massucato, M. (2019). Capturing and Creating Value in the Assistive Technologies Landscape through a Mission-Oriented Approach: A New Research and Policy Agenda. Retrieved from https://at2030.org/wp- content/uploads/sites/51/2019/08/AT2030_Working_Paper_01_Albala_Value_STYLE-2.pdf
- Waldron, D., & Layton, N. (2008). Hard and Soft Assistive Technology: defining roles for clinicians. Australian Occupational Therapy Journal, 55(1), 61-64.
- Jamwal, R., Callaway, L., Farnworth, L., & Winkler, D. (2018). Electronic assistive technology use in Australian shared supported accommodation: Rates and user characteristics. Disability and Rehabilitation: Assistive Technology, 13(7), 634-64.
- Smith, R., Scherer, M., & Layton, N. (2019) Committing to AT outcomes and synthesizing practice, research and policy. In: Layton, N., Borg, J. (eds.). Global perspectives on assistive technology: proceedings of the GReAT Consultation 2019. Geneva: World Health Organization
- Layton, N., Volkert, A., & Joyce, R. (2018). ARATA/AFDO/OTA Breakfast Forum on Assistive Technology Outcomes Paper presented at the Australian Assistive Technology Conference, Melbourne. Retrieved from www.arata.org.au
- ISO (2016). Assistive products for persons with disability — Classification and terminology. Geneva: Author.
- WHO (2001). International Classification of Functioning, Disability and Health. Geneva: Author.
- AAATE. (2012). Service Delivery Systems for Assistive Technology in Europe: Position Paper. Retrieved from http://aaate.net/wp-content/uploads/sites/12/2016/02/ATServiceDelivery_PositionPaper.pdf http://aaate.net/wp-content/uploads/sites/12/2016/02/ATServiceDelivery_PositionPaper.pdf
- de Jonge, D., Layton, N., Vicary, F., & Steel, E. (2015). Motivations and incentives: Exploring Assistive Technology service delivery from the perspectives of multiple stakeholders. In: Yih-Kuen Jan, Ray Grott ed. RESNA (2015). New Frontiers in Assistive Technology. Arlington, VA: RESNA.
- United Nations. (2006). Convention on the rights of persons with disabilities and optional protocol. Geneva: Author. Retrieved via http://www.un.org/disabilities/convention/conventionfull.shtml/
- Watchorn, V. & Layton, N. (2011) Advocacy via human rights legislation - the application to assistive technology & accessible environments. Australian Journal of Human Rights, 17(1), 117-138.
Acknowledgements:
The authors wish to acknowledge our project collaborators. PET Framework: Simpson, G; Bould, E; Rendell, R; Thomas, S; Hurst, J; MacLean, L My Outcomes Framework: Doyle, L; Noonan, M; O'Connor, J; Thompson, C, Wilson, E