Introduction

For people with disabilities, assistive technology (AT) can play a critical role in living a healthy, independent lifestyle. However, if the AT fails to fully meet the user’s needs, they risk living a lesser quality of life with limited mobility. According to a national survey, abandonment rates for ATs are nearly 30% (Phillips & Zhao, 1993) resulting in a waste of time, money and freedom for individuals with disabilities. The two main factors associated with such abandonment are change of user needs and lack of user involvement in the selection process (Hocking, 1999). This suggests the importance of gathering incremental feedback from AT users. Comprehensive design methodologies involve users in the design process to identify unmet needs, generate product ideas, gather feedback on prototypes and field test the final product for further refinement (Ulrich & Eppinger, 1995).

Currently, the most common way to involve users in the designing and testing of new ATs is to gather their feedback at several steps of the design process through focus groups (Ulrich & Eppinger, 1995). While this approach can be effective, face-to-face (FTF) meetings can be challenging for those with disabilities due to travel constraints. Online focus groups (OFGs) provide an alternative method to gain similar amounts and quality of information (Underhill, 2003) through an online platform, such as a video conferencing system. This offers several advantages over FTF groups for both participants and researchers. Studies show that participants strongly favor OFGs because of the convenience they offer to participate on one’s own schedule and preferred location (Zwaanswijk & van Dulman, 2014). Participants also appreciate the higher level of anonymity in OFGs allowing them to speak more freely, thereby offering more constructive feedback (Montoya-Weiss et al., 1998). Furthermore, the asynchronous feedback produces immediately available data, which considerably decreases costs and time needed for data analysis (Zwaanswijk & van Dulman, 2014).

As a research laboratory developing products that support the independence of people with disabilities, we are constantly seeking end-user feedback to understand design priorities and device potential. Due to the recent popularity and advantages of OFGs, we decided to recruit AT users and rehabilitation professionals for multiple online discussion to gather feedback for a series of products currently being developed and to generate new ideas about potential products. The AT focus groups provided insight into user preferences for a personalized navigation tool as well as direction on how to elicit open discussion for more constructive feedback to better suit user needs. The purpose of this paper is to describe a successful case-study and best practices of conducting online focus groups for assistive technology.

METHODOLOGY

Participants

The study was approved by the The University of Pittsburgh Institutional Review Board (IRB). Participants were recruited from IRB approved registries developed by the Human Engineering Research Laboratory (Pitt IRB #PRO12080311), the Department of Physical Medicine and Rehabilitation (Pit IRB #0304069), or via flyers and advertisements in print media.  The inclusion criteria were AT users or potential AT users over the age of 18 and rehabilitation professionals who (1) were over the age of 18, (2) had five or more years of experience working with AT, and (3) were currently licensed and/or certified through a professional organization appropriate to their field of practice.

Study Protocol

Participants who contacted the clinical team with an interest in the study were sent a screening questionnaire on Qualtrics (Qualtrics, Provo, UT) via electronic mail to determine their eligibility. Based on the responses from the screening survey, participants who qualified were then emailed another questionnaire to schedule a time to conduct the OFG via Skype for Business (Microsoft Inc., Redmond, CA), a multimedia conferencing platform. Attendees did not need to own or download the Skype for Business application but did need to download a plugin to enable access to the conference over a standard web browser. A maximum of 6 users were allowed to participate in each group. Participants were provided with detailed instructions for downloading and installing Skype for Business according to their operating system (OS) of choice. Those without personal computers were encouraged to relocate to a nearby public internet access point, such as a library, coffee shop or Barnes & Noble for the duration of the OFG. Participants were allowed to join the meeting group up to 30 minutes before the OFG for testing and troubleshooting. Participants also had the opportunity to call toll-free to route audio in addition to viewing the meeting online in case of audio/internet connectivity issues. Additionally, sharing of video feed from the participant was disabled and participants were encouraged to use anonymous screen names assigned by the study personnel for the conference.

After introductions from the researchers, informed consent was obtained verbally from all participants at the start of the OFG. Participants who elected to continue stayed online while those who did not consent were requested to log off.  Researchers then conducted a quick overview of the platform and turned on the audio recording before providing an URL of the first questionnaire to the participants via the IM window.  The questionnaire is composed of two sections. The first section asks individuals about their demographics - age, ethnicity, gender, geographical location (rural or urban), and socioeconomic status (employment status, household income, etc) - and the extent of disability (years disabled and comorbid conditions). The second section asks participants about current and past experiences with AT, such as the type of AT used to complete instrumental activities of daily living (IADLs), years of usage and age of technology. The questionnaire was completed outside Skype for Business. For the case of this study, the technology being assessed was the personalized accessibility map (PAM), a navigation tool that can personalize routing options for individuals with disabilities as well as those who have preferences not commonly provided by current mapping solutions.

Information on the AT was presented via a slide show presentation on Skype for Business. Participants were encouraged to ask questions about the technology either verbally or through instant messaging (IM) (Figure 1). They were also able to annotate with tools such as laser pointers, pens, highlighters, etc (Figure 2). Researchers then provided discussion sessions to allow all participants to discuss the technology and provide feedback. At the end of the discussion, final survey questionnaires capturing some quantitative information related to the technology being presented were shared. This was once again accomplished using Qualtrics and done outside the conference call. Participants who completed all aspects of the study were emailed an Amazon gift card containing monetary compensation for participating in the study.

Post OFG, all audio recordings were transcribed for data analysis. Length of recordings varied according to the number of participants per focus group but were typically less than three hours long. Documents, sketches or annotations on the slide deck provided by participants during the OFG were screen captured and saved. Any identifiable information from the screenshots were removed.

DISCUSSION

The OFG study was very successful in terms of effectively presenting information on our latest AT to participants and getting valuable end-user feedback. There were, however, several limitations that we hope to address through modifications in a new IRB. Firstly, we were not able to prove participants actually had a disability. By eliminating the need for participants to travel for a FTF group meeting through use of OFGs, we cannot see if the participant does in fact use AT. Also, through use of Skype for Business as our online platform, voice, video and file sharing require more bandwidth and may be impractical or frustratingly slow for participants with limited internet connections. Lastly, we had a few “no-shows”. Even after agreeing to a convenient time for holding the OFG, several participants did not log on to join the meeting at the specified time/date.

As one of the modifications moving forward, we plan to add the Clinical & Translational Science Institute (CTSI) registry as a recruitment venue. This would greatly increase variety in focus group, thereby providing more wholesome feedback for our AT prototypes. User-centered innovation is a vital source for manufacturer-centered innovation in many fields, especially AT.

Another important modification will be switching online platforms from Skype for Business to Adobe Connect. Skype presents some moderation problems as it is more feasible for communication between friends or peers, rather than a more structured form of communication that is necessary for focus groups. Adobe Connect allows for different authorizations or rights to be granted to different participants, permitting the moderator to remove a non-cooperating participant or silence their account temporarily to allow other participants to freely express their opinions as well. Adobe Connect features customizable meeting rooms, recording, screen sharing, polling, notes, chat, virtual whiteboards, sophisticated user permissions management, and audio and video conferencing, among other functions. Because the costs is approximately $55/month, it is an accessible option for research groups with limited budgets as well.

An increase in anonymity can decrease social pressure to continue participation in an OFG, and therefore, participants become disengaged and drop out. (Johns, et al., 2004). In order to reduce “no-shows”, we plan to phone call the day before a focus group to remind participants of the scheduled OFG and associate a voice to the email request. This is more personable and will hopefully increase feelings of responsibility to keep a prior commitment. In addition, we can post profiles of research team members and allow time for introductions and informal communication in the beginning of OFGs to establish rapport (O’Conner, et al., 2008).

Furthermore, in order to incentivize participation, we plan to consider encouraging the use of emoticons to compensate for lack of visual cues and provide insight to feelings (Fox, Morris & Rumsey, 2007). While this may seem unorthodox, emoticons and typical text or chat abbreviations are very popular in indicating tone and welcoming friendly discussion with more honest opinions. Perhaps we can also decrease the time allotted for OFGs to two hours maximum instead of three hours. Lastly, we will consider asking participants if they would be more willing to join the OFGs if they have the chance to be entered to a raffle for a small unknown prize for their full participation. This may possibly save the researchers money as well.

RESULTANT GUIDELINES

The following section serves as a listing of recommendation for effectively conducting OFGs for AT based on our findings and supportive studies. While these guidelines have proven useful for both researchers and participants in our pilot study, it is under the jurisdiction of the researcher(s) to determine whether an OFG would be the most appropriate method of gaining end-user feedback for their specific technologies.

For the purposes of online feedback, researchers must be well-versed in the use of online data collection tools and able to teach such skills to participants via the online environment. They should feel comfortable manipulating graphic, sound and video files at a relatively fast pace during OFGs (Wilkerson, et al. 2014). Prior practicing in-house with fellow researchers should enhance such skills to optimize performance during presentations and interactions with participants. We found this to be useful not only when training new researchers to conduct OFGs but also to sharpen the skills of those already familiar with such processes. Researchers should also be trained in collecting and analyzing qualitative data, such as asking relevant questions that encourage detailed narratives from participants. It should also be clearly stated that any data collected is the property of the research institution and that saving a copy of the data file or sharing with others is prohibited.

Users should be provided with a detailed document of how to install and participate in the online platform of choice before scheduled OFG. This can help ensure that participants are able to log in at the time of the focus group and help eliminate unnecessary frustration during the conference. We provided participants with a detailed document for joining and participating in an online meeting with Skype for Business either for Mac or Windows OS at least a week prior to holding the OFG. The standard operating procedure documents enumerate the steps required and also contain screenshots of multiple windows one should see during proper installation and usage. In addition, we set aside time before and during the first twenty minutes to setup and provide tech support in case there was still any confusion.

It is recommended that researchers ask participants to create unique user identifications. This increases anonymity among participants and confidentiality between participants and the research team. Researchers should meet with their institution’s accounting department to develop an online payment procedure that allows for minimal collection of participants’ personally identifiable information. We want to provide participants with nominal monetary compensation for their time and feedback without infringing on their personal information. For this study, participants were emailed an amazon gift card.

Using online demos to show active AT features elicits more narrative discussion from participants. This was very successful in terms of demonstrating the PAM on the interactive website. However, this may only be appropriate for certain technologies since physical items are harder to mock up.

Researchers should use voice/video for feedback collection. Relying heavily on text-based data collection methods may inhibit participation from participants with limited typing skills, dyslexia or low literacy levels (Clark, 2007). It is also recommended that researchers host small OFGs to encourage group discussions, or else one person may dominate the conversation. Some people with less dominant personalities may feel discouraged to speak up in larger group settings.

Lastly, researchers working with hard-to-reach populations must be mindful of their perceived influence on the community (Wilkerson, et al. 2014). It is recommended that researchers perform relevant research to reduce any stigmas and promote well-being.

CONCLUSION

The results of this study suggest that OFGs can and should be utilized to assess AT prototypes and collect end-user feedback. The resultant guidelines should serve as a recommendation for best practices in future implementation.