INTRODUCTION

People with severe physical disabilities have found it difficult or impossible to independently use assistive robotic manipulators (ARM) due to their lack of access to the conventional control methods and the cognitive/physical workload associated with operating the ARMs (Maheu, Frappier, Archambault, & Routhier, 2011). To address this issue, several researchers have investigated vision-based autonomous control (Chung, Wang, & Cooper, 2013; Driessen, Kate, Liefhebber, Versluis, & van Woerden, 2005; Jiang, Zhang, & Wachs, 2014; Kim, Lovelett, & Behal, 2009; Laffont et al., 2009; Srinivasa et al., 2010; Tanaka, Sumi, & Matsumoto, 2010; Tijsma, Liefhebber, & Herder, 2005; Tsui, Kim, Behal, Kontak, & Yanco, 2011).

Some researchers attached an eye-in-hand camera (Driessen et al., 2005; Kim et al., 2009; Tanaka et al., 2010; Tijsma et al., 2005) to the robot’s end-effector or wrist to guide the robot toward an object of interest; however, because this approach needs to update object locations continuously until the end-effector acquires the target object, the computational cost for three- dimensional (3D) object pose estimation, path finding, and motion planning tends to be high. Other researchers have mounted a camera on a fixed position at the robot base or shoulder (Chung et al., 2013; Corke, 1996). This approach has the advantage of finding a path and grasping plan, even when the object is occluded from the starting location or folding position (Srinivasa et al., 2010); however, it requires knowledge of the target object, as well as its surroundings in advance to localize the target object and plan a trajectory (Corke, 1996). Chung and colleagues evaluated an assistive robotic manipulator that operated with a high resolution webcam, mounted on the robot shoulder (Chung et al., 2013). They measured the task completion time and the success rate for a drinking task, which consisted of various subtasks, including: picking up the drink from a start location, and conveying the drink to the proximity of the user’s mouth. The average task completion time for picking up a soda can on the table was 12.55 (±2.72) seconds including the average object detection time of 0.45 (±012) seconds. The success rate of the pick-up task was 70.1% (44/62).  

Other researchers use a combination of these two approaches to provide a more reliable and robust control (Srinivasa et al., 2010; Tsui et al., 2011). Tsui et al. developed a vision-based autonomous system for a wheelchair-mounted robotic manipulator using two stereo cameras, with one mounted over the shoulder on a fixed post and one mounted on the gripper. When the user indicated the object of interest by pointing to the object on a touch screen, the autonomous control automatically completed the rest of the task by reaching toward the object, grasping , and bringing it back to the user (Tsui et al., 2011). However, the combined approach can significantly increase the implementation cost and system overheads caused by complex image processing. The adoption of a 3D depth- sensing camera could be a viable solution to this, as it could significantly reduce the computational cost for 3D object pose estimation in comparison to conventional approaches that require several images of the object in various poses (i.e., front side, backside, and all possible 3D rotations). Furthermore, a 3D depth-sensing camera is less dependent on ambient lighting conditions in comparison to conventional image processing, which requires images of an object under different lighting conditions/sources in order to improve the algorithm invariance to diverse lighting conditions.

In this study, investigators developed a 3D vision-based semi-autonomous assistive robot arm control method (AROMA-V) that enables individuals with impaired motor control to more efficiently operate ARMs. A working prototype AROMA-V was implemented based on one of the most popular assistive robotic manipulators, the JACO manufactured by Kinova Technology (Montreal, QC, Canada). JACO was combined with a low-cost, short-range 3D depth-sensing camera (Senz3D manufactured by Creative Labs, Inc., Milpitas, CA) mounted on the robot base, as shown in Figure 1.

METHODS

Algorithm Development

JACO is composed of six inter-linked segments with a three-fingered hand. Through the controller, the user can move the robot’s hand in three dimensional space and grasp or release objects. The JACO arm has a weight of 5.6kg, and can reach approximately 90 cm in all directions and lift objects of up to 1.5 kg. The Senz3D uses the 'time-of-flight' technique to obtain depth information within its field of view (diagonal 70 degrees) and working range (20-90m) at a maximum resolution of 320x240. It constructs a 3D point cloud in which each point represents the distance to objects. Based on the 3D point cloud, the shape and the dimensions (width and height) of the target object is estimated. Based on the estimated position and dimension of the target object, the end-effector pose (position and orientation) is calculated and fed to a custom inverse kinematic (IK) algorithm for the JACO, which enables the semi-autonomous operation.  In general, while it is not so difficult for people to move the ARM to the proximity of the target object, it can be challenging to fine manipulate the object. AROMA-V focuses on automating the difficult part and still leave users in control during the overall task completion.

To perform actual robotic manipulation tasks, a user starts operating the JACO using an available manual control method (e.g., joystick, touch pad, or voice recognition). During the operation, when detecting objects within a set range, AROMA-V automatically stops the robotic arm, and provides the user with possible manipulation options (e.g., “a light switch is detected. What do you want me to do? You can say switch on or switch off or do nothing”) through audible text output, based on the identified object characteristics. Then it waits until the user selects one by saying a voice command. Once the user feedback is provided, the AROMA-V drives the robotic arm autonomously until the given command is completed. Using voice recognition for controlling the AROMA-V is because it can not only provide completely hands-free operation, but also helps a user to maintain a better working posture and allows him or her to work in postures that otherwise would not be effective for operating an assistive robotic manipulator (i.e., reclined in a chair or bed).

We conducted a small-scale empirical evaluation to determine the efficacy and user satisfaction level toward AROMA-V.

Hypotheses

The following 2 hypotheses were tested:

1. Performance time for robotic manipulation tasks would be different between the AROMA-V and the manual control method;
2. Users' subjective workload would be different between the AROMA-V and the manual control method.

Subjects

Five able-bodied subjects (4 males and 1 female; age range of 22-28) participated in the experiments. The inclusion criteria were: (1) Participants are over 18 years of age; (2) participants have good speech to operate a voice control; (3) participants have normal vision to perform the manipulation tasks. Written informed consents from all participants were obtained in accordance with the Institutional Review Board at the University of Pittsburgh.

Procedures

Five able-bodied subjects (4 males and 1 female; age range of 22-28 years) participated in the study. The inclusion criteria were: (1) Participants are over 18 years of age; (2) participants have good speech to operate a voice control; (3) participants have normal vision to perform the manipulation tasks. Written informed consents from all participants were obtained in accordance with the Institutional Review Board at the University of Pittsburgh.

After the informed consent process, participants were asked to complete a questionnaire regarding general demographics, health status, and assistive technology experience. Then, subjects were introduced to the JACO verbally and through demonstrations led by the investigator. Afterward, subjects were assessed on their ability to use the three types of manual control methods (i.e., a 3D joystick, a touch screen interface, and an automatic voice recognition control interface) by performing simple directional movements. After the subject became acclimated to the JACO using the different manual control methods, they selected their preferred control interface. Next, investigators provided the subject with in-depth, hands-on training using their selected control interface. Subjects also learned to use the AROMA-V combined with their selected control interface. Once participants reported that they felt comfortable with the robotic manipulator and its operation, they were asked to complete a pretest session that consisted of three simple manipulation tasks. The pretest session determined their eligibility for the subsequent experimental test procedures. If the participant did not pass the pretest, more practice and training was provided. Prior to starting the timed performance evaluation, the task sequence was randomized. Subjects were asked to complete three trials of three tasks using a custom task board (figure 2), composed of mounted common household interfaces (i.e., a door handle, a knob, a flip/light switch, and buttons). In addition, participants were asked to complete three trials of two “real life” tasks that included picking-and-placing objects (i.e., a ball and a bottled water). After the completion of all trials for each task, participants were asked to complete the National Air and Space Administration Task Load Index (NASA-TLX). Finally, an investigator conducted a brief interview with the participants regarding their preference on the control methods.

Data Analysis

During each trial, all user interface events were recorded, time-stamped and stored in an XML file format for data analysis. The task completion time and user perceived workload via NASA-TLX were compared between the manual and AROMA-V control using the Wilcoxon signed-rank test. The statistical significance level for all statistical analyses was set to .05.

RESULTS

Task Completion Time

The task completion times for different manipulation tasks are plotted in Figure 3.

Across all threeontrol interfaces, significant difference was detected in manipulating the door handle (p=.004). Manual control showed faster performance than the AROMA-V. For the manipulation with a flip switch, there was no statistically significance difference (p=.42), though the AROMA-V showed faster performance than the manual control. For the task with a knob, significant difference was detected (p=.0005). The performance with AROMA-V showed significantly faster than the manual control. A statistical difference was detected in manipulating the light switch (p=.031). The manual control was faster than the AROMA-V. For the task with the elevator button, significant difference was found (p=.031). The performance with manual control was faster than the AROMA-V. While pushing the button with the AROMA-V, there were 3 times of missing the target. For the picking-and-placing task with a ball, significant performance difference was detected (p<.0001). The manipulation with the AROMA-V was significantly faster than the manual control. However, there were 2 times of grasping failures while performing the task with the AROMA-V. For the task with a bottled water, there was also significant performance difference (0=.004). The AROMA-V showed faster than the manual control.

Even with the small sample size, as shown in Figure 4, 5 and 6, we observed that, overall, touch screen interface have the fastest performance followed by 3D joystick and voice control interface. In addition, the relative performance advantage of AROMA-V was found most significantly when it is used with the voice recognition interface.

Perceived Workload

The user perceived workload for different control methods are reported in table 2.

Table 1. Perceived Workload

Workload	Manual Control	AROMA-V	Sig.
Mental Demand	10.0 (±4.9)	7.6 (±5.8)	.5
Physical Demand	2.6 (±1.7)	2.0 (±1.7)	.5
Temporal Demand	5.6 (±2.9)	3.6 (±3.7)	.5
Performance	4.2 (±1.3)	4.4 (±3.9)	.99
Effort	6.6 (±1.1)	5.4 (±4.8)	.69
Frustration	2.8 (±1.8)	3.6 (±3.2)	.5

As shown in Table 2, for all six scales, no significant difference on perceived workload between the two control methods was detected across all manipulation tasks.

DISCUSSION

The results above show that AROMA-V has the potential to enable users who have difficulty in using a conventional control interface for operating an assistive robotic manipulator. In terms of task completion time, for the relatively simple tasks (e.g., door handle, light switch, and elevator switch) that do not require switching between different command modes (translational, rotational, and finger mode), the AROMA-V was lower than the manual control. But, for the relatively complex tasks (e.g., knob-turning, ball-picking, and bottle-gasping) which require fine motion control imposing frequent command mode change, the AROMA-V showed significantly faster performance than the manual control. This suggests that the AROMA-V can be used as an alternative control method to enable individuals with impaired motor control to more efficiently operate the ARMs by facilitating their fine motion control. Furthermore, since the AROMA-V was developed under a Windows operating system, it can not only be easier to integrate new and existing alternative input devices without developing additional driver software, but also increase the likelihood of adoption by users and clinical professionals.

However, data from this research still have several limitations. First the given tasks did not directly represent real situations that people encounter in their day-to-day object manipulation. Actually, when developing the current algorithm, we assumed that there is no obstacle between the manipulator and the target object. In addition, the objects used in the experiments were simple shaped and had smooth surfaces. Second, data collection was performed with unfamiliar equipment in unfamiliar environments where there performance was being observed by the investigators. Third, variability in performance might be affected by fatigue or boredom due to the simple and repetitive nature of the tasks. Most of all, because findings from this study were drawn from a small number of able-bodied participants, it is necessary to collect data from the users who have physical disabilities