INTRODUCTION

Many challenges with alternative controllers (AC) are hindering their ability to satisfy both mobility and digital needs for persons living with complete tetraplegia [1,2]. Alternative drive systems for power wheelchairs (PWC) require regular oversight and setup by a caregiver [1], and AC for digital devices are expensive [3,4] and require even more caregiver intervention. Thus, users require a solution that reduces dependence on their caregivers and better integrates their mobility and digital needs [1].

Kinemo is a wearable AC that translates head movement and facial gestures into input controls that allow a user to drive their PWC and access their digital devices such as a smartphone, tablet, or computer. As shown in Figure 1, Kinemo relies on small custom-designed tracers placed behind the ear and on the face (cheeks, eyebrows) to generate both proportional (head motion) and discrete (facial gestures) controls to drive a PWC and efficiently access a digital device. Our solution can remove the need to buy multiple expensive ACs by replacing them all with one integrated system since Kinemo uses a 9-pin cable to connect to any PWC brand, and uses Bluetooth to connect to any modern digital device.

The purpose of this study is to validate whether a user of Kinemo would be able to (1) safely drive their PWC indoor and (2) control multiple digital devices to perform activities of daily living at home and independently. By identifying the technical challenges that will be experienced by the subjects and by learning from their feedback, the outcome of this study will help our team develop a user-ready version of Kinemo for subsequent unsupervised at-home testing by end-users.

METHODS

Seven able-bodied participants completed this study, comprised of five male and two female subjects with ages ranging 21-38. The study was approved by the Georgia Tech Institutional Review Board (#H21457) and written informed consent was received from all subjects. The study was split into three sessions: session #1 focused on training the subjects on controlling connected devices, session #2 trained them on driving a PWC, and session #3 combined both controls (PWC driving and connected devices) to perform tasks of daily living inside a house. Each session lasted between 2 and 3 hours. Additionally, the subjects were asked to provide their feedback after completing each task, along with responding to survey questions to record their experience with key aspects of the system. More details are provided below.

Session #1. Training for Smart Control

Kinemo was paired via Bluetooth to a smartphone (Pixel 4a, Android) and controlled as follows: head motion moved the pointer, a twitch of the left cheek emulated a finger tap (i.e., Select), and holding the right cheek for few seconds toggled the Scroll mode which translated head movements into scrolling. This session was composed of four tasks: (#1) Turn on/off a smart lamp, (#2) Play a song on Spotify, (#3) Play a video on a smart TV (Roku), and (#4) Browse the internet (scroll through pages of a website to play a specified video). Each task was timed and repeated three times.

Session #2. Training for PWC Driving

Subjects were trained to drive a PWC (Permobil M3) with Kinemo on indoor courses delimited by cones. The speed profile was set to Indoor Slow (1.4 mph max speed forward). Proportional driving was performed via head motion with tilting (i.e., ear to shoulder) for turning, and nodding up/down for rolling forward/backward. The twitch of the right cheek was used to stop the PWC. The tasks were as follows: (#1) Roll forward (13 m) and backward to the start line, (#2) Stop the PWC with a facial gesture, (#3) U-turn around a cone (11 m), (#4) Turn in a zigzag along 7 cones whose intervals were decreasing from 2 m to 1 m, (#5) Enter and exit a mock bedroom through a narrow doorway (75 cm) delineated by cones. Each task was completed first with the joystick of the PWC (except task #2), then repeated three times with Kinemo. The driving was done in unlatched mode, except for task #2.

Session #3. Smart Control and Indoor Driving in a Home Setting 

Both controls were combined to perform tasks that an end-user would typically do when coming back home: (#1) Enter the house through the main entrance, (#2) Play a movie on the smart TV, (#3) Turn on a smart lamp and browse the internet on a computer (scroll through pages of a website to play a specified video), and (#4) Prepare for bed by driving through a hallway, use a smartphone app to open the automated bedroom door, and park parallel to the bed. First, the subjects completed this sequence of tasks in one trial with the joystick to drive the PWC and their hand/fingers to control the phone and computer. Then, they were setup with Kinemo and completed the sequence three times. Two facial tracers were placed on the cheeks and issued the same discrete commands as described in the previous sessions. For computer control, the twitch of the left/right cheek issued a left/right mouse click, and holding the right cheek for few seconds toggled the scroll mode. Since there were three devices to be controlled (PWC, smartphone, and computer), we implemented a method to allow the subjects to independently switch between the devices by using a head clicker (Figure 1): 1 click for PWC driving, 2 clicks for phone control, and 3 clicks for computer control.

RESULTS & DISCUSSION

The completion time of each trial was recorded, along with a difficulty score ranging from 1 (very difficult) to 5 (very easy). For tasks with PWC driving, the number of collisions with an object was also recorded, which was a cone in session #2, but in session #3, the objects were a piece of furniture, a door, the side of a doorway, or even the wall.

Session #1. Training for Smart Control

As shown in Table 1, the fastest subject (#7) was on average twice as fast than the slowest subject (#5). Interestingly, the subjects #6-7 were not only the fastest but also the only participants with prior experience with Kinemo, and thus they had much more practice time than others. Excluding the subjects #6-7, the next fastest subject (#2) completed the tasks on average 40% faster than the slowest. The peculiarity of subject #2 is this subject had a spinal cord injury two years prior that left him with an incomplete tetraplegia for a short amount of time (four months). Before this subject fully recovered, he was using a hand stylus to control his mobile devices, and therefore, it is not clear whether this prior history with tetraplegia had any influence on his performance since he did not use a head tracker for alternative control, or whether being the youngest participant (21 y.o.) in this study had any impact.

Table 1 . Completion time (in sec) for session #1

Task	S1	S2	S3	S4	S5	S6	S7
#1	25	16	24	21	27	15	18
#2	53	34	41	56	61	34	30
#3	55	50	63	70	77	37	39
#4	57	51	71	72	81	42	40

Although not shown in Table 1, the third trial was completed faster than the first in three tasks and across subjects. However, this increased performance in completion time became less significant as the subjects progressed through the tasks. These results might indicate that the subjects were becoming more proficient with Kinemo's control as they practiced throughout the session, which validates the previous observation that subjects #6-7 were the fastest because they had prior practice time than the others. Although comparing Kinemo's performance to other ACs would have been useful to validate this assumption, the main purpose of this session was to train subject on Kinemo for smart control in preparation for session #3.   

Session #2. Training for PWC Driving

Table 2 . Completion time (in sec) for session #2

Task	Control	S1	S2	S3	S4	S5	S6	S7
#1	Joystick	69	69	65	65	77	73	70
	Kinemo	88	82	80	78	100	90	83
#2	Kinemo	1.1	1.2	1.2	1.1	0.8	1.5	1.0
#3	Joystick	45	45	45	43	49	45	47
	Kinemo	50	50	53	58	63	47	46
#4	Joystick	46	42	43	48	56	45	46
	Kinemo	62	52	77	128	169	70	53
#5	Joystick	32	25	40	30	48	31	22
	Kinemo	71	43	66	70	63	44	31

Table 2 shows that for task #1, the average completion time across all subjects is 23% slower than with joystick. We found that driving backward with Kinemo was the main contributor to this reduced performance. This might be due to the PWC not remaining in a straight line when switching driving between forward and backward. Subjects mentioned that it seemed easier to correct this deviation with the joystick because they can push the joystick in diagonal positions. It is more difficult to drive diagonally with the head, therefore subjects had to issue coarser and slower corrections with Kinemo that also included stopping the PWC at times. In practice, this difference in completion time between joystick and Kinemo might be less significant because end-users roll forward most of the time, and even when they do roll backward, it is typically not for a long distance that would require making such corrections.

For task #2, all subjects were able to stop the PWC with a facial gesture in around 1 second. This indicates that our system was quickly responsive and could be used in case of emergency. However, more testing will need to be done in various road surfaces (e.g., bumps, gravels, unpaved roads, etc.) to better validate its reliability.

For task #3, subjects completed the U-turn maneuver in an average of 46 sec with the joystick and 52 sec (13% slower) with Kinemo. Although not shown in Table 2, the completion time remained consistent across the three trials of Kinemo for each subject, thus indicating that the subjects were confident in turning using Kinemo's head tracking and the control was intuitive enough. Moreover, none collided with the cone, indicating that subjects were able to accurately drive the PWC around an object if ample space is provided.

For task #4, the average completion time with Kinemo was 85% slower than with the joystick. Because of the difficulty of the task, significant differences in performance with Kinemo were found between subjects, such as subject #5 completing the task 3x slower than the fastest subject. Prior experience with PWC driving might affect performance since the fastest subjects (#2 and #7) were experienced PWC drivers. Although this task was designed to be challenging, all subjects were able to zigzag through the cones without any collision, except for subject #1 (1 collision) and #4 (3 collisions). Regardless, these four collisions account for less than 3% of the total number of collisions that could have occurred, and only one was a head-on collision against a cone.

For task #5, the average completion time with Kinemo was 66% slower than with joystick. There were also significant differences in performance with Kinemo with subject #7 completing this task in average 2.3x faster than the slowest subject. Beside subject #4, all other subjects were able to clear a mock doorway unimpeded in at least one trial, which confirmed that they could drive the PWC inside the house for session #3. Subject #4 had many technical issues that drastically limited her ability to drive with fine control. The head tracer was placed too close to the neck which degraded proper tracking of the head, and the calibration was lacking parameter adjustments that could have reduced the impact of this issue, among other problems. Fortunately, we learned from it and made changes to the Kinemo software and to the tracer attachment procedure that resulted in improved satisfaction by all subjects, included subject #4, as shown by an increase in their scoring of difficulty level in session #3.

Session #3. Smart Control and Indoor Driving in a Home Setting 

In task #1, Table 3 shows the average completion time across all subjects is 42% longer with Kinemo than joystick. Subjects #6-7 were the fastest while being the ones with prior experience with Kinemo, and the average completion time of subject #7 was only 17% longer than with the joystick. All subjects were able to clear the doorway unimpeded in their third trial, and only one head-on collision was reported for subject #4 in her second trial. Three other collisions were recorded across all subjects and trials, but they were all merely a brushing of either the opened door or the side of the doorway.

In task #2, the completion time with Kinemo was in average 2.5x slower than a joystick. However, the main contributor to this difference in performance was that subjects could look at the TV while using their hand/fingers to use the Roku app on the phone, but with Kinemo, they had to look back-and-forth between the screens of the phone and the TV. Only one collision occurred for subject #4 in her first trial with Kinemo who slightly bumped into the table, but the subject learned from it and completed all other trials without collision.

Similar observations are seen in task #3 for which the total completion time for Kinemo was 2.4x longer than with joystick/hand. It is more efficient to use hand/fingers to use a smartphone with a touchscreen, and a computer mouse to move a pointer and click. However, the subjects were able to switch between three devices independently (PWC, then smartphone, followed by a computer) and perform non-trivial human computer interactions (e.g., scrolling, navigating a website, play a video) in less than 3 min for the slowest subject (154 sec). No collisions were reported with Kinemo, indicating that all subjects became proficient with indoor driving.

Table 3 . Completion time (in sec) for session #3

Task	Control	S1	S2	S3	S4	S5	S6	S7
#1	Jysk/hand	18	17	23	19	21	16	16
	Kinemo	33	27	28	27	36	23	18
#2	Jysk/hand	37	24	31	26	32	27	27
	Kinemo	89	58	101	66	75	50	61
#3	Jysk/hand	49	43	45	40	62	48	44
	Kinemo	128	93	154	106	138	97	85
#4	Jysk/hand	59	59	59	55	56	58	53
	Kinemo	107	92	129	104	102	98	89

Task #4 was completed on average 1.8x slower with Kinemo. Opening the automated door with the smartphone was the main contributor of the delay because it was 5x slower, however, this duration includes switching control between PWC to the phone and back to PWC. There was a noticeable difference in performance between subjects that was not observed with the joystick/hand. Indeed, the fastest subject (#7) completed the task with Kinemo in average 1.5x faster than the slowest (#3), indicating again that increased practice time with Kinemo may lead to improved performance in driving and smart control. As expected, more collisions were reported in this task because it is the most challenging in terms of fine driving due to the doorway that was not ADA compliant. However, all subjects had at least one trial with Kinemo in which they cleared the doorway unimpeded, and even two subjects (#1 and #7) completed all trials without any collision. Additionally, only subject #3 had one head-on collision with the side of the door for which he corrected the trajectory of the PWC and cleared the doorway unimpeded. The other 10 collisions were merely a brush of the opened door or the sides of the doorway, which did not require the subject to correct trajectory.

Subject Feedback

For smart control, all subjects were generally satisfied with Kinemo's performance. The main concern was an issue with overshooting small targets on the smartphone, but all acknowledged they might perform better with more practice and without the pressure of being timed. Nonetheless, new users will be encouraged to start with a low pointer speed to increase accuracy and reduce the frustration of overshooting targets. Once the user will become proficient enough with head control, the pointer speed can then be increased.

For PWC driving, beside the issue with subject #4 in session #2 that was ultimately resolved in session #3, all subjects reported that they felt in control of the PWC and were overall satisfied. The main challenge was with fine driving control that was made harder due to the casters on the PWC that proved difficult to maintain straight driving. This made clearing a doorway much more difficult that it would be in real life because, in this study, the doorways and thresholds were not ADA-compliant, thus reducing the margin of error to such an extent that a small misalignment due to the casters resulted in brushing or colliding with the door frame. Nonetheless, there was only one head-on collision with the door frame that was reported in the house (session #3), all others were merely brushing the opened door or the door frame. Regardless, most subjects mentioned that their performance with Kinemo could have been significantly improved if they could have corrected their trajectory with small diagonals rather than a grid-like motion that they could mostly do with Kinemo. Therefore, our head tracking algorithm must be improved in the future to enable users to drive in diagonal more easily.

CONCLUSION

Seven able-bodied subjects completed a three-session study that demonstrated the capabilities of Kinemo as a wearable alternative controller that can drive a PWC indoors and outdoors, efficiently control multiple connected devices, and independently switch between them. The subjects were first trained on the use of Kinemo to control a smartphone and connected devices (session #1), then on driving a PWC to perform simple and advanced maneuvers (session #2). Lastly, they combined both control capabilities to complete activities of daily living that an end-user would typically do at home (session #3). All subjects were able to complete the tasks without major difficulties, except for the subject #4 that experienced significant challenges in driving the PWC in session #2. After troubleshooting the issue, we learned valuable information that resulted in a better understanding of the effect of tracer placement on driving performance, along with changes to the Kinemo's calibration procedure to improve some key driving capabilities. These changes had a strong positive effect on all subjects in session #3, including subject #4 that mentioned in their survey that they felt more confident and in control of the PWC. More importantly, only one head-on collision was reported in session #3 across all subjects and tasks. Even then, that subject was able to back out and complete the task unimpeded. Therefore, the subjects developed a proficiency with Kinemo that, in session #3, resulted in (1) safe indoor PWC driving since there were only minor collisions that did not impede the completion of the tasks, and (2) all subjects reporting being satisfied with smart control and independent device switching. Nonetheless, our team will improve Kinemo thanks to the results of this study and the feedback provided by the subjects before conducting an at-home study by actual end-users with tetraplegia to validate the usefulness and the added value of an all-inclusive and wearable alternative controller.