Driver-assist technologies in modern automobiles, including emergency braking and lane assistance, incorporate robotics autonomy to transform traditionally human-operated machines into shared-control systems.

Through the integration of sensors and practical machine intelligence, robotics autonomy could also enable life-changing independent mobility for millions of people with severe motor impairments for whom current, commercially available power wheelchairs can be overly burdensome or entirely inaccessible.

Project Drive — a multidisciplinary partnership of academic, industry, and non-profit members including Northwestern Engineering’s Brenna Argall — aims to bring to market the first active driving assistance system for power wheelchairs, increasing access to safe, independent wheelchair operation.

Project Drive will receive a $5 million award as one of six teams advancing to Phase 2 of the National Science Foundation (NSF) Convergence Accelerator funding for the 2022 cohort of Track H: “Enhancing Opportunities for Persons with Disabilities.” Launched in 2019 within the NSF Directorate for Technology, Innovation and Partnerships, the Convergence Accelerator program supports the sustainable development of assistive and rehabilitative technologies to provide persons with disabilities and their caregivers enhanced independence and engagement within the community and workforce.

Building on the nine-month Phase 1 planning and prototyping effort, the three-year, $5 million Phase 2 project, titled “Mobility Independence Through Accelerated Wheelchair Intelligence,” is accelerating the accessibility and utility of power wheelchairs by radically changing how control inputs are interpreted from movements of the human body and communicated to the wheelchair control system and by leveraging practical and deployable machine intelligence to enhance safety and facilitate independent wheelchair operation.

“We're aiming to address barriers to independent wheelchair mobility,” said Argall, associate professor of computer science and of mechanical engineering at the McCormick School of Engineering, and associate professor of physical medicine and rehabilitation at Northwestern University’s Feinberg School of Medicine.

Argall, who heads the Assistive and Rehabilitation Robotics Laboratory (argallab) at the Shirley Ryan AbilityLab, is the principal investigator (PI) of the NSF Convergence Accelerator project. The argallab team working on the project also includes mechanical engineering PhD students Larisa Loke and Andrew Thompson; research assistant Joel Goh, who is pursuing a bachelor’s degree in biomedical engineering and a master’s degree in computer science; and project administrator Stacie Brown.

Jered Dean, chief technology officer and co-founder of LUCI Mobility Inc., is a co-PI. LUCI is an advanced wheelchair technology company that pioneered the first commercially available, sensor-based wheelchair driver assistance system.

Three fronts of research

Project Drive’s NSF Convergence Accelerator Phase 2 effort will focus on advancing and evaluating progress within three research thrusts.

The wheelchair intelligence thrust — co-led by Argall and Dean — will integrate two active driver assistance paradigms built by the argallab team into the commercial, LUCI-enabled power wheelchair.

The REACT active corrective assistance system adjusts the user’s command to avoid collisions and drop-offs. Project Drive aims to roll out the REACT active assistance add-on to LUCI beta-testers by the end of 2024 and to all LUCI users as an opt-in feature by the end of 2025.

“This technology that started in my lab will be deployed to LUCI beta users by the end of the year,” Argall said. “Seeing technology from my lab roll out into a commercial product is super exciting. We're doing this work because we want to see it make an impact in the world.”

Project Drive’s ROUTE assistance system will autonomously drive the wheelchair to a target destination within a known environment mapped via recorded data from sensors. Industry partner Sung Kim, president of Function Engineering, is collaborating with Argall and Dean to develop an improved odometry system for accurate localization.

Prospective end-users provided valuable insights regarding instances when they would prefer complete autonomy control versus shared or partial driving control.

“We learned through our interviews that there are times when people don't want to retain control because driving is such a burden for them,” Argall said.

In Phase 1 of the communication protocol thrust, the LUCI team developed and released the open-source Wheelchair Digital Interface (WDI) API for bilateral communication between the wheelchair control system and input devices.

The WDI has already been adopted by LUCI and LifeDrive, an alternative drive controls adaptive technology company. In Phase 2, Project Drive will expand WDI’s capabilities and advocate for its adoption as the International Organization for Standardization (ISO) and Rehabilitation Engineering and Assistive Technology Society of North America (RESNA) standard for wheelchair communication.

Within the control input mechanism thrust, the argallab is using the WDI communication protocol to develop the hardware, software, and usage guidance for a flexible and multimodal control interface that conforms to the power wheelchair user and their preferences and can be used in combination with current control interfaces such as a gaming controller.

Bridging the gap in accessible wheelchair control between drivers who can operate a joystick and those who cannot, the control interface will capture control signals from nearly anywhere on their body that has some residual movement and enable safe, proportional, and multi-dimensional control.

“It was completely straightforward to drop in other controllers using the WDI. Simply because of this communication protocol change, LUCI users can simply receive a software push and then use a gaming controller to drive their wheelchair,” Argall said. “LUCI beta users are currently driving their wheelchairs with adaptive gaming controls that meet their needs much better than traditional wheelchair controls.”

Engaging Stakeholders

Project Drive will continue engaging assistive technology stakeholders in rigorous cycles of design, evaluation, and adoption throughout all three research focus areas via virtual and in-person testing.

Clinical partners from the Shirley Ryan AbilityLab — including physical therapist Devon Butts and occupational therapist Sarah Van Dyck — are providing expertise in wheelchair use and evaluation. Occupational therapist Daniel Vance, an assistive technology and equipment specialist, has been an instrumental partner since the Phase 1 project launched.

Project Drive is also working closely with caregivers and wheelchair end-users and beta-testers from the Shirley Ryan AbilityLab participant registry who have a broad range of diagnoses, including amyotrophic lateral sclerosis (ALS), cerebral palsy, muscular dystrophy, multiple sclerosis, spinal muscular atrophy, traumatic brain injury, and Parkinson’s disease.

Engineer and end-user Kevin Rowland, who is living with ALS and has developed his own custom interfaces to operate his power wheelchair, wrote extensive code during Phase 1 for the proof-of-concept flexible interface. During Phase 2, he is providing design guidance and code for the flexible controller research. All of Rowland’s code is written using an eye-gaze interface.

“Kevin is phenomenal,” Argall said. “He has the unique perspective of the lived experience, the technical know-how, and a deep understanding of our system. He knows exactly what we're capable of or what we're one step away from being capable of, and exactly what it means to live with ALS. He is such a strength on our team.”

Networking engineer and inventor Shawn Sexton, the father of LifeDrive founder John Sexton and co-creator of the company’s EyeDrive, CareDrive, and VoiceDrive technologies, is also a partner on Project Drive. Shawn Sexton was diagnosed with bulbar onset sporadic ALS in 2014. As an early adopter of the WDI, he continues to provide feedback on wheelchair control.

If we find that this interface has utility for wheelchair control and computer use, that could potentially open access doors for a lot of different people. That's really exciting. Brenna ArgallDirector, Assistive and Rehabilitation Robotics Laboratory (argallab), Shirley Ryan AbilityLab

Project Drive’s secondary goal in Phase 2 is to translate advances in accessible interfaces and input mechanisms from wheelchairs to computer use to achieve a broader impact beyond mobility and power wheelchairs.

The team is partnering with the AbleGamers Foundation, a nonprofit organization of avid gamers and advocates for adaptive interfaces, to beta-test the WDI to operate computer games, complete virtual tasks, and operate power wheelchairs.

Project Drive will establish six-month internships for members of the AbleGamers network to work on the novel control interface with the argallab as part of an effort to broaden participation in computing.

“If we find that this interface has utility for wheelchair control and computer use, that could potentially open access doors for a lot of different people,” Argall said. “That's really exciting.”