Helping Wheelchairs Navigate Autonomously

A power wheelchair is designed to help people with limited mobility move. There are multiple ways users can navigate with the wheelchair, including with a joystick or a device the user breathes into. 

But what if a wheelchair could move autonomously for the user? 

That was the question Graham Clifford (MSR '24) explored for his final project in Northwestern Engineering's Master of Science in Robotics (MSR) program. Clifford worked with researchers at the Shirley Ryan AbilityLab in Chicago for the project. 

"Heading into this project, I was most excited about trying to make a difference in patients' lives, and helping the researchers at Shirley Ryan to do that, too," Clifford said. "I was also excited to learn about visual SLAM (simultaneous localization and mapping) methods because they are extremely powerful and complex algorithms, and because I was and still am interested in working on robot autonomy." 

Clifford worked with a wheelchair equipped with LUCI, an attachable hardware and software product designed to enhance safety, independence, and confidence in users. The wheelchair had multiple cameras and sensors, including an inertial measurement unit (IMU) to measure and record acceleration and the direction of rotation. 

His goal was to use cell phone images of a space, such as a wheelchair user's home, and create a semantically labeled map. The wheelchair would then take that map and be able to localize itself within the environment. The labels would allow users to to easily and autonomously navigate between different labeled areas of the map. 

If a user was sitting in their living room, for example, and wanted to go to the kitchen, they could select the kitchen on the map, and the wheelchair would then navigate them to that room. 

"From the start, we knew that the biggest challenge would be creating maps from cell phone data that were navigable by the wheelchair," Clifford said. "The iPhones we used for mapping and the wheelchair have different sensors, and accounting for that is a challenge. Additionally, getting the data we needed to perform mapping from cell phones was a tall hurdle." 

Clifford worked on different solutions to the challenge before identifying an iPhone app and SLAM library he could use to create accurate maps. He filtered out unnecessary noise in the images and used a deep learning model to help with object recognition.  

Ultimately, he was able to get the wheelchair to localize itself and accurately estimate its position over long periods of time without getting lost.  

He credited the lessons learned in MSR, the program's faculty, and his classmates for helping him accomplish his ultimate goal. 

"Throughout MSR, the importance of project planning was stressed heavily," Clifford said. "We were always encouraged to approach our projects with a core goal, a reach goal, and a fallback goal. This structure significantly aided my work and helped me avoid procrastination while working hard to finish my core goal as fast as possible so that I could move on to my stretch goal." 

Prior to MSR, robotics was just a hobby for Clifford. He enjoyed building and programming his own machines and projects, but thanks to MSR, his passion became a profession. In February, he accepted a full-time role as a software engineer for mission autonomy at robotics engineering company Modalic. 

"I grew faster and more immensely in MSR than any other time in my life," he said. "The knowledge and skills I gained throughout the program is extremely invaluable to me. Additionally, MSR's intensity helped me develop a great work ethic, and that's something I hope to hold on to for a long time." 

Clifford will also hold onto the feelings he had upon completing his final project. He spent many hours working at the Shirley Ryan AbilityLab, and during that time he got to know a number of patients who use LUCI wheelchairs and who could benefit from his work.  

"I learned how rewarding it is to create something for someone in need," he said. "Those patients helped to inspire and motivate me to try and make something that would, hopefully, truly improve their lives."