Teaching a Robot to Learn
Kris Weng (MSR '25) used his MSR final project to develop a dextrous robotic hand and arm that learns as it provides assistance.
In 1989, video gamers around the world were sold the potential of Mattel's Power Glove, an accessory for the Nintendo Entertainment System that fit over a player’s hand and was designed to allow gamers to control the action with arm and hand movements. Users quickly panned the cumbersome device and it is now included in a traveling exhibit called the Museum of Failure.
More than 35 years later, Kris Weng (MSR '25) used his final project in Northwestern’s Master of Science in Robotics (MSR) program to do something far more ambitious than the Power Glove. Weng combined a Franka robotic arm and RobotEra XHand and taught them how to learn.
“There is quite a paradigm shift in terms of how robots are getting controlled with AI,” Weng said. “This was quite an interesting opportunity to investigate the limit of what a robot hand could do.”
The MSR final project is a license for students to put the skills, lessons, and ideas learned throughout the program into practice. Weng’s work built off his MSR independent project to create a hand that mimicked human capabilities.
For his final project, Weng’s focus traveled upward to the arm and hand system.
“With just the hand, I could have a mimic queue and have it do simple stuff, but there's no autonomy there,” he said. “The goal of this project was to have the robot do things autonomously and have it be able to sense its environment, to be able to tell what's going on.”
The result?
LeFranX, an open-source software library that uses VR data collection and training through 100-plus demonstrations to actively learn how a human hand and arm work. The goal was for LeFranX to accomplish dexterous tasks like emptying a bin or putting bread in a toaster. The work could prove foundational to future robot/AI collaborations and give greater autonomy to people with mobility issues, among other uses.
Weng’s project reflects an ongoing shift in robotics — from mechanics and power driven by hardwired coding to a data-driven approach that empowers learning through demonstrations, experiences, and exploration.
The results could move robots from metaphorical map-followers to pioneering trailblazers.
“Before, you knew exactly what the robot’s going to do,” he said. “AI changed everything because you don’t write what the robot should do anymore. You write the code for the robot to learn from the data you provide to it.”
Weng said his MSR classes and independent project prepared him well for the final project’s challenges. MSR director Matt Elwin also served as an important source of information and inspiration, providing Weng with advice and research references at key points along the journey.
Weng is currently a senior mechanical engineer with Johnson & Johnson MedTech, where he is involved with research and development for the hardware underpinning the company’s MONARCH surgical robot platform.
The MSR program and his work on the final project broadened his horizons — and his future career opportunities, Weng said.
“The project itself was very holistic,” he said. “It allowed me to explore a lot of different sides of robots and gave me the qualifications to work in a lot more AI-oriented fields.”
