Teaching Robots to Build With Legos

The possibilities of what humans can build with Legos are virtually limitless. From life-size replicas of giraffes and panda bears to full cityscapes, people are able to build mesmerizing creations using nothing but tiny plastic bricks.

But what about robots? 

How difficult would it be for a robot to build something out of Legos? That was the question a group of Northwestern Master of Science in Robotics (MSR) students tried to answer this past quarter. The students tried to understand what it would take to have a robot build a pyramid out of Legos as part of Embedded Systems in Robotics, a project-based MSR course that focuses on exposing students to a variety of software tools that could be valuable to a robotics engineer.

Senthil Palanisamy recently took some time to talk about the challenge, the obstacles he faced and the lessons he learned from trying to build with robots and Legos.

What were the biggest challenges you faced with your project?

This work is more involved than what might catches the eye as it involves various aspects of computer vision, motion planning, and force control. The biggest challenge with our work was the level of precision required to do the task. When we really calculated the amount of precision required to execute a single cycle of moving a Lego brick, we were talking about precise localization and execution with tolerance below +/- 0.5 mm. If an error goes beyond that threshold, the pickup or dropping of a Lego block would result in failure.  

The other big challenge was debugging during integration. We each had our own parts for the project — computer vision detection and localization, trajectory planning, trajectory execution, and force control — working perfectly in isolation. When we started to integrate them, it was sometimes difficult to find out where exactly problems were occurring since the error limits imposed were so small. 

How do you think the final presentation went?

We did well. The final presentation was about showcasing what we did and explaining what each part is. It is always quite exciting to get a robot working and demonstrating its abilities to a lot of people in a demo. There were a few things we would have liked to have done better, but we were running on a tight timeline and this was very difficult work. I am happy with what we achieved. 

What do you hope other students can learn from the work you did?

1. Understanding your precision limits and choosing an appropriate platform is essential. In our case, a Sawyer robot would have been better since its arm movements are more precise than a Baxter robot.
2. Apriltags were better than AR tags. Though I didn't use apriltags myself, comparing with my peers who used apriltags, they definitely seem more reliable and robust than AR tags.
3. Calibrate your camera properly when trying to localize using a camera. I got significant accuracy improvements when I calibrated the intrinsic camera parameters properly.
4. Always think about an end-to-end solution. Looking back at the work now, there was a way we could have brought up the error tolerances had we designed a custom end effector. This would have given us more room to work with in terms of accuracy, but I never thought about this during our work since I was searching for or building algorithms to solve my problem and not about improvements to the actual hardware itself, which could have made our lives easier.
5. Choose difficult work. Even if you don't achieve exactly what you proposed, you will learn a lot more trying to meet the difficult challenges.

Is there anything else you'd like to add?

ROS is an important and necessary skill. I know you can learn this little by little anytime as part of a need-based exploration, but a course like this that properly exposes you to the whole framework enriches your skill set substantially in the long run.