Posts

Professor Stefanie Tellex of Brown University’s Computer Science Department (Brown CS) has just won the Best Paper Award at the Robotics: Science and Systems (RSS) Conference, widely considered the top single-track conference in robotics. Her paper, “Asking for Help using Inverse Semantics”, was co-authored with Ross Knepper of MIT, Adrian Li, Daniela Rus, and Nicholas Roy.

The paper focused on a team of robots that autonomously assemble IKEA furniture. Despite their advanced design and careful programming, the robots encountered failure every few minutes. For the average person, using robots for this task would be extremely difficult.

“We wanted to enable the robots to recover from failure without requiring a roboticist around to help them,” Stefanie says. “But it's very hard for people not familiar with the robots to figure out what to do.

The first breakthrough began by looking at the current state of the art. “We thought, what if the robot could use words to ask people for help? Then a person could just follow the instructions, as if they were helping another person,” Stefanie says.

Identifying that need was the crucial step: allowing the robot to ask for help. By using words to describe what it wanted a person to do, the robot could collaborate with a person to recover from failure and continue operating autonomously.

The second breakthrough was the idea of using inverse semantics. Instead of forward semantic models that take a verbal human request (“attach the table leg to the underside of the table”) and translate it into robot actions and perception, using inverse semantics reverses the flow of communication, allowing the robot to generate requests.

Now, the robots simply say what they want, and it's easy for a person to provide help:

“Screw in the white leg that is near the white table.”

“Flip the white table.”

Finally, success: “Thanks, I’ll take it from here.”

You can watch a video of the robots below.

As Stefanie explains, the opportunities for future work in this area are limitless. “There’s so much to be done in terms of interactive approaches,” she says. “We’re going to keep moving toward dialogue, toward robots that try to infer helpful actions. We’re on our way to truly effective collaboration and help, to the future where robots prepare dinner, do laundry, and clear the table.”

Part of what made winning the Best Paper Award so satisfying for Stefanie was the planning, innovative approach, and eventual success of the corresponding talk that she gave at RSS with co-author Ross Knepper. Out of forty papers, only five were invited to give twenty-minute talks. “We decided that Ross and I would give a joint talk,” explains Stefanie. “Some of the other authors had reservations: we knew it could be either really great or really awful, and possibly ruin our chances for an award! We practiced at least a dozen times, and we were really happy with the results.”

Brown CS professor Michael Littman was an observer for one of Tellex’s practice talks: “The work that Stefanie is doing on inverse semantics is really interesting, leveraging the idea that language production can be thought of as running the process of language understanding in reverse. In addition to the lovely ideas and impressive results with robots, the tag-team talk she did was innovative and exciting.”

“It went extremely well,” says Master’s student David Abel of Brown CS, an RSS attendee. “They get high marks for attracting a huge number of people and creating an engaging dialogue, with one of them posing a question and the other resolving it. It’s wonderful to see great research followed up with really effective communication.”

You can see a video of the talk by clicking here.