This robot won’t trip people up

At a crowded student center, people toting backpacks hurry to class. Among them is a robot on wheels. It moves left to pass a column blocking its way. Then it veers right to make room for an oncoming group of people. Crowds are no problem for this robot, which obeys human social rules to get around.

The robot’s name is Jackal. It gets where it’s going at a brisk walking pace without bumping into anybody. Jackal can “plan paths around people in a way that makes them feel comfortable and makes our robot stay safe,” says Michael Everett. He’s part of a team of students who helped design the robot. Yu Fan Chen led the team. The students attend the Massachusetts Institute of Technology (MIT) in Cambridge.

You probably walk through crowds without thinking much about it. But robots are just starting to learn how to do this. “Navigating a crowd is a difficult thing to do,” says Everett. The people in a crowd move in many directions and at different speeds. A robot has to try to predict where these people will go, then plan its own path around them.

A robot uses a set of rules called an algorithm to calculate how it should move. Sometimes, robots that are around lots of moving people get stuck. This is called the “freezing robot” problem. Their algorithms might find no way to move without bumping into someone.

But a robot that could make its way through a crowd without freezing would be very useful. For example, “I could send a robot to pick up a pizza and drive it back,” Everett says. Ford Motor Co. sponsored the MIT team’s research. Ford hopes to one day turn Jackal into a product it can sell. Such robots could help deliver packages. They might even be able to carry people who need help getting around.

Ready, set, go!

Chen and Everett thought they could help make this robotic assistant a reality. They decided to use a new computer-programming technique. Called deep learning, it lets an algorithm learn by practicing. For example, a deep-learning system might practice finding faces in photos by looking at millions of pictures where the faces have first been labeled. Later, it can find faces in photos it never saw before.

Computer scientists used deep learning to write an algorithm called AlphaGo. It plays a board game called Go. Like chess, it’s a game of strategy. But the board is bigger and more complex. For computers, the game is far harder than chess. Yet in 2016, AlphaGo defeated the human world champion at Go. This was a major triumph for computer science.

And the MIT students took notice.

AlphaGo’s win “was an inspiration to us,” Everett recalls. “It gave us a lot of confidence.”

Walking through a crowd obviously isn’t a board game. But one could think of it that way. “It’s like a game where you want to get to a certain place,” explains Everett. The player — here, the robot — has to get to its destination without freezing. And along the way it has to follow certain rules.

Though we may not realize it, people follow many rules as they walk. These social customs are part of our culture. For example, in North America, folks tend to stay to the right side of a sidewalk or hallway. If they need to get around someone who’s slow or stopped, they tend to move to the left.

When Everett’s team first started testing Jackal, the robot didn’t know these rules. Sometimes it moved strangely. People gave it funny looks, Everett recalls. So the students tweaked the algorithm. It calculated a score for each test trip based on how well the robot did at reaching its goal. But now, the robot lost points if it broke social rules.

Practice makes perfect

To learn, Jackal had to practice — but not using a real hallway. The students set up a computer model. It was like a video game. Four simulated robots started at random points on a map. Each had to reach a separate goal while obeying social rules. The computer model ran through 3,000 practice scenarios over a period of about 9 hours.

Finally, this algorithm was ready for show time.

Today, Jackal can navigate a crowd without freezing. People no longer give the robot funny looks. They do, however, stop sometimes to take selfies with it. “We’ve seen [Jackal] go for over 20 minutes at a time,” says Everett. “We’re pretty proud of that.”

Chen’s team presented its research with Jackal in September at an International Conference on Intelligent Robots and Systems. The conference was held in Vancouver, British Columbia in Canada. Jackal’s team won best student paper at the meeting. 

“It’s definitely an interesting piece of work,” says Silvio Savarese. He’s a computer scientist at Stanford University in Palo Alto, Calif. His team is working on a very similar robot. Theirs is called Jackrabbot.

Both teams used deep learning to train their robots. But Savarese’s team didn’t define any social rules. Jackrabbot had to write its own. It does this using lots of data. To get those data, a drone flies over campus. Using the drone’s video, Savarese explains, “We look at people from the top.”

Students then review each frame of that video. Along the way, they identify people. They also pick out other moving objects, including bikes, skateboards and carts. Jackrabbot’s computer program then studies how each of these objects navigate situations, such as intersections, without colliding with each other. From those, it puts together a set of observed social rules. For example, it may note that when people are in a rush, they move quickly and turn suddenly.

“Our robot can capture subtle rules,” says Savarese. But in teaching robots rules of the road — or the sidewalk — the best approach might be a combination of the methods used by each team. Some rules, like keeping to the right, are obvious enough to define. But recognizing people who are dawdling or rushing isn’t as simple. A robot might need to learn how to do this on its own.  

Neither team’s robot is ready to deliver pizzas yet. Jackal uses very expensive equipment to sense its environment. The MIT students hope to build a less costly version. They are also working to improve their algorithm to handle even bigger crowds.

Meanwhile, Savarese’s team wants to add another level to Jackrabbot’s social skills. People communicate with each other as they move around. A person on a bike may lift a hand to indicate a turn. A person waiting to cross the street may smile and wave to say, “go ahead.” Robots need to learn to communicate in a similar way. “It’s important for humans to be able to predict what the robot’s going to do,” notes Savarese.

As robots get better at cruising through crowds, they’ll start to show up in more places. That means one may be come soon to a hallway or sidewalk near you — maybe even bearing pizza.

This is one in a series presenting news on technology and innovation, made possible with generous support from the Lemelson Foundation.