A robot referee can really keep its ‘eye’ on the ball
Here’s how computers are helping to judge tennis, baseball, gymnastics and more
When Taylor Townsend, here, competes in World TeamTennis matches, robots now judge whether her serves and volleys land in or out of the court’s lines.
World TeamTennis
Thwack! Taylor Townsend’s serve sends the tennis ball hurtling over the net. Her opponent returns it, but the ball lands just past the line. “Out,” says a voice. Townsend, a professional tennis player from Chicago, Ill., wins a point.
This may seem like an ordinary tennis match. But who yelled “out”? It wasn’t a real umpire, at least not a living one. It was a computer system. Called Hawk-Eye Live, it tracks each ball in World TeamTennis matches. Its mission: Figure out exactly where a ball lands, then make the call.
In sports, things move quickly. In the summer Olympics, you see tennis serves that propel balls at more than 190 kilometers (120 miles) per hour. Baseball pitches can zoom over the plate at 150 kph (90 mph) or more. A gymnast launching off the vault will flip and twist multiple times within a single second.
Even at these speeds, accuracy matters. A tennis ball must land inside the lines of the court for play to continue. A baseball must pass through an invisible box near the hitter to count as a strike. And gymnasts will be graded based on how many flips and twists they do — and how well they do them.
To make these calls, human referees, umpires and judges train for years. But “the human eye is flawed,” says Bryan Hicks. He’s a tennis umpire and director of officiating for World TeamTennis in Carlsbad, Calif. Human eyes can’t always catch high-speed action. Even the most experienced officials make mistakes. In an Olympic event, an official’s mistake could send the wrong player or team home with the gold medal.
Can technology like Hawk-Eye do a better job at keeping an eye on the ball? Yes, say many coaches, players — and even umpires.
Hicks says that Hawk-Eye Live has made tennis “more accurate.” Similar systems could make many sports fairer. Already, some organizations are experimenting with systems that take over part of the job of a human umpire or judge. You may not see these systems in action yet in professional competitions and Olympic qualifying events. But in coming years, they will likely become common, if not the norm.
A virtual tennis court
In most professional tennis matches, the main umpire — called the chair umpire — sits right beside the net. At the same time, up to nine more umpires watch the lines all around the court. These line umpires decide whether a ball lands in or out of the court. If the ball is in, the player on the receiving side must return it or lose the point. If the ball is out, the other player loses the point.
The Hawk-Eye computer system watches the ball, too. If a player disagrees with the umpire’s call, she can challenge it. Then, the umpires look at Hawk-Eye’s result.
In World TeamTennis, things are different.
This organization runs a series of team-based matches every summer. In 2018, they replaced all human line umpires with a new version of Hawk-Eye that makes live calls. The chair umpire is still responsible for running the game. But the computer system makes all calls at the lines.
Taylor Townsend says that the system gives her peace of mind. “In tennis, it’s a very thin line between in and out,” she says. “I thought it was great to take human error out.”
Human umps may feel hot or tired. They may have the sun in their eyes or become distracted by a mosquito. They may even unintentionally favor players of certain nationalities, races, ages or backgrounds. A machine will not experience any of these problems.
So how does the machine do it? Engineers must first spend several days setting up each stadium that will use the system. They measure the precise position of all the lines and “create a virtual-reality world to mirror what is in the stadium,” explains Hicks. They also set up 12 cameras. These will watch every part of the area where the game takes place. Then the engineers run tests — lots of them — to make sure everything works as it should.
During a match, those cameras capture a ball’s flight. Software finds the tennis ball in the video. It can do this in bright, overcast or shadowy conditions. A video camera doesn’t capture every single moment of the ball’s flight, however. It actually takes many still photos very quickly. The number of photos it can take in one second is called the frame rate. In each frame, the ball will be in a new position. The system uses math to calculate a smooth path between all these positions. It also takes wind conditions into account.
The system now places this ball’s path into the virtual court. When the ball touches the ground in the real world, it also touches the ground in the virtual one. The system instantly knows which side of the virtual line the ball is on. And it can play back a video of the ball landing in the virtual court.
But is that where it landed in reality? Tests by Hawk-Eye Innovations show the system is accurate to within 2.6 millimeters (a tenth of an inch).
Radar to the rescue
A tennis court’s lines are painted on the ground. In that sense, they are fixed. In baseball, the pitcher aims to throw the ball through something called the strike zone. “It’s an imaginary floating box,” explains Brian deBrauwere of Hershey, Penn. He’s an umpire with Atlantic League baseball, now in its 23rd year. (So far, Major League Baseball teams have signed contracts with more than 100 Atlantic League players.)
The strike zone is as wide as home plate. Its height extends from a batter’s knees to the middle of his chest. So its size varies with the stance and build of a player. This gives each batter a fair chance to hit the ball. If a baseball travels through this imaginary box on its way from the pitcher to the catcher, or if the batter swings and misses, the umpire calls a strike. If the batter doesn’t swing and the baseball doesn’t pass through the strike zone, the umpire calls a ball. (If the pitcher gets three strikes, the batter’s out. A batter who gets four balls gets a free walk to first base.)
Human umpires tend to see this imaginary strike zone somewhat differently. Fred DeJesus, an umpire with Atlantic League baseball, says the strike zone used to “change day to day, umpire to umpire.” In his league today, the zone is much more consistent. That’s because a computer system named TrackMan now calls all balls and strikes.
TrackMan doesn’t use any cameras. Instead, a box-shaped radar dish located above home plate sends pulses of radio waves toward the pitcher. When these waves hit the moving baseball, they bounce back. The system measures these returning waves to figure out the ball’s position and to figure out how quickly it’s moving. This is called Doppler radar tracking.
TrackMan and similar systems were originally developed to give golfers and baseball players detailed information about how fast they hit or threw the ball and how its arc curved. This can help them improve their skills. But the same system also can act as a robot umpire. All it needs to know is where the strike zone is.
Once again, engineers have to carefully set up the system for each ballpark. They measure the precise position of home plate. They also collect all the players’ heights. The system then maps a strike zone for each successive batter based on his height in its database. The system does not look for the player’s knees or chest. No matter how the player stands, his strike zone will not vary. It will remain in the same place.
As each pitch comes in, the system calculates its path and decides whether the ball goes through that player’s strike zone. The home plate umpire then hears a voice through an earpiece. It says “ball” or “strike.” If the system makes an obvious mistake or fails to track the pitch, the human umpire must make the call.
DeBrauwere used this system during the 2019 Atlantic League season. At first, “it was very odd and disconcerting,” he says. He didn’t really like having a computer tell him what to call. Though the experience soon began to feel normal, he thinks the system is not quite good enough. It’s pretty accurate for balls that move in a straight path, he says. But it doesn’t work as well for sliders and curveball pitches, or in windy conditions. In these cases, there’s a greater chance that the computer’s calculations won’t match reality.
Other factors can mess with TrackMan’s results, too. For one thing, players haven’t always been honest about their height. DeJesus says they tended to add a few inches. When they realized how TrackMan worked, though, they had to come clean. Another time, when a player slid into home, the plate moved slightly. The system had no way of knowing that home plate had moved. So it calculated the strike zone where home plate used to be.
“We had to realign home plate,” recalls Rick White, president of the Atlantic League in New York City.
With each such situation, the league has been working out the kinks of its new system. Overall, White says, the goal is for the strike zone to be “a little more consistent and predictable.” Major League Baseball (MLB) sponsored the setup of TrackMan systems in Atlantic League ballparks. Minor League Baseball plans to soon begin using TrackMan to call balls and strikes for its games. If all goes well, within five years MLB parks could start using this technology or something similar.
Flips and twists
If tracking a fast-moving ball’s precise position seems tricky, imagine trying to track how well an entire person spins or flips. This is what gymnastics judges do. In the Olympics and other major competitions, gymnasts are scored on the difficulty and execution of certain moves. Difficulty measures the number and types of flips or twists. But the movements often are so fast that they’re hard for the naked eye to register, explains Brett McClure. A former Olympic athlete, he now coaches the USA Gymnastics men’s team. Judges often guess the number of flips or spins based on how long the athlete stayed in the air and what position he or she landed in.
Gymnasts also get an execution score based on the angle of their joints. For example, in a handstand, the athlete’s arms should be perfectly straight. A judge will deduct points based on the degree of any bending. Once again, human eyes can only guess what the actual angle had been.
A new computer system doesn’t have to guess. Shoichi Masui and other engineers at Fujitsu Laboratories Ltd. in Kanagawa, Japan, developed the system. It uses 3D laser-sensor technology, also known as lidar, to capture the contours of a gymnast’s body. It can count twists and flips and measure body angles.
“This project started from a joke,” says Masui. The International Gymnastics Federation is the group that governs the sport of gymnastics. In 2015, its president joked that by 2020 robots would score gymnastics. Now, Masui says, it is no joke. It’s reality.
Movie makers and video-game developers regularly use motion-capture technology to record human actors. Then they map the actor’s motions onto digital characters. Usually, the people being tracked have to wear bright white markers all over their bodies. The markers show the system exactly what to track. But “it is impossible for gymnasts to wear markers in actual competitions,” says Masui. Those markers would get in the way.
Instead, Masui’s team used artificial intelligence. The Fujitsu team built an artificial-intelligence program that could learn to recognize and outline a human skeleton within these motion data.
During the 2019 Artistic Gymnastics World Championships in Stuttgart, Germany, judges could use the system during the pommel horse, still rings and men’s and women’s vault events. They only turned to the system for difficulty scores, and then only if they disagreed with each other or if a coach questioned a gymnast’s score.
McClure and his team were there. Since everyone’s skeleton is different, the gymnasts each had to get measured. They stood on a platform and lifted an arm. Then they lifted a leg. All the while, lidar measured the exact dimensions of their bodies. Once the system had these data, it could track that person’s skeleton. The display looks like “a stick figure flying through the air,” says McClure. “It’s pretty cool.”
Robots vs. humans
McClure is glad to have a system that can take the guessing out of difficulty scores in gymnastics. Eventually, the system may be tapped to help judge execution scores, too. Still, he notes, there will always be a place for human judges.
In some gymnastics routines, artistry matters. In the women’s floor routine, for instance, the athlete dances to music. “How does a machine tell who is a good dancer and who is a bad dancer?” McClure asks. That’s not something you can figure out with math.
Human umpires in baseball and tennis still have important jobs, too. Hicks points out that a tennis umpire is always paying attention to the players’ moods and how they act on court. Part of his job is to keep things under control — and a computer can’t calm down an upset or angry player. Thanks to Hawk-Eye Live, “the umpire has more attention to give to things that are better for humans to do,” he says. Baseball umpires often similarly attempt to keep the peace.
Technology could certainly make all these sports and many others more consistent and fairer. But is that what fans really want?
“Many people care about sports in the same way they care about music or art,” says Karen Levy. She’s a lawyer and sociologist at Cornell University in Ithaca, NY. You would probably rather watch a person sing your favorite song than see a robot do it. In the same way, you’d probably prefer to watch people play and referee sports.
Sometimes, she points out, human error actually makes the sport more exciting. A bad call or a mistake riles up and energizes the fans.
“People root for sports teams like heroes,” says Levy. And when heroes face difficulty or injustice, it sometimes makes us like them even more.
