Will this smartphone app become your exercise coach?

When the COVID-19 pandemic closed gyms and put school sports on hold, many teens looked for other ways to stay active. Some took up at-home yoga or running. For high-school sophomore Michelle Hua, that wasn’t enough. This 16-year-old student at Cranbrook Kingswood School in Bloomfield Hills, Mich., invented an app to track her movements. It identifies her exercises and even gives her coaching advice.

It’s helped her stay active during the pandemic. Even more rewarding, that app helped her win the $75,000 George D. Yancopoulos Innovator Award, this week. It’s the top prize at this year’s Regeneron International Science and Engineering Fair. (For more award winners, see box at bottom.) Created by Society for Science (which publishes this magazine), ISEF brought together almost 2,000 high school finalists. This year, the annual science competition was held virtually.

Michelle is a rhythmic gymnast — someone who does floor exercises with props such as a hoop, ribbon or ball. The sport blends gymnastics and dance. But during COVID-19, her gym shut down. She continued to practice online at home, but Michelle wasn’t satisfied. She wanted to up her training.

So the teen developed an app that tracks her movements. It even tells her whether she is performing them correctly.

Some movement-identifying apps use models of skeletons to determine movement. They analyze a video to identify body parts and identify movement based on that. But that approach is not very accurate, Michelle says. “It has to know where the head, shoulders, arms, legs, feet, etc. are in each frame of the video,” she notes.

Michelle decided instead to use silhouettes, outlines of whole people. With silhouettes, the program wouldn’t “need information about the location of body parts,” she explains. “It only needs to separate the shape of the human — regardless of where the head, arms, [and] legs are — from the background it is in.”

The gymnast designed her program using a neural-net system. This is a type of artificial intelligence program that can learn from the data on which it trains. Michelle’s trained hers using data from different sets of movement files. Those thousands of videos show people in all sorts of motions, from sitting to jumping and running. Her program analyzed each video, drew a silhouette and then learned what that silhouette was doing.

The program now can recognize everything from brushing your hair to chewing gum. It also can recognize exercises such as jumping jacks. But rhythmic gymnastics wasn’t in any of those movement data sets on which her program trained. So the teen took videos of herself performing. “I labeled my own data and trained my model with it,” she says.

https://youtu.be/BfbI4s7beDs

Her new app knows what each exercising silhouette should be doing. When someone performs a jumping jack, for example, the app takes a silhouette — and then might tell the user to lift her arms higher. “Feedback from the app helps users correct their position to prevent any exercise-related injury,” Michelle says. She hopes people will use her app to exercise more effectively. People also could use it to analyze how well they perform the physical therapy prescribed to people recovering from injuries.

In developing her project, the teen worked with Zichun Zhong. He’s a computer scientist at Wayne State University in Detroit, Mich. Together, they published results of Michelle’s research in the journal Computer Aided Geometric Design.  

The next step, Michelle says, is to put her app on the Apple app store. In the meantime, she notes, “my younger brother and I have been using my app. It has helped us by keeping us active and exercising throughout the year.”