Cool Jobs: Doing real science in virtual worlds

Strap on a virtual reality headset and you’ll enter a different world. Without leaving your house, you can fly a spaceship through a make-believe galaxy. You can play pool with friends. Or you can perform surgery on an alien.

Virtual reality, or VR, uses special technology to trick the brain into thinking these experiences are real. A technique called stereoscopy (STAIR-ee-OSS-kuh-pee) sends a slightly different image to each eye. This can create the illusion of depth. It certainly makes video games feel more realistic. But VR isn’t just for fun. It also can help scientists do their research or share it with others.

Scientists are using VR to learn more about people and the planet. One engineer uses this technology to let kids build mountains and carve out rivers with their bare hands. A scientist who studies language puts people in a virtual restaurant to learn what happens in their brains as they converse. A doctor takes patients on a virtual field trip to swim with dolphins. The worlds they visit are not real, but the science is.

Decoding dialogue

David Peeters loved learning foreign languages when he was growing up. His first language was Dutch. He studied three others at school — German, French and English.

In college and graduate school he focused on linguistics. It’s the science of human language. And the more he learned, the more Peeters began to wonder what happens inside our brains as people converse. He began to look at language through the lens of neuroscience — the study of the brain.

“There’s a lot about the way the brain processes speech that we don’t understand,” he says. Peeters is a scientist at the Max Planck Institute for Psycholinguistics in Nijmegen (Nih-MAY-jin), a city in the Netherlands. Peeters studies the way people communicate. To answer some of his questions, he built a virtual restaurant.

Real people stroll into it. To do this, they wear 3-D glasses. The small room they walk in has screens on every wall. It’s called a cave automatic virtual environment — CAVE for short. It’s basically a theater with a 3-D movie projected on every wall. To someone wearing 3-D glasses, it feels almost like a real world. (For people familiar with Star Trek: Next Generation, CAVEs are essentially a real-life version of the holodeck.)

Story continues below image.

The screens show scenes inside the virtual restaurant. Each person who takes part in the study “becomes” a waiter or waitress through an avatar. That avatar is a make-believe character. It can be moved around and used to talk to others in a virtual world. Participants move their avatar simply by walking around the CAVE.

Peeters wants to find out what happens in people’s brains as they speak with virtual restaurant customers through their avatar. He does this by having each person wear a cap covered in electrodes.

These small sensors on wires attach to the outside of the head. Cells in the brain communicate with each other by sending tiny zaps of electricity back and forth. Electrodes listen for these electrical impulses and then report them to a computer. The computer records this brain activity as a set of wavy lines called an EEG. That’s short for electroencephalogram (Ee-LEK-troh-en-SEFF-uh-low-gram).

Peeters uses the EEG data to see which parts of the brain are most active during a conversation. This gives him clues about how the brain processes or understands different patterns of speech.

For example, there are direct and indirect ways to say something. “Please bring me another soup” is a very direct way to communicate a need, Peeters points out. But a lot of our conversations are indirect. In the virtual restaurant, a customer may simply say, “My soup is cold.”

“We understand this means the customer wants another soup, even though they didn’t ask for it,” says Peeters. That’s indirect language.

Peeters studies the differences in brain activity when a person hears direct versus indirect speech patterns. He hopes such research will one day help scientists better understand disorders such as autism. That’s a condition in which people have a hard time processing speech and communicating.

A new way to relax

For many years, Wim Veling used VR to help patients overcome phobias, or fears. As a psychiatrist, he treats patients with mental-health disorders. Veling works at the University of Groningen in the Netherlands.

A person with a fear of heights, for instance, might wear a VR headset and practice standing on top of a virtual building. When that person feels comfortable with a low-rise building, he or she can move to a higher one, Veling explains. This technique is called exposure therapy. It involves exposing people to frightening situations without putting them in true danger.  

Veling also treats patients with other mental-health disorders. These include depression and anxiety. Such patients can have problems with nervousness, irritability, sleeping and concentrating. “It can be very difficult to relax if you are feeling depressed or anxious,” he notes.

Some people with these disorders try therapy with animals to help them feel calm. One form, known as dolphin therapy, has patients swim with the marine mammals. But there can be drawbacks to this. Dolphins are big, strong animals. So swimming with them can be dangerous. Some people, of course, cannot swim . They may even be afraid of immersing themselves in open waters. This activity also can cost a lot of money if it requires traveling to where the dolphins live.

Veling wondered whether VR might offer a safer and easier alternative.

Simply watching a television show about dolphins can feel relaxing too. But VR might create a much more powerful response in the brain, says Veling.

When you put on a VR headset, a virtual world surrounds you on all sides: Schools of tiny fish pass by. Waves splash overhead. These images and sounds trick the brain into thinking you’re really in the water with dolphins swimming all around you, even though you’re sitting on a comfy couch. And, Veling adds, “When you’re really immersed in that environment, you begin to forget your worries.”

Veling teamed up with a filmmaker who uses a special VR camera. This camera can rotate 360 degrees to make virtual-reality movies. The filmmaker took the camera into the ocean to make a VR video of a pod of wild dolphins.

But Veling needed evidence that the video actually helped people. So he started a small experiment to see if virtually swimming with dolphins helped his patients relax. To test this, he had his patients watch the video through a VR headset. At the same time, Veling watches them for signs of stress. He counts how fast his patients’ hearts beat. He also measures whether they’re tensing their muscles and how much they sweat.

A rapid heartbeat, muscle tension and sweating are signs that someone is not very relaxed, he says. He cannot say with certainty that the VR sessions are helpful. He’s still gathering data. But he does hope to have some answers soon.

Veling plans to make the dolphin immersion video available to the public in the next year so that anyone with a VR headset can try it at home. He hopes it will be a new way for people to relax.

Story continues below video. 

Moving rivers

In California, Jeanette Newmiller wants to solve a very different kind of problem. She’s a water-resources engineer at the University of California, Davis. Her mission? “Figure out the best way to get water where we need it, when we need it,” she says.

Sometimes, though, engineers like Newmiller get concerned with the fish in those waters. Newmiller is working with biologists on a project to move young salmon from the Sacramento River onto flooded rice fields. That may seem like an odd place for young fish, but it could help them reach adulthood.

The Sacramento River is California’s biggest. It runs from mountains in the North of the state through its Central Valley. From there, it flows to the Pacific Ocean through San Francisco Bay. Every spring, salmon are born in the Sacramento River. These small fry travel down the river and into the ocean where they will grow into adults. When it’s time to lay eggs, those adults will return to spawn in the river of their birth.

But several years ago, biologists noticed the baby salmon were reaching the ocean too quickly. They were too tiny when they got there. Small salmon are more likely to be eaten by predators such as orcas, sharks and seals.

The biologists showed, though, that they could fatten up the baby salmon by slowing their migration. They put young fish from a hatchery onto trucks. Then they drove these fish to flooded rice fields. The fields are purposely flooded in late winter to create a temporary home for the salmon. (That flooding doesn’t hurt the farm because it occurs during a part of the year when rice isn’t grown.) In these fields, the baby salmon feasted on tiny bugs for a few weeks before being released into the Sacramento River. These young fish grew bigger than did salmon that swam straight out to sea.

The rice fields could be a pit stop for salmon on their way to the ocean. But it isn’t practical to take the fish in and out on trucks. So scientists need to reroute the water to carry the tiny fish there and away again. That’s where engineers such as Newmiller come in. She makes computer models to help design gates along the waterways that connect the Sacramento River to the rice fields.

How big should we make the gates? Where should we put them? How will water flow through them? Newmiller tests out different real-life scenarios with computer models.

She uses augmented reality (AR) to bring her computer models to life. Augmented reality is a technology that lets you see virtual images layered on top of real-life environments. (Think Pokémon GO.)

Newmiller combines AR technology with an ordinary sandbox. Another UC Davis researcher created the first AR sandbox. Newmiller made a version on wheels and takes it to classrooms. Kids can use their hands to push its sand into hills and valleys. A Microsoft Kinect 3D camera detects any changes to the sand. A projector displays colors on top of the landscape like a topographical map — one that depicts hills and valleys. If someone holds her hand over the sand like a raincloud, virtual water appears beneath it. If playful fingers carve out a riverbed or build a dam, the flow of the virtual water will follow a new path.

Newmiller uses the AR sandbox to teach kids and adults about her work. “People in California vote on water resources all the time. Teaching them about what I am doing helps them become better citizens,” she says.

This engineer hopes her AR sandbox might get kids interested in engineering too. Newmiller admits she had no idea what engineers did when she was a teenager. “As a result, I almost missed out on a really cool career. I want to show kids what we do,” she says. With her help, they won’t miss out.

This is one in a series on careers in science, technology, engineering and mathematics made possible with generous support from Arconic Foundation.