��������

By Shannon Mullane (MJour’19)

You’re flying over a luscious landscape filled with clouds, streams and blossoming plants. You turn your head, and an eagle comes to join you. Music rises and falls depending which way you look, how fast you move.

That’s the basic idea for a new virtual reality experience designed specifically for the health care setting—an experience created in part by CMCI lecturer Sean Winters. 

Winters, who works in the Department of Critical Media Practices, has spent years investigating the countless relationships that exist between sound and images. In 2019, he partnered with Reality Garage, a VR lounge and maker lab in Boulder, and started using his audio expertise to help young patients undergoing treatment at a Colorado hospital.

It has been one of the biggest, hardest and most complicated projects he’s ever undertaken, Winters said—and it has left him inspired to do more.

“It’s extremely challenging . . . The stakes are real. The patients are real,” he said. “I’ve had to utilize so many of the skills I’ve learned through the years.”

The VR experience is designed for patients undergoing a type of physical therapy in which electrical currents help muscles contract. The therapy can help strengthen muscles, address stiffness, improve balance and more, according to health care organizations like the National Multiple Sclerosis Society. Health professionals use it as a treatment for conditions like multiple sclerosis, strokes, spinal cord injuries and nerve damage.

The therapy can be challenging for patients, especially if they have to repeat it for long durations several times each week. In the hospital, patients will don VR goggles and noise-canceling headphones on a bicycle-like apparatus. As they pedal at different speeds to improve the use of their muscles, the experience will allow them to fly.

“The therapy can be oppressive, and boring, and just not exciting,” Winters said. “On the one hand, we’re simply trying to create an experience that’s fun, entertaining, looks cool, and is replayable. . . . But then at the same time, we spent a lot of time crafting this whole narrative that’s based around healing and growth.”

The designers, however, faced one initial hitch: Typically, users control VR experiences with the help of a hand-held remote—which isn’t an option during this type of therapy.

That’s where Winters’ audio expertise became crucial.

Winters and the Reality Garage team decided to make the program audio-driven and visually navigated. As users fly through the water-rich, lush biome, their gaze activates different game elements, like collectibles and power sites. But sound guides the user just as much as the visuals. 

Some audio elements are 2D: No matter which way users turn their heads, the sound stays the same. Then there is immersive spatial audio, a 3D element. When users pass something, its sound is only heard in one ear—then both ears after they turn their head to look at it.

Because patients need to meet certain rotations per minute on the bicycle for their treatment, rhythm is vital. Winters made sure the beats per minute of the audio would match the exercise program: slower for warmups and cool downs, faster for higher intensity pedaling.

But one of the most innovative elements of the program involves real-time data. The therapeutic goal is for patients to hit a target speed. If patients are pedaling slowly, the real-time data will inform the program, and they’ll hear only half of the sound frequencies. Once they start pedaling faster, they can hear both halves of the frequency spectrum, Winters said. 

“Real-time audio synthesis in virtual reality—it’s pretty serious business,” he said.

This year, the design, testing and final stages of the project are winding down. As Winters reflected on the project’s progress, he said the experience has tied in everything he has studied, practiced and explored in classrooms with students.

Looking ahead, he envisions more virtual and augmented reality programs in health care settings—it’s an intuitive, logical and very exciting prospect, he said.

“Thinking about patients in hospitals is extremely intense, and I can only begin to imagine what that experience is like,” Winters said. “To bring even just a tiny, tiny amount of positivity or joy to that situation is so, so thoroughly inspiring and motivating to me.”