Immersive Tech Experts Foresee Bright Future For Apple Vision Pro In Health Care

While the awaited Apple Inc. Vision Pro mixed-reality headset is being marketed currently for consumer use, experts in surgery simulation and immersive technologies and a surgeon who has used Google Glass in the operating room envision a bright future for the device in health care.

Richard Vincent, founder and CEO of FundamentalVR, which designs immersive surgery simulation software to run on VR headsets and hand-held haptic tools, and Sam Glassenberg, co-founder and CEO of Level Ex, which creates medical video gaming to train surgeons, expressed excitement about the Apple Vision Pro’s potential uses for training surgeons, among other possibilities in health care.  (Also see "Minute Insight: FundamentalVR Raises $20M In Series B, Further Develop HapticVR For Surgical Training" - Medtech Insight, 11 Aug, 2022.)

Richard Vincent, CEO of FundamentalVR FundamentalVR

“It’s a fantastic headset, and headsets keep evolving very fast,” Vincent told Medtech Insight. “The Meta Quest 3 [mixed-reality headset] that came out at the end of last year was a big step forward, and this is another big step forward. It’s got some amazing capabilities … and our use case in health care [is in] training, education and skills transfer where we want to get people to competency through a virtual reality experience.”

Glassenberg noted, “The Vision Pro offers some specific benefits over other headsets out there. You’re still not going to be able to use it in the operating room, but from a training and practice perspective, it is very helpful.”

The executives find many surgeons tend to be Apple enthusiasts. Vincent said he was asked by several surgeons if they could try out the Vision Pro before it launched on 2 February.

Rafael Grossmann, a trauma surgeon at Portsmouth Regional Hospital in New Hampshire, who is known as the first surgeon to introduce Google Glass to the operating room in 2013, also expects interest to be high in the health care sector. “The fact that it’s an Apple device will help us certainly push the interest and the penetration of this device – not just in the general public, but also in the [health care] industry,” he said. 

He expects the Vision Pro will have wide-ranging applications from surgical training and educating patients and health professionals to connecting patients with loved ones at home through the virtual immersive experience. It also should help to streamline burdensome tasks in hospitals, including electronic medical record documentation.

According to Bloomberg News, Apple executives cited health care, training and education as key areas for its still nascent technology. Mike Rockwell, vice president of Apple’s technology development group, was cited as saying that surgeons often struggle to look at displays during procedures, and this is where the Vision Pro “could bring all of that together and hopefully improve outcomes.”

Standout Features

The experts Medtech Insight spoke to agreed that the Vision Pro stands apart from other VR headsets on the market when it comes to the quality of its displays and high resolution, eye-tracking and gesture-based controls, and its transference between immersive virtual experiences and real-world view, a feature known as “passthrough.”

Apple markets the Vision Pro as “spatial computing.” It is designed to be a device that lets users create a virtual desk space and do real work. Users can open up virtual windows, web browsers, apps, videos and photos and move them around in space. To navigate, the user simply looks at an icon – the device precisely tracks the eye movement – and, by pinching their thumb and a finger together, selects the icon. 

Vincent said the displays inside the Vision Pro are a huge leap forward, sharp enough to read even small text. He called the eye-tracking “phenomenal” and movement between mixed reality and the real world “seamless.” He noted, “If you raise your hands up, you can see your hands in a virtual reality space.”

Glassenberg also praised Vision Pro’s eye-tracking, which is so important for training future surgeons. “It helps you understand is the medical professional looking where they’re supposed to be looking? Are they catching the things they are supposed to catch?” Glassenberg said.  (Also see "Medical Video Games For Sharper Surgeons, Healthier Astronauts; Level Ex CEO Provides Demo" - Medtech Insight, 15 Dec, 2022.)

Vision Pro-Powered Haptics

London, UK-based FundamentalVR combines virtual reality (VR) with haptics, which allows surgeons in training to feel virtual objects and experience, for example, the sensation of cutting or drilling without actually being in the operating room. FundamentalVR’s customers are mainly medical device and pharmaceutical companies who buy an annual subscription to gain access to the company’s platform and then distribute it to their customers, which tend to be health systems, surgeons, nurses, and other health care personnel.

FundamentalVR’s software is hardware-agnostic, and the start-up has been working closely with Apple over the last two to three months to better understand the Vision Pro ecosystem and how it could fit with FundamentalVR’s platform and be used alongside other devices.

“The ecosystem around the Vision Pro is still quite early, but we’ve been able to navigate through that,” Vincent said. FundamentalVR developed an app for the Vision Pro, currently available on an “invite-only” basis for their customers in the App Store, for a simulated catheter placement procedure.

“The thing that excites us and I think excites Apple as well, it’s a combination of the visual fidelity of the Vision Pro plus kinesthetic haptics,” he said. “We’re using the Vision Pro to power a haptic device, so we’re removing part of the infrastructure that we normally use, which is really exciting. So, today, if the team was to share any photos of our current setup you’d see three things – the headset, you’d see a laptop or Mac, and you’d see the haptic devices. With Vision Pro, we get rid of the laptop or the Mac. You don’t need that anymore.”

By eliminating the computer, it could lower the cost of FundamentalVR’s set-up with a Vision Pro. That said, the current price tag of $3,500 for the Vision Pro is seven times greater than for the Meta Quest 3 headset. Pricing aside, Vincent doesn’t envision that customers who currently use other headsets will readily switch to the Vision Pro. 

“There will be use cases where it’s absolutely perfect, and there will be others where it doesn’t make sense to spend that much money,” Vincent said. He expects that the fastest adoption of the Vision Pro will be in cases where haptics and imaging really matter.

Potential For Surgery, But Currently Limited

While the Vision Pro has potential for use by surgeons in the operating room, Glassenberg and Vincent believe there are factors that could limit such use. Maintaining sterility could be an issue, they said, and Glassenberg believes that “passthrough augmented reality” could be tricky during surgery, because data is still passing through cameras, not glass.

“In the operating room where you have very high contrast – extremely bright lights, you have lots of screens, you have a lot of very bright reflections – passthrough VR becomes problematic,” he said. “The cameras don’t do a good enough job of what we call the dynamic range of the operating room. The surgeons can see the MRI data projected on the patients, but it’s now harder to see the patient and see tools.”

Sam Glassenberg, CEO of Level Ex Level Ex

Grossmann dismissed the sterility concerns, noting that unlike a device that is touching the patient and therefore needs to be sterile, “anything on my head is absolutely non-sterile.”

As for problematic passthrough, the surgeon believes that limitations could be overcome by making adaptations to the environment in the operating suite.

“It depends on the type of surgery you’re performing. It depends on the type of device you’re using. All of that is somewhat adjustable,” he said. “The OR is a pretty bright place sometimes. Sometimes it’s a pretty dark place, depending on the type of surgery you’re doing. That’s not a major issue. You can always play with that.”

Grossman thinks that modifications to the Vision Pro could make it more attractive to surgeons for use in the operating room. “The device is still bulky … so I think that’s a factor, and a battery life of two and half hours is also a factor.”

The bigger issue, he said, would be ensuring that the device meets regulatory considerations such as keeping patient’s health data secure and private.

One company that is already ahead when it comes to headset use in the surgical suite is Magic Leap. The Florida-based company announced in January 2023 that it earned the IEC 60601 certification for its second-generation augmented reality headset, Magic Leap 2, which gave it the green light to be used in an operating room and other clinical settings.

Magic Leap reportedly secured about $590m from a Saudi Arabia’s sovereign wealth fund last month, just weeks before the release of the Apple Vision Pro.

“Magic Leap 2 doesn’t do passthrough AR,” Glassenberg explained. “With Magic Leap 2, it is like looking through a pair of glasses with digital content projected onto the lenses. Magic Leap 2 has its own limitations. Its digital content has a much more limited field-of-view relative to Apple Vision Pro.”

Wider Applications

Grossmann believes that the possibilities for Vision Pro in health care go much further.

Dr. Rafael Grossmann

By creating a fully immersive experience, patients could be transported into the operating room before surgery to help them better understand what to expect during surgery.

“Imagine the power of that tool to decrease the anxiety and the fear of having a procedure,” he said.

He also discussed how the device could be used to enhance communication and connectivity between providers, patients and their families, fostering empathy and compassion. 

“Imagine [virtually] bringing a patient in the ICU to their family members in the living room and seeing all the family together and how that would make a patient feel or a patient who is dying getting the last view of the family gathering at home, all talking to the patient and saying their goodbyes.”

It could also be used to optimize and streamline tasks, such as electronic medical record documentation. Instead of having to enter data into a computer that is wheeled around the hospital, Grossmann envisions a future where health professionals use “special computing” for data retrieval and notetaking aided by generative AI, while walking around the hospital or while talking to patients in their rooms. By reducing burdensome tasks, it would also help to alleviate burn-out, which is a serious problem among health care workers, he said. 

“The possibilities are endless …. [to] change the way we do computing, change the way we interact with the physical screens by using this device,” he said. He believes that the health care community would be very receptive to these use cases, citing his own research.

“Last year I finished a master’s [degree] in Health Delivery Sciences at Cedars-Science and my capstone project for that program was a qualitative analysis of the acceptance of clinical providers using a head-mounted display to interact with patients doing rounds in the hospital. Out of the roughly 20 clinicians, mostly nurses [in the study], 90% said they couldn’t wait to use it."