Hey Surgeon, Is That a HoloLens on Your Head?

Imagine maneuvering your car through a dark tunnel that bulges unexpectedly in places and then turns sharply through a maze-like passage. The perilous journey feels safer with a light and camera showing the way ahead. It’s even better if digital lines lay out a track, assuring you stay in your lane.

In a rudimentary way, that scenario illustrates the advantage mixed reality (or augmented reality) is bringing to surgery, starting with the delicate pathways of the sinus. “When you’re doing minimally invasive surgery in a narrow corridor, you want to stay in the corridor, do whatever needs to be done, and not disturb anything around you,” says Martin J. Citardi, a rhinologist at the University of Texas Health Science Center at Houston, who in 2018 became the first in the United States to perform sinus surgery using a blended view of real and digital imagery.

Digital markings superimposed on real-world camera feeds have become commonplace in everyday life, as in the rear-view cameras on cars and the first-down lines in televised football games, but it’s a new phenomenon in surgery. In 2017, the guidance system Citardi used became the first to gain approval from the US Food and Drug Administration for sinus surgery. Two different surgical planning apps designed for HoloLens, Microsoft’s mixed reality headset, have since also garnered FDA approval. Coming soon are the shimmering holograms of sci-fi imagination in surgery itself: A clinical trial is now evaluating whether HoloLens headsets help surgeons and radiologists zap tumors.

These emerging tools are “a dramatic step forward,” says Jenny Chen, a neuroradiologist and founder of 3D Heals, which promotes innovation in medical 3D printing. “The next step is to see how companies fine-tune these products to be more user-friendly and clinically more effective.”

Microsoft displayed its HoloLens 2 headset at Mobile World Congress in February.

Photograph: Joan Cros/Getty Images

The HoloLens headset, whose second iteration is due for release by the end of the year, is finding a niche among surgeons and biomedical engineers. “When you put on the HoloLens, it’s like having X-ray vision,” says Karl West, an inventor and biomedical engineer at The Cleveland Clinic who developed a HoloLens app that is licensed to the Cleveland-based startup MediView. “You can see the anatomy inside the patient.” There are no actual X-rays; the images come from pre-operative CT scans or MRIs, projected holographically through a head-mounted display and coupled with ultrasound and electromagnetic tracking devices.

The Microsoft headset fits over the surgeon’s head and displays transparent images that hover in the surgeon’s field of vision. The apps align images of the patient’s anatomy with the real-life view. When it’s all set up, a surgeon can walk around the patient, viewing three-dimensional holographic images of internal structures—such as arteries, veins, and internal organs—from different vantage points.

The headset could mean surgeons never have to look away to get information while they perform a procedure. They can use voice commands or hand gestures to enlarge images or move information around. Even the patient’s vital signs can be projected onto the field of vision. “It’s going to allow surgeons to be able to do things faster, more efficiently, and more accurately in an environment that is real,” says West. MediView is currently testing its HoloLens app in liver, kidney, and other abdominal tumor ablation procedures.

In a typical procedure, an interventional radiologist uses a biopsy needle to deliver microwave energy that heats and destroys abnormal issue. Seeing the patient’s anatomy in 3D will provide “a markedly increased ability to target cancerous masses deep within the body,” West says. (MediView has been using HoloLens 1 and plans to upgrade to HoloLens 2 when it becomes available.)

In May, New York startup Medivis received FDA approval for its HoloLens-based imaging product, SurgicalAR, which can be used for holographic surgical planning. It is the brainchild of two medical residents at New York University—a neurosurgeon and radiologist, two technophiles who began brainstorming about how mixed reality devices could help reveal surgical anatomy.

Somehow amid the long clinical hours, Osamah Choudhry and Chris Morley formed a company. They now have partnership agreements with Microsoft, Magic Leap (for an educational use of its device), and Verizon (to test products in the Verizon 5G Lab) in New York). A 3D image helps surgeons visualize the best approach in a way that traditional two-dimensional images cannot, says Choudhry, Medivis’s CEO, who is in his seventh (and final) year of his residency. “That is the most accurate representation of the patient during the operation,” he says.

As surgical navigation systems improve, a primary challenge is lining up the data from the pre-op scans with the actual patient, who, of course, is breathing. A surgical probe can alter the anatomy. As the patient’s body and organs shift, the digitally imposed image needs to shift in unison.

One fix is to place trackers on the patient, which the AR system can detect. Whenever the markers shift, an algorithm makes real-time adjustments. “The accurate fusion of the medical images with physical patient is the most critical step in the acceptance of AR for medical procedures,” says Raul Uppot, an interventional radiologist at Massachusetts General Hospital in Boston, who has used virtual reality in teaching at Harvard Medical School.

Physicians who have donned a HoloLens and reached out to touch the 3D images are struck by how the headset could transform surgery. They can physically rotate a virtual image to see it from a different angle. They can give spoken commands. “You can say ‘show me the next scan,’ and it goes to the next scan,” says Uppot.

Holographic surgical guidance systems need FDA approval. MediView and Medivis hope to gain that and bring their products to market by 2021—opening up a new, more visionary world of surgery.

More Great WIRED Stories

Read More