An augmented reality microscope with real-time AI for cancer detection


Processed tissue slide viewing and assessment is crucial in determining the diagnosis and cancer staging. This protocol is thus instrumental in deciding the treatment therapy for a patient. Applications of deep learning and AI in the medical fields such as dermatology, radiology, ophthalmology, and pathology have shown great potential in providing great accuracy in diagnosis.

Although AI promises to provide quality healthcare, the cost of slide digitization and lack of infrastructure for AI deployments remain as barriers for widespread adoption of digital pathology in clinical settings. Google recently published a paper in Nature demonstrating the prediction of metastatic breast cancer in lymph nodes using convolutional neural networks at an accuracy comparable to pathologists. Here Google proposed a low-cost solution for the application of AI in real-world clinics — so-called augmented-reality microscope (ARM). The ARM can be retrofitted on top of an existing light microscope found in hospitals and clinics around the world. This use of affordable and readily-available components can completely diminish the need for expensive whole slide imaging for analyzing tissue slides.

ARM system is designed with three major components:

  1. An augmented microscope- a bright-field microscope (Nikon Eclipse Ni-U) augmented with two custom modules. (a)A camera that captures high-resolution images of the current field of view (FOV). (b) microdisplay that superimposes digital information into the original optical path.
  2. A computer with a high-speed image grabber (BitFlow CYT) and an accelerated computing unit (NVidia Titan Xp GPU).
  3. Trained deep learning algorithms. The training phase involves training an algorithm using a large dataset, while the inference phase involves processing an image with the trained deep learning algorithm. To accelerate inference they custom built an architecture called -InceptionV3-FCN, where they applied the concept of fully convolutional networks (FCN) to the deep learning architecture of InceptionV3.
Hardware components of ARM: A digital camera retrofitted on a microscope captures the same field of view (FoV) as the user and passes the image to an attached compute unit capable of running real-time inference of a machine learning model. A deep learning algorithm processes each image to produce an inference output (such as a heatmap) with an accelerated computing unit. The inference output is fed back into a custom AR display which is inline with the ocular lens and projects the model output on the same plane as the slide.

The ARM can give a wide range of visual feedback, together with text, arrows, contours, heatmaps, and animations, and is capable of running many varieties of machine learning algorithms aimed toward solving totally different issues like object detection, quantification, or classification. To investigate the potential of ARM as a platform, the authors tested the algorithms for two clinical tasks: the detection of metastatic breast cancer in lymph nodes and the identification of prostate cancer in prostatectomy specimens.


These models can run at magnifications between 4–40x, and the result of a given model is displayed by outlining detected tumor regions with a green contour. These contours help draw the pathologist’s attention to areas of interest without obscuring the underlying tumor cell appearance.

Sample views through the lens of the ARM with green outlines highlighting the predicted tumor region for lymph node metastasis detection at ×4, ×10, ×20 and ×40.

Both cancer models were originally trained on images from a whole slide scanner with a different optical configuration. The training data for breast cancer was taken from here, and here for prostate cancer. The models performed remarkably well on the ARM with no extra re-training. The lymph node metastasis model had an area-under-the-curve (AUC) of 0.98 and prostate cancer model had an AUC of 0.96 for cancer detection in the field of view (FoV) when running on the ARM, only slightly decreased performance than obtained on WSI.

The ARM can be used for a wide range of application with AI as well as AR capabilities. Clinical application of ARM can include highly subjective tasks like stain quantification, disruptive tasks such as estimation of size measurements using a physical ruler, tasks that take place in low-resource environments and require (but lack) skilled personnel — such as infectious disease detection (for example, malaria or tuberculosis) and tedious tasks such as cell or mitosis counting. The ARM could also be useful as a teaching tool by leveraging reverse image search tools that can help trainees quickly search reference resources and find what histologic feature they are looking at. The ARM could be used in combination with the digital workflow at the hospitals in the near future for tasks such as cytology, fluorescent imaging, or intra-operative frozen sections where scanners still face major challenges or where rapid turnaround is required. Although cheaper than “conventional whole-slide scanners” by about one or two magnitudes, it will still be a challenge to adapt to expensive accelerated computing units in clinical settings. Nevertheless, the ARM can help accelerate the adoption of machine learning for positive impact around the world.

Show More

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button

Adblock Detected

Please consider supporting us by disabling your ad blocker