Joint learning of stereo vision reconstruction and hyperspectral imaging upsampling for binocular surgical guidance

Project summary: Optimal outcomes in oncology surgery are hindered by the difficulty of differentiating between tumour and surrounding tissues during surgery. Real-time hyperspectral imaging (HSI)provides rich high-dimensional intraoperative information that has the potential to significantly improve tissue characterisation and thus benefit patient outcomes. Yet taking full advantage of HSI data in clinical indication performed under binocular guidance (e.g. microsurgery and robotic surgery) poses several methodological challenges which this project aims to address. Real-time HSI sensors are limited in the spatial resolution they can capture. This further impacts the usefulness of such HSI sensors in multi-view capture settings. In this project, we will take advantage of a stereo-vision combination with a high-resolution RGB viewpoint and a HSI viewpoint. The student will develop bespoke learning-based computational approaches to reconstruct high-quality 3D scenes combining the intuitiveness of RGB guidance and the rich semantic information extracted from HSI.
 

Background:

Real-time tissue characterisation has the potential to transform surgery across many clinical indications. This project will advance the field by embedding,for the first time, hyperspectral imaging capability in multi-view surgical guidance. Two use cases will be targeted in the PhD to address the main embodiments of surgical binocular vision: surgical microscope and robotic stereo laparoscope.

 

Neuro-oncology.  Each year, in the UK, approximately 11,500 patients are diagnosed with a primary central nervous system (CNS) tumour while about 59,000 others are diagnosed with secondary brain tumours (Ferlay 2018). Brain tumour survival remains low, accruing the need to improve outcomes for affected patients. The vast majority of neuro-oncology surgeries are performed under guidance of a white light binocular surgical microscope.

 

Gastrointestinal surgery.  Each year, approximately 26,000 colorectal operations are performed in the UK (Wan 2014). Anastomotic Leakage (AL) is one of the most serious complications arising from colorectal surgery, occurring in 8-15% cases, with associated mortality rates as high as 35% (Turrentine 2015). Ischaemia is one of the main predisposing factors for AL but is difficult to assess intraoperatively (Hirst 2014). With the trend of performing minimally invasive surgery robotically, more and more gastrointestinal surgeries are being performed under guidance of a stereo laparoscope

​

Project plan:

The student will design novel computational approaches to optimally embed hyperspectral imaging in binocular surgical vison, there by providing novel tissue characterisation capabilities. Real-time mosaic sensor based hyperspectral imaging (HSI) is a camera-based optical imaging technique that split light into multiple narrow spectral bands across the pixels of the sensors.It enables the acquisition of much richer information than the RGB information that can be seen with the naked eye or standard cameras but can only be done in real-time with a restricted spatial resolution. It also faces additional challenges related to the unknown geometry of the scene which hinders some of the quantitative spectral measurement capabilities. The project plan aims at addressing these challenges. We hereby concisely list the deliverables of the PhD and associated timelines.

 

Year 1

• Set up of a first stereo vision rig with a high-resolution(4k)RGB camera and a mosaic sensor(1080p across channels) HSI camera separated by a baseline

• Geometric calibration of the imaging system

• Acquisition software to capture synchronised data stream

• Implementation and validation of learning-based dense stereo RGB reconstruction using the high-resolution RGB combined with RGB-from-HSI (as reconstructed using a state-of-the-art standalone HSI super-resolution pipeline)

• Phantom creation for validation purposes

 

Year 2

• Refinement of the acquisition rig

• Development of RGB-guided HSI super-resolution approaches to achieve 4K HSI spatial reconstruction(factor 2 beyond the state of the art)

• Implementation of the RGB-guided upsampling to improve dense depth reconstruction

• Publication

​

Year 3

• Algorithm development to use the recovered depth information to achieve better quantitative spectral measurement

• Software stack optimisation and development of end-to-end approaches to achieve (near) real-time reconstruction and spectral calibration

• Publication

 

Year 4

• Embedding of HSI-derived semantic segmentation and tissue features (e.g. perfusion) in the binocular reconstruction

• Phantom refinement

• Thesis write-up and publication

 

​