Computational extraction of semantic information from hyperspectral imaging for surgical guidance

Project reference: DTP_SIE_02 This project has already been allocated and is no longer available for applications

First supervisor: Tom Vercauteren 
Second supervisor: Christos Bergeles


Project summary:  Optimal outcomes in neuro-oncology surgery are hindered by the difficulty of differentiating between tumour and surrounding tissues during surgery. In this project, we will advance intraoperative computational hyperspectral imaging (HSI) to achieve real-time tissue characterisation and thus enhance the neurosurgeon's visualisation. HSI provides rich highdimensional intraoperative information but this cannot be directly visualised by the surgeon. This mandates the development of novel machine learning approaches for real-time semantic processing of HSI. We will consider interpretability by the surgeon of the computed information as a design constraint to deliver trustworthy learning-based semantic information extraction with algorithms that shine light on their decision process and inform the user when the data cannot be processed in a reliable manner.  


Project description: Clinical studies have demonstrated that patients who undergo surgery and have their CNS tumour radically removed have a significantly longer survival than those who are left with residual tumour tissue after surgery. Surgery is often the primary treatment and the aim of neurooncology surgery is to remove as much abnormal tissue as safely possible. Successful neurosurgery to remove brain tumours depends on achieving maximal safe tumour removal; avoiding damaging sensitive areas that undertake vital functions and preserving crucial nerves and blood vessels. However, even with the most advanced current techniques, it may still not be possible to reliably identify critical structures during surgery. The identification of tumour and surrounding tissue is currently still based on surgeons’ subjective visual assessment. 


During surgery, neuronavigation solutions can map preoperative information to the anatomy of the patient on the surgical table. However, navigation does not account for intraoperative changes. Interventional imaging and sensing, such as surgical microscopy, fluorescence imaging, point-based Raman spectroscopy, ultrasound and intra-operative MRI, may be used by the neurosurgeon, but partly due to stringent operative constraints, tissue differentiation remains challenging. Advanced optical imaging techniques provide a promising solution for intraoperative wide-field tissue characterisation, with the advantages of being non-contact, non-ionising and non-invasive. 
Hyperspectral imaging (HSI) is a camera-based optical imaging technique that exploits the ability to split light into multiple narrow spectral bands far beyond the conventional red/green/blue channels. It enables the acquisition of much richer information than what can be seen with the naked eye. While bearing rich high-dimensional information, the raw 2D+wavelength+time data that HSI produces is difficult to interpret for clinicians as it generates a temporal flow of three-dimensional information which cannot be simply displayed in an intuitive fashion on standard monitors (including “3D”/stereo displays). Combined with a general increase in the use of imaging, with real-time HSI, the clinical team will face a data deluge that needs to be addressed. 


We will develop real-time machine learning approaches to extract surgically relevant information from the massive high-dimensional HSI data stream and reduce the cognitive workload for the surgeon. Interpretability of the inference results will be put as a design constraint with methodologies rooted in Bayesian learning. Although snapshot HSI sensors allow to capture HSI data in real-time, their resolution is limited in terms of spectral and spatial domain, especially in comparison to modern endoscopic camera heads. Data is acquired with spatially interleaved and subsampled spectral bands. As an initial goal of the project, we will design a bespoke data-driven reconstruction algorithm to recover a fully sampled spatial and spectral response from the snapshot HSI. We will then develop algorithms for semantic interpretation of HSI video streams and pave the way for clinical translation of these solutions. 

Surgeons in theatre.jpg

©2020 by CDT SIE