Contextual AI for tool-tissue interaction-related surgical tasks

Project reference: SIE_08_22
First supervisor:  Alejandro Granados Martinez
Second supervisor: Jonathan Shapey

Start date:  October  2022

Project summary:  The aim of this project is to design and implement surgical data science algorithms that are capable of assessing safe tool-tissue manipulation levels during (robotic-assisted) laparoscopic or endoscopic interventions to avoid surgical complications. The specific objectives of this proposal are as follows: 


Objective 1: Understand tool-tissue interaction from a phantom model and datasets 

Design an experimental study to estimate interaction forces from videos and to investigate the effect instruments have on tissue deformation on a phantom model and on datasets. 

Objective 2: Map tool-tissue interactions to surgical complications  

Develop data-driven algorithms to compute underlying spatial and temporal representations of videos that are correlated with surgical complications. 

Objective 3: Prototype a safe tool-tissue manipulation tool in the mock OR 

Implement a contextual Artificial Intelligence (AI) application that will process laparoscopic/endoscopic videos in real-time to warn surgeons of tissue manipulation levels. 

Project description

Scientific background 

In the UK over 1200 pituitary operations are performed every year with the endonasal endoscopic transsphenoidal surgical (eTSS) approach being the approach of choice for resecting most pituitary adenomas. Cerebrospinal fluid (CSF) rhinorrhea remains a frequent complication with potentially serious consequences including meningitis, pneumocephalus, low-pressure headaches, and prolonged admission. It is often difficult to determine the relationship between tumour and dura and excessive or inappropriate tissue-tool interaction is a significant contributing factor in causing an intraoperative CSF leak. 


The problem is that endoscopic/laparoscopic/robotic surgery distorts the natural force feedback surgeons use to control safe tissue manipulation. The limits of safe loading beyond which tissue damage occur are unknown; thus, identifying these limits could be used during computer-assisted surgery to minimise tissue damage. 


Simulation of forces acting on soft tissue is one way to understand safe loading. However, modelling tool-tissue interaction is time consuming, requires extensive expertise, and is based on assumptions that aim to simplify complexity. Recent AI-based methods have been proposed to improve safe tissue manipulation including: 1) policy-based methods that exploit all possible scenarios within a simulation environment to find optimal performance but the reality gap between simulated and real environments results in poor performance, and 2) vision-based force sensing (VBFS) techniques that estimate force values from visual inputs by mapping visual geometric information and applied force in a supervised and semi-supervised manner using temporal networks. 



This project will leverage a dataset to be collected as part of a clinical trial evaluating the use of a novel endonasal pituitary ultrasound probe. Using a dataset of intraoperative videos with matched clinical outcomes of patients undergoing eTSS we will develop machine learning algorithms as follows:  


WP1. Tool-tissue interaction (Y1) 

The student will design an experimental setting to simulate eTSS consisting of a phantom model, instruments and a prototype for force sensing using bespoke sensors embedded into the instruments. The CMR Versius system will also be used to acquire force sensing data on abstract phantoms and other simulated surgical procedures. The goal is to study tool-tissue interaction within a controlled environment. One additional output will evaluate the benefit of using  hyperrealism, i.e. change the domain from phantom to surgical, to generate synthetics videos. The student will then work on VBFS


WP2. Linking surgical data to complications (Y2) 

The student will work on specific eTSS surgical tasks that are likely to lead to complications of CSF rhinorrhea. The goal is to develop surgical data science algorithms that generate spatial and temporal tool-tissue interaction embeddings as underlying representations to map levels of tissue manipulation to surgical outcomes as a classification problem. Transformersvector quantisationsurgical workflow anticipation from instrument interaction, and task fingerprinting techniques will be proposed. 


WP3. Contextual AI (Y3) 

The student will design predictive models that warn surgeons of possible complications depending on the level of tissue manipulation captured from real-time video inputs. The goal is to evaluate the performance of the proposed application on a phantom model within a mock operating room. In addition to accuracy, the student will investigate explainability and generalisability of the proposed models. 



Figure 1: Endoscopic endonasal approach. a) T1w midline sagittal MRI illustrating the range of midline corridors to the skull base MRI; b) Intraoperative setup; c) Endoscopic view of the sphenoid sinus paragraph.