Image stitching for real-time laparoscopic hyperspectral imaging

Abstract

Abstract Hyperspectral imaging (HSI) shows significant promise in the medical field for tissue detection and perfusion assessment. To extend its application to intraoperative diagnosis, laparoscopic cameras combining a high resolution color video and simultaneous HSI were developed. Spatial scanning in these cameras is performed through a push-broom motor driving a line-scan spectrograph. However, long acquisition times and the necessity of absolute immobility for patient and operator currently limit its usability in the operating room. To provide a hyperspectral acquisition alternative to the traditional push-broom motor approach, we have developed an HSI stitching pipeline that enables freehand line scanning. Our method utilizes the dual recording capability of the camera, which has both an RGB and an HSI sensor. It applies the transformations observed in the RGB video to the corresponding HSI data, then seamlessly merges this data to create a coherent panorama. This allows operators to visualize hyperspectral data as an incrementally expanding overlay on the color video by scanning the scene with the laparoscope. The pipeline evaluation confirms the generation of globally consistent and well-interpretable panoramas with a high level of detail. The registration error is not only comparable to the push-broom method but also corresponds to a real-world error of less than 0.4 mm in 95 % of the cases. Therefore, the proposed method enhances the practicability of intraoperative hyperspectral imaging by providing a dynamic, video-like experience of HSI visualizations.