Automated correlative light and electron microscopy (CLEM) using deep learning

Abstract

Correlative light and electron microscopy (CLEM) has emerged as a pivotal technique in the detailed structural analysis and precise identification of biological specimens by integrating fluorescence labelling with electron microscopy. This combined modality harnesses the advantages of both imaging techniques, allowing for enhanced visualization e.g. biological samples at multiple scales and resolutions. The advent of deep learning (DL) approaches has fundamentally transformed microscopy imaging within the biological sciences. By leveraging DL algorithms, it is now possible to extract intricate and meaningful features from complex imaging data, thereby enabling sophisticated analytical tasks that were previously unattainable. These tasks encompass a broad spectrum of challenges, including, but not limited to, image segmentation, classification, object detection, and resolution enhancement. The application of DL methodologies to these problems has demonstrated substantial improvements in accuracy and efficiency compared to conventional image processing techniques1,2. Within the scope of CLEM, one of the foremost challenges is the accurate registration of images acquired from disparate modalities. Image registration (IR) defined as the process of geometrically aligning two or more images of the same object or source, is essential for correlating and integrating complementary spatial information derived from different imaging systems. The inherent differences in image characteristics such as contrast, scale, and spatial resolution between light microscopy and electron microscopy necessitate sophisticated multi-dimensional and multi-modal registration techniques. Effective registration not only facilitates meaningful data fusion but also enables comprehensive characterization of biological and any other visual entities, including cells, tissues, and subcellular structures such as proteins. A variety of image registration methods have been developed to address this challenge3,4,5. Most existing techniques rely heavily on supervised learning frameworks, which, despite delivering high registration accuracy, require extensive manual intervention for the annotation or identification of key reference points or landmarks. This dependency on human input inherently limits the scalability and automation potential of these methods, presenting a bottleneck for high-throughput CLEM applications.