Abstract: Objective To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy (OM), immunofluorescence microscopy (IM), and transmission electron microscopy (TEM). Methods We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi-instance model for classification of 3 immune-mediated glomerular diseases, namely immunoglobulin A nephropathy (IgAN), membranous nephropathy (MN), and lupus nephritis (LN). This model adopts an instance-level multi-instance learning (I-MIL) method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient. By comparing this model with unimodal and bimodal models, we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion. Results The multi-modal multi-instance model combining OM, IM, and TEM images had a disease classification accuracy of (88.34±2.12)%, superior to that of the optimal unimodal model [(87.08±4.25)% ] and that of the optimal bimodal model [(87.92±3.06)% ]. Conclusion This multi-modal multi-instance model based on OM, IM, and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.

Key words: renal biopsy pathology; glomerular disease; deep learning; multi-modal fusion; multi-instance learning