南方医科大学学报

南方医科大学学报 ›› 2024, Vol. 44 ›› Issue (3): 585-593.doi: 10.12122/j.issn.1673-4254.2024.03.21

• • 上一篇    下一篇

基于多模态多示例学习的免疫介导性肾小球疾病自动分类方法

龙楷兴，翁丹仪，耿 舰，路艳蒙，周志涛，曹 蕾   

  1. 南方医科大学生物医学工程学院//广东省医学图像处理重点实验室//广东省医学成像与诊断技术工程实验室，基础医学院，中心实验室，广东 广州 510515；广州华银医学检验中心，广东 广州 510515
  • 出版日期:2024-03-20 发布日期:2024-04-03

Automatic classification of immune-mediated glomerular diseases based on multi-modal multi-instance learning

LONG Kaixing, WENG Danyi, GENG Jian, LU Yanmeng, ZHOU Zhitao, CAO Lei   

  1. School of Biomedical Engineering//Guangdong Provincial Key Laboratory of Medical Image Processing//Guangdong Provincial Engineering Laboratory for Medical Imaging and Diagnostic Technology, School of Basic Medical Sciences, Central Laboratory, Southern Medical University, Guangzhou 510515, China; Guangzhou Huayin Medical Laboratory Center, Guangzhou 510515, China
  • Online:2024-03-20 Published:2024-04-03

RichHTML

4

PDF

27

摘要： 目的 探讨如何利用多模态深度学习方法，联合光学显微镜（OM）、免疫荧光显微镜（IM）及透射电子显微镜（TEM）对应的3种图像进行免疫介导性肾小球疾病分类。方法 基于273例患者的病理图像进行回顾性研究，构建多模态多示例模型对3种免疫介导性的肾小球疾病——免疫球蛋白A肾病（IgAN）、膜性肾病（MN）、狼疮性肾炎（LN）进行分类。该模型采用示例水平的多示例学习（I-MIL）方法挑选患者的TEM图像并与同一患者的OM图像和IM图像进行多模态特征融合。通过该模型与单模态、双模态模型的比较，探究3种模态之间的不同组合形式以及模态特征融合方式的特性。结果 联合OM、IM以及TEM图像建立的多模态多示例模型准确率为（88.34±2.12）%，优于准确率为（87.08±4.25）%的最优的单模态模型，以及准确率为（87.92±3.06）%的最优的双模态模型。结论 本研究成功建立基于OM、IM及TEM三种模态图像的多模态多示例模型，并验证了采用多示例学习结合多模态学习方法对免疫介导性肾小球疾病分类的有效性。

关键词: 肾活检病理；肾小球疾病；深度学习；多模态融合；多示例学习

Abstract: Objective To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy (OM), immunofluorescence microscopy (IM), and transmission electron microscopy (TEM). Methods We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi-instance model for classification of 3 immune-mediated glomerular diseases, namely immunoglobulin A nephropathy (IgAN), membranous nephropathy (MN), and lupus nephritis (LN). This model adopts an instance-level multi-instance learning (I-MIL) method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient. By comparing this model with unimodal and bimodal models, we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion. Results The multi-modal multi-instance model combining OM, IM, and TEM images had a disease classification accuracy of (88.34±2.12)%, superior to that of the optimal unimodal model [(87.08±4.25)% ] and that of the optimal bimodal model [(87.92±3.06)% ]. Conclusion This multi-modal multi-instance model based on OM, IM, and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.

Key words: renal biopsy pathology; glomerular disease; deep learning; multi-modal fusion; multi-instance learning