Colorectal cancer is one of the most common malignant tumors of the digestive system in the world. Lymph node metastasis (LNM) status is a key indicator for tumor staging (N Stage), surgical planning, prognosis assessment and individualized treatment. The risk of recurrence in patients with positive lymph node metastasis is significantly increased, which directly determines the patient's choice of adjuvant treatment. Although preoperative imaging examinations can assess LNM status, the final diagnosis must rely on postoperative histopathological diagnosis. In recent years, some pathological omics, radiomics and pathological basic models have shown advantages in LNM status prediction tasks. However, most AI prediction models are limited to single-modal models, which cannot simultaneously integrate macroscopic imaging features and microscopic tissue pathological morphology, and have limitations in clinical application. They have not been verified by pathologists through user research. Therefore, there is an urgent need to develop multi-modal intelligent prediction models that have been verified in multiple dimensions to facilitate the accurate assessment of lymph node metastasis in colorectal cancer.

This study included 3 large tertiary hospitals in China and 1 public data set (TCGA), collecting retrospective and prospective patient data of 1501 cases, including an additional 110 gastric cancer exploratory cohorts for cross-cancer generalization verification. The model uses a dedicated pre-trained encoder to extract high-dimensional features of H&E whole slice images (WSI), image CT, and clinical text respectively. ； Introducing the Missing Modality Aggregator to solve the problem of missing clinical modality (CT) data ； The SIM-MIL (Stochastic Instance Mask Multi-Instance Learning) module is proposed to complete multi-modal feature aggregation and complete patient-level lymph node metastasis prediction.

▲Research summary diagram (queue, algorithm technology roadmap, performance display)

Research results show that DeepLNM’s internal AUC reaches 0.922； The AUCs of the two external validation cohorts were 0.872 and 0.848 respectively. ； The prospective cohort AUC can reach 0.845, demonstrating robust cross-center generalization performance. Compared with single-modal models, multi-modal models perform better than single-modal models that only rely on pathology, CT or clinical text, and the benchmark is better than the existing 7 MILs. In the exploratory verification for gastric cancer, the AUC of DeepLNM was 0.793. Although the performance has declined due to limitations such as cancer type and morphological differences, the decision curve analysis (DCA) has a net benefit and is better than other baseline models, and has certain migration potential across cancer types. Pathological heat map analysis showed that the areas of high concern to the model in WSI predicted to be LNM positive were poorly differentiated tumors, necrosis and other areas, while in WSI predicted to be negative the model paid high attention to areas such as lymphatic infiltration and well-differentiated glands, which has good interpretability. In clinical verification, user study results showed that with the assistance of DeepLNM, the overall diagnostic efficiency of pathologists was improved. Among them, the diagnostic accuracy of junior pathologists increased from 95.0% to 97.3%, and the film reading efficiency and diagnostic confidence were improved.

The DeepLNM constructed in this study is the first multi-modal model of colorectal cancer lymph node metastasis that has been retrospectively and prospectively verified in multiple dimensions. It takes into account model performance, interpretability and clinical practicability, and completes the research chain of "multi-center review-prospective verification-generalization exploration-interpretability analysis-clinical user verification". DeepLNM is expected to alleviate the diagnostic pressure caused by the shortage of pathological talents and further promote the clinical implementation of pathological AI.

"Advanced Science" is a well-known open source journal under Wiley Publishing House (impact factor 14.1, JCR Q1 area). It is committed to publishing top-level basic and applied research results in the fields of materials science, physics, chemistry, medicine, life sciences and engineering, and has high international academic influence.