Multi-tissue segmentation model of whole slide image of pancreatic cancer based on multi task and attention mechanism_Journal of Biomedical Engineering

Authors：

GAO Wei ¹ , JIANG Hui ² , JIAO Yiping ¹ , WANG Xiangxue ¹ ,  XU Jun ¹

1. Institute for AI in Medicine, School of Artificial Intelligence, Nanjing University of Information Science and Technology, Nanjing 210044, P. R. China;
2. Department of Pathology, Changhai Hospital Affiliated to Navy Medical University, Shanghai 200433. P.R.China;

Corresponding author：

XU Jun, Email: jxu@nuist.edu.cn

Keywords：

Multi-task learning; Whole slide image; Attention mechanism; Pancreatic cancer; Tissue segmentation

DOI：

10.7507/1001-5515.202211003

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Accurate segmentation of whole slide images is of great significance for the diagnosis of pancreatic cancer. However, developing an automatic model is challenging due to the complex content, limited samples, and high sample heterogeneity of pathological images. This paper presented a multi-tissue segmentation model for whole slide images of pancreatic cancer. We introduced an attention mechanism in building blocks, and designed a multi-task learning framework as well as proper auxiliary tasks to enhance model performance. The model was trained and tested with the pancreatic cancer pathological image dataset from Shanghai Changhai Hospital. And the data of TCGA, as an external independent validation cohort, was used for external validation. The F1 scores of the model exceeded 0.97 and 0.92 in the internal dataset and external dataset, respectively. Moreover, the generalization performance was also better than the baseline method significantly. These results demonstrate that the proposed model can accurately segment eight kinds of tissue regions in whole slide images of pancreatic cancer, which can provide reliable basis for clinical diagnosis.

Citation： GAO Wei, JIANG Hui, JIAO Yiping, WANG Xiangxue, XU Jun. Multi-tissue segmentation model of whole slide image of pancreatic cancer based on multi task and attention mechanism. Journal of Biomedical Engineering, 2023, 40(1): 70-78. doi: 10.7507/1001-5515.202211003 Copy

1.	Hu J X, Zhao C F, Chen W B, et al. Pancreatic cancer: A review of epidemiology, trend, and risk factors. World J Gastroenterol, 2021, 27(27): 4298-4321.
2.	Xu Y, Jia Z, Wang L B, et al. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinf, 2017, 18(1): 1-17.
3.	Yan R, Ren F, Wang Z, et al. Breast cancer histopathological image classification using a hybrid deep neural network. Methods, 2020, 173: 52-60.
4.	Elmore J G, Longton G M, Carney P A, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA, 2015, 313(11): 1122-1132.
5.	Campanella G, Hanna M G, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med, 2019, 25(8): 1301-1309.
6.	Xing F, Xie Y, Su H, et al. Deep learning in microscopy image analysis: A survey. IEEE Trans Neural Netw Learn Syst, 2017, 29(10): 4550-4568.
7.	支佩佩, 邓健志, 钟震霄. 基于卷积和注意力机制的医学细胞核图像分割网络. 生物医学工程学杂志, 2022, 39(4): 730-739.
8.	Wang X, Chen Y, Gao Y, et al. Predicting gastric cancer outcome from resected lymph node histopathology images using deep learning. Nat Commun, 2021, 12(1): 1-13.
9.	Fu H, Mi W M, Pan B J, et al. Automatic pancreatic ductal adenocarcinoma detection in whole slide images using deep convolutional neural networks. Front Oncol, 2021, 11: 665929.
10.	Janssen B V, Theijse R, van Roessel S, et al. Artificial intelligence-based segmentation of residual tumor in histopathology of pancreatic cancer after neoadjuvant treatment. Cancers, 2021, 13(20): 5089.
11.	Huang X, Ding L, Liu X, et al. Regulation of tumor microenvironment for pancreatic cancer therapy. Biomaterials, 2021, 270: 120680.
12.	金旭, 文可, 吕国锋, 等. 深度学习在组织病理学中的应用综述. 中国图像图形学报, 2020, 25(10): 1982-1993.
13.	Sirinukunwattana K, Alham N K, Verrill C, et al. Improving whole slide segmentation through visual context—a systematic study// International Conference on Medical Image Computing and Computer-Assisted Intervention. Granada: MICCAI Society, 2018: 192-200.
14.	Zhang R. Making convolutional networks shift-invariant again// International Conference on Machine Learning. Long Beach: IMLS, 2019: 7324-7334.
15.	Azulay A, Weiss Y. Why do deep convolutional networks generalize so poorly to small image transformations? J Mach Learn Res, 2018, 20(184): 1-25.
16.	Mikołajczyk A, Grochowski M. Data augmentation for improving deep learning in image classification problem// 2018 International Interdisciplinary PhD Workshop (IIPhDW). Świnoujście: IEEE, 2018: 117-122.
17.	张墺琦, 亢宇鑫, 武卓越, 等. 基于多尺度特征和注意力机制的肝脏组织病理图像语义分割网络. 模式识别与人工智能, 2021, 34(4): 375-384.
18.	Zhao B, Han C, Pan X, et al. RestainNet: a self-supervised digital re-stainer for stain normalization. Comput Electr Eng, 2022, 103: 108304.
19.	Xu Z. Computational models for automated histopathological assessment of colorectal liver metastasis progression. London: Queen Mary University of London, 2019.
20.	Ruder S. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv, 2017: 1706.05098.
21.	Zhang Y, Li H, Du J, et al. 3D multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification. IEEE Trans Med Imaging, 2021, 40(6): 1618-1631.
22.	Woo S, Park J, Lee J Y, et al. CBAM: Convolutional block attention module// Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 3-19.
23.	Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks// International Conference on Machine Learning. Long Beach: IMLS, 2019: 6105-6114.
24.	Hu J, Shen L, Sun G. Squeeze-and-excitation networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
25.	Cui Y, Jia M, Lin T Y, et al. Classbalanced loss based on effective number of samples// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 9268-9277.
26.	Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection// Proceedings of the IEEE International Conference on Computer Vision. Hawaii: IEEE, 2017: 2980-2988.
27.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
28.	Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 1492-1500.
29.	Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 4700-4708.
30.	Radosavovic I, Kosaraju R P, Girshick R, et al. Designing network design spaces// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 10428-10436.

1. Hu J X, Zhao C F, Chen W B, et al. Pancreatic cancer: A review of epidemiology, trend, and risk factors. World J Gastroenterol, 2021, 27(27): 4298-4321.
2. Xu Y, Jia Z, Wang L B, et al. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinf, 2017, 18(1): 1-17.
3. Yan R, Ren F, Wang Z, et al. Breast cancer histopathological image classification using a hybrid deep neural network. Methods, 2020, 173: 52-60.
4. Elmore J G, Longton G M, Carney P A, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA, 2015, 313(11): 1122-1132.
5. Campanella G, Hanna M G, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med, 2019, 25(8): 1301-1309.
6. Xing F, Xie Y, Su H, et al. Deep learning in microscopy image analysis: A survey. IEEE Trans Neural Netw Learn Syst, 2017, 29(10): 4550-4568.
7. 支佩佩, 邓健志, 钟震霄. 基于卷积和注意力机制的医学细胞核图像分割网络. 生物医学工程学杂志, 2022, 39(4): 730-739.
8. Wang X, Chen Y, Gao Y, et al. Predicting gastric cancer outcome from resected lymph node histopathology images using deep learning. Nat Commun, 2021, 12(1): 1-13.
9. Fu H, Mi W M, Pan B J, et al. Automatic pancreatic ductal adenocarcinoma detection in whole slide images using deep convolutional neural networks. Front Oncol, 2021, 11: 665929.
10. Janssen B V, Theijse R, van Roessel S, et al. Artificial intelligence-based segmentation of residual tumor in histopathology of pancreatic cancer after neoadjuvant treatment. Cancers, 2021, 13(20): 5089.
11. Huang X, Ding L, Liu X, et al. Regulation of tumor microenvironment for pancreatic cancer therapy. Biomaterials, 2021, 270: 120680.
12. 金旭, 文可, 吕国锋, 等. 深度学习在组织病理学中的应用综述. 中国图像图形学报, 2020, 25(10): 1982-1993.
13. Sirinukunwattana K, Alham N K, Verrill C, et al. Improving whole slide segmentation through visual context—a systematic study// International Conference on Medical Image Computing and Computer-Assisted Intervention. Granada: MICCAI Society, 2018: 192-200.
14. Zhang R. Making convolutional networks shift-invariant again// International Conference on Machine Learning. Long Beach: IMLS, 2019: 7324-7334.
15. Azulay A, Weiss Y. Why do deep convolutional networks generalize so poorly to small image transformations? J Mach Learn Res, 2018, 20(184): 1-25.
16. Mikołajczyk A, Grochowski M. Data augmentation for improving deep learning in image classification problem// 2018 International Interdisciplinary PhD Workshop (IIPhDW). Świnoujście: IEEE, 2018: 117-122.
17. 张墺琦, 亢宇鑫, 武卓越, 等. 基于多尺度特征和注意力机制的肝脏组织病理图像语义分割网络. 模式识别与人工智能, 2021, 34(4): 375-384.
18. Zhao B, Han C, Pan X, et al. RestainNet: a self-supervised digital re-stainer for stain normalization. Comput Electr Eng, 2022, 103: 108304.
19. Xu Z. Computational models for automated histopathological assessment of colorectal liver metastasis progression. London: Queen Mary University of London, 2019.
20. Ruder S. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv, 2017: 1706.05098.
21. Zhang Y, Li H, Du J, et al. 3D multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification. IEEE Trans Med Imaging, 2021, 40(6): 1618-1631.
22. Woo S, Park J, Lee J Y, et al. CBAM: Convolutional block attention module// Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 3-19.
23. Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks// International Conference on Machine Learning. Long Beach: IMLS, 2019: 6105-6114.
24. Hu J, Shen L, Sun G. Squeeze-and-excitation networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
25. Cui Y, Jia M, Lin T Y, et al. Classbalanced loss based on effective number of samples// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 9268-9277.
26. Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection// Proceedings of the IEEE International Conference on Computer Vision. Hawaii: IEEE, 2017: 2980-2988.
27. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
28. Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 1492-1500.
29. Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 4700-4708.
30. Radosavovic I, Kosaraju R P, Girshick R, et al. Designing network design spaces// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 10428-10436.

Journal of Biomedical Engineering

Multi-tissue segmentation model of whole slide image of pancreatic cancer based on multi task and attention mechanism

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content