Cross modal translation of magnetic resonance imaging and computed tomography images based on diffusion generative adversarial networks_Journal of Biomedical Engineering

Authors：

 SHAO Hong , JING Yixuan , CUI Wencheng

School of Information Science and Engineering, Shenyang University of Technology, Shenyang 110870, P. R. China;

Corresponding author：

SHAO Hong, Email: jyxswag@163.com

Keywords：

Cross-modal medical image translation; Diffusion generated adversarial network; Frequency decomposition; Computed tomography; Magnetic resonance imaging

DOI：

10.7507/1001-5515.202404056

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

To address the issues of difficulty in preserving anatomical structures, low realism of generated images, and loss of high-frequency image information in medical image cross-modal translation, this paper proposes a medical image cross-modal translation method based on diffusion generative adversarial networks . First, an unsupervised translation module is used to convert magnetic resonance imaging (MRI) into pseudo-computed tomography (CT) images. Subsequently, a nonlinear frequency decomposition module is used to extract high-frequency CT images. Finally, the pseudo-CT image is input into the forward process, while the high-frequency CT image as a conditional input is used to guide the reverse process to generate the final CT image. The proposed model is evaluated on the SynthRAD2023 dataset, which is used for CT image generation for radiotherapy planning. The generated brain CT images achieve a Fréchet Inception Distance (FID) score of 33.159 7, a structure similarity index measure (SSIM) of 89.84%, a peak signal-to-noise ratio (PSNR) of 35.596 5 dB, and a mean squared error (MSE) of 17.873 9. The generated pelvic CT images yield an FID score of 33.951 6, a structural similarity index of 91.30%, a PSNR of 34.870 7 dB, and an MSE of 17.465 8. Experimental results show that the proposed model generates highly realistic CT images while preserving anatomical accuracy as much as possible. The transformed CT images can be effectively used in radiotherapy planning, further enhancing diagnostic efficiency.

1.	Andreasen D, Edmund J M, Zografos V, et al. Computed tomography synthesis from magnetic resonance images in the pelvis using multiple random forests and auto-context features//Medical Imaging 2016: Image Processing. San Diego: SPIE, 2016, 9784: 323-330.
2.	Yang X, Lei Y, Shu H K, et al. Pseudo CT estimation from MRI using patch-based random forest//Medical Imaging 2017: Image Processing. San Diego: SPIE, 2017, 10133: 775-782.
3.	Chen Z, Qiu T, Tian Y, et al. Automated brain structures segmentation from PET/CT images based on landmark-constrained dual-modality atlas registration. Physics in Medicine & Biology, 2021, 66(9): 095003.
4.	Newton M D, Junginger L, Maerz T. Automated MicroCT-based bone and articular cartilage analysis using iterative shape averaging and atlas-based registration. Bone, 2020, 137: 115417.
5.	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks. Communications of the ACM, 2020, 63(11): 139-144.
6.	Nie D, Trullo R, Lian J, et al. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 2018, 65(12): 2720-2730.
7.	Dar S U, Yurt M, Karacan L, et al. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Transactions on Medical Imaging, 2019, 38(10): 2375-2388.
8.	Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks//IEEE International Conference on Computer Vision (ICCV), Venice: IEEE. 2017, 24: 2242-2251.
9.	Jin C B, Kim H, Liu M, et al. Deep CT to MR synthesis using paired and unpaired data. Sensors, 2019, 19(10): 2361.
10.	Peng Y, Chen S, Qin A, et al. Magnetic resonance-based synthetic computed tomography images generated using generative adversarial networks for nasopharyngeal carcinoma radiotherapy treatment planning. Radiotherapy and Oncology, 2020, 150(3): 217-224.
11.	Fetty L, Löfstedt T, Heilemann G, et al. Investigating conditional GAN performance with different generator architectures, an ensemble model, and different MR scanners for MR-sCT conversion. Physics in Medicine & Biology, 2020, 65(10): 105004.
12.	Wang J, Wu Q M J, Pourpanah F. DC-cycleGAN: bidirectional CT-to-MR synthesis from unpaired data. Computerized Medical Imaging and Graphics, 2023, 108: 102249.
13.	Wolterink J M, Dinkla A M, Savenije M H F, et al. Deep MR to CT synthesis using unpaired data//Simulation and Synthesis in Medical Imaging: Second International Workshop. Québec: MICCAI, 2017: 14-23.
14.	蔡昕, 侯学文, 杨光, 等. 生成对抗网络在磁共振图像重建领域的应用. 生物医学工程学杂志, 2023, 40(3): 582-588.
15.	Bi L, Kim J, Kumar A, et al. Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs). Lecture Notes in Computer Science, 2017, 1055: 43-51.
16.	Chartsias A, Joyce T, Dharmakumar R, et al. Adversarial image synthesis for unpaired multi-modal cardiac data//Simulation and Synthesis in Medical Imaging: Second International Workshop. Québec: MICCAI, 2017: 3-13.
17.	Kingma D P, Welling M. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 2019, 12(4): 307-392.
18.	Liu M Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks. Advances in Neural Information Processing Systems, 2017, 30(3): 5-11.
19.	Li Y, Shao H C, Qian X, et al. FDDM: unsupervised medical image translation with a frequency-decoupled diffusion model. arXiv preprint, 2023, arXiv: 2311.12070.
20.	潘丹, 贾龙飞, 曾安, 等. 生成式对抗网络在医学图像处理中的应用. 生物医学工程学杂志, 2018, 35(6): 970-976.
21.	Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
22.	Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
23.	Meng C, Song Y, Song J, et al. Sdedit: image synthesis and editing with stochastic differential equations. arXiv preprint, 2021, arXiv: 2108.01073.
24.	Zhao M, Bao F, Li C, et al. EGSDE: unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 2022, 35: 3609-3623.
25.	Özbey M, Dalmaz O, Dar S U H, et al. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging, 2023, 42(12): 3524-3539.
26.	Li Y, Shao H C, Liang X, et al. Zero-shot medical image translation via frequency-guided diffusion models. IEEE Transactions on Medical Imaging, 2023, 43(3): 980-993.
27.	Jiang L, Dai B, Wu W, et al. Focal frequency loss for image reconstruction and synthesis//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 13919-13929.
28.	Xiao Z, Kreis K, Vahdat A. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint, 2021, arXiv: 2112.07804.
29.	Shannon M, Poole B, Mariooryad S, et al. Non-saturating GAN training as divergence minimization. arXiv preprint, 2020, arXiv: 2010.08029.
30.	Thummerer A, van der Bijl E, Galapon Jr A, et al. SynthRAD2023 Grand Challenge dataset: generating synthetic CT for radiotherapy. Medical Physics, 2023, 50(7): 4664-4674.

1. Andreasen D, Edmund J M, Zografos V, et al. Computed tomography synthesis from magnetic resonance images in the pelvis using multiple random forests and auto-context features//Medical Imaging 2016: Image Processing. San Diego: SPIE, 2016, 9784: 323-330.
2. Yang X, Lei Y, Shu H K, et al. Pseudo CT estimation from MRI using patch-based random forest//Medical Imaging 2017: Image Processing. San Diego: SPIE, 2017, 10133: 775-782.
3. Chen Z, Qiu T, Tian Y, et al. Automated brain structures segmentation from PET/CT images based on landmark-constrained dual-modality atlas registration. Physics in Medicine & Biology, 2021, 66(9): 095003.
4. Newton M D, Junginger L, Maerz T. Automated MicroCT-based bone and articular cartilage analysis using iterative shape averaging and atlas-based registration. Bone, 2020, 137: 115417.
5. Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks. Communications of the ACM, 2020, 63(11): 139-144.
6. Nie D, Trullo R, Lian J, et al. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 2018, 65(12): 2720-2730.
7. Dar S U, Yurt M, Karacan L, et al. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Transactions on Medical Imaging, 2019, 38(10): 2375-2388.
8. Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks//IEEE International Conference on Computer Vision (ICCV), Venice: IEEE. 2017, 24: 2242-2251.
9. Jin C B, Kim H, Liu M, et al. Deep CT to MR synthesis using paired and unpaired data. Sensors, 2019, 19(10): 2361.
10. Peng Y, Chen S, Qin A, et al. Magnetic resonance-based synthetic computed tomography images generated using generative adversarial networks for nasopharyngeal carcinoma radiotherapy treatment planning. Radiotherapy and Oncology, 2020, 150(3): 217-224.
11. Fetty L, Löfstedt T, Heilemann G, et al. Investigating conditional GAN performance with different generator architectures, an ensemble model, and different MR scanners for MR-sCT conversion. Physics in Medicine & Biology, 2020, 65(10): 105004.
12. Wang J, Wu Q M J, Pourpanah F. DC-cycleGAN: bidirectional CT-to-MR synthesis from unpaired data. Computerized Medical Imaging and Graphics, 2023, 108: 102249.
13. Wolterink J M, Dinkla A M, Savenije M H F, et al. Deep MR to CT synthesis using unpaired data//Simulation and Synthesis in Medical Imaging: Second International Workshop. Québec: MICCAI, 2017: 14-23.
14. 蔡昕, 侯学文, 杨光, 等. 生成对抗网络在磁共振图像重建领域的应用. 生物医学工程学杂志, 2023, 40(3): 582-588.
15. Bi L, Kim J, Kumar A, et al. Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs). Lecture Notes in Computer Science, 2017, 1055: 43-51.
16. Chartsias A, Joyce T, Dharmakumar R, et al. Adversarial image synthesis for unpaired multi-modal cardiac data//Simulation and Synthesis in Medical Imaging: Second International Workshop. Québec: MICCAI, 2017: 3-13.
17. Kingma D P, Welling M. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 2019, 12(4): 307-392.
18. Liu M Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks. Advances in Neural Information Processing Systems, 2017, 30(3): 5-11.
19. Li Y, Shao H C, Qian X, et al. FDDM: unsupervised medical image translation with a frequency-decoupled diffusion model. arXiv preprint, 2023, arXiv: 2311.12070.
20. 潘丹, 贾龙飞, 曾安, 等. 生成式对抗网络在医学图像处理中的应用. 生物医学工程学杂志, 2018, 35(6): 970-976.
21. Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
22. Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
23. Meng C, Song Y, Song J, et al. Sdedit: image synthesis and editing with stochastic differential equations. arXiv preprint, 2021, arXiv: 2108.01073.
24. Zhao M, Bao F, Li C, et al. EGSDE: unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 2022, 35: 3609-3623.
25. Özbey M, Dalmaz O, Dar S U H, et al. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging, 2023, 42(12): 3524-3539.
26. Li Y, Shao H C, Liang X, et al. Zero-shot medical image translation via frequency-guided diffusion models. IEEE Transactions on Medical Imaging, 2023, 43(3): 980-993.
27. Jiang L, Dai B, Wu W, et al. Focal frequency loss for image reconstruction and synthesis//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 13919-13929.
28. Xiao Z, Kreis K, Vahdat A. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint, 2021, arXiv: 2112.07804.
29. Shannon M, Poole B, Mariooryad S, et al. Non-saturating GAN training as divergence minimization. arXiv preprint, 2020, arXiv: 2010.08029.
30. Thummerer A, van der Bijl E, Galapon Jr A, et al. SynthRAD2023 Grand Challenge dataset: generating synthetic CT for radiotherapy. Medical Physics, 2023, 50(7): 4664-4674.

Journal of Biomedical Engineering

Latest ArticlesCross modal translation of magnetic resonance imaging and computed tomography images based on diffusion generative adversarial networks

Abstract Full text Figures/Tables Video References Cited by

Format

Content