Skip to main content
Top

17-06-2024 | Pathology

Prediction of Tumor-Associated Macrophages and Immunotherapy Benefits Using Weakly Supervised Contrastive Learning in Breast Cancer Pathology Images

Authors: Guobang Yu, Yi Zuo, Bin Wang, Hui Liu

Published in: Journal of Imaging Informatics in Medicine

Login to get access

Abstract

The efficacy of immune checkpoint inhibitors is significantly influenced by the tumor immune microenvironment (TIME). RNA sequencing of tumor tissue can offer valuable insights into TIME, but its high cost and long turnaround time seriously restrict its utility in routine clinical examinations. Several recent studies have suggested that ultrahigh-resolution pathology images can infer cellular and molecular characteristics. However, few study pay attention to the quantitative estimation of various tumor infiltration immune cells from pathology images. In this paper, we integrated contrastive learning and weakly supervised learning to infer tumor-associated macrophages and potential immunotherapy benefit from whole slide images (WSIs) of H &E stained pathological sections. We split the high-resolution WSIs into tiles and then apply contrastive learning to extract features of each tile. After aggregating the features at the tile level, we employ weak supervisory signals to fine-tune the encoder for various downstream tasks. Comprehensive experiments on two independent breast cancer cohorts and spatial transcriptomics data demonstrate that the computational pathological features accurately predict the proportion of tumor-infiltrating immune cells, particularly the infiltration level of macrophages, as well as the immune subtypes and potential immunotherapy benefit. These findings demonstrate that our model effectively captures pathological features beyond human vision, establishing a mapping relationship between cellular compositions and histological morphology, thus expanding the clinical applications of digital pathology images.
Appendix
Available only for authorised users
Literature
1.
go back to reference Shiravand, Y. et al. Immune checkpoint inhibitors in cancer therapy. Current Oncology 29, 3044–3060 (2022). Shiravand, Y. et al. Immune checkpoint inhibitors in cancer therapy. Current Oncology 29, 3044–3060 (2022).
2.
go back to reference Wang, Z. et al. Niraparib activates interferon signaling and potentiates anti-pd-1 antibody efficacy in tumor models. Scientific Reports 9, 1853 (2019). Wang, Z. et al. Niraparib activates interferon signaling and potentiates anti-pd-1 antibody efficacy in tumor models. Scientific Reports 9, 1853 (2019).
3.
go back to reference Qianhui, X., Shaohuai, C., Yuanbo, H. & Wen, H. Landscape of immune microenvironment under immune cell infiltration pattern in breast cancer. Frontiers in Immunology 12, 711433 (2021). Qianhui, X., Shaohuai, C., Yuanbo, H. & Wen, H. Landscape of immune microenvironment under immune cell infiltration pattern in breast cancer. Frontiers in Immunology 12, 711433 (2021).
4.
go back to reference Fuchou, T. et al. mrna-seq whole-transcriptome analysis of a single cell. Nature Methods 6, 377–382 (2009). Fuchou, T. et al. mrna-seq whole-transcriptome analysis of a single cell. Nature Methods 6, 377–382 (2009).
5.
go back to reference Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nat Methods 12, 453–457 (2015). Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nat Methods 12, 453–457 (2015).
6.
go back to reference Subramanian, A. et al. Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences 102, 15545–15550 (2005). Subramanian, A. et al. Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences 102, 15545–15550 (2005).
7.
go back to reference Yoshihara, K., Shahmoradgoli, M., Martínez, E. et al. Inferring tumour purity and stromal and immune cell admixture from expression data. Nature Communications 4, 2612 (2013). Yoshihara, K., Shahmoradgoli, M., Martínez, E. et al. Inferring tumour purity and stromal and immune cell admixture from expression data. Nature Communications 4, 2612 (2013).
8.
go back to reference Yann, L., Yoshua., B. & Geoffrey, H. Deep learning. Nature 521, 436–444 (2015). Yann, L., Yoshua., B. & Geoffrey, H. Deep learning. Nature 521, 436–444 (2015).
9.
go back to reference He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (2016). He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (2016).
10.
go back to reference Deng, J. et al. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition 248–255 (2009). Deng, J. et al. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition 248–255 (2009).
11.
go back to reference Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun.ACM 60, 84–90 (2017). Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun.ACM 60, 84–90 (2017).
12.
go back to reference Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).
13.
go back to reference Wang, S. et al. Computational staining of pathology images to study the tumor microenvironment in lung cancer. Cancer Research 80, 2056–2066 (2020). Wang, S. et al. Computational staining of pathology images to study the tumor microenvironment in lung cancer. Cancer Research 80, 2056–2066 (2020).
14.
go back to reference Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine 25, 1301–1309 (2019). Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine 25, 1301–1309 (2019).
15.
go back to reference Ilse, M., Tomczak, J. M. & Welling, M. in Chapter 22 - deep multiple instance learning for digital histopathology (eds Zhou, S. K., Rueckert, D. & Fichtinger, G.) Handbook of Medical Image Computing and Computer Assisted Intervention 521–546 (Academic Press, 2020). Ilse, M., Tomczak, J. M. & Welling, M. in Chapter 22 - deep multiple instance learning for digital histopathology (eds Zhou, S. K., Rueckert, D. & Fichtinger, G.) Handbook of Medical Image Computing and Computer Assisted Intervention 521–546 (Academic Press, 2020).
16.
go back to reference Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nature Methods 12, 453–457 (2015). Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nature Methods 12, 453–457 (2015).
17.
go back to reference Chen, C. L. et al. An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning. Nature Communications 12, 1193 (2021). Chen, C. L. et al. An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning. Nature Communications 12, 1193 (2021).
18.
go back to reference Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5, 555–570 (2021). Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5, 555–570 (2021).
19.
go back to reference Kather, J. N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nature Medicine 25, 1054–1056 (2019). Kather, J. N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nature Medicine 25, 1054–1056 (2019).
20.
go back to reference Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nature Cancer 1, 789–799 (2020). Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nature Cancer 1, 789–799 (2020).
21.
go back to reference Schmauch, B. et al. A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature Communications 11, 1193 (2020). Schmauch, B. et al. A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature Communications 11, 1193 (2020).
22.
go back to reference Yoo, S. Y. et al. Whole-slide image analysis reveals quantitative landscape of tumor-immune microenvironment in colorectal cancers. Clinical Cancer Research 26, 870–881 (2020). Yoo, S. Y. et al. Whole-slide image analysis reveals quantitative landscape of tumor-immune microenvironment in colorectal cancers. Clinical Cancer Research 26, 870–881 (2020).
23.
go back to reference Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Reports 23, 181–193 (2018). Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Reports 23, 181–193 (2018).
24.
go back to reference Oord, A. v. d., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). Oord, A. v. d., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:​1807.​03748 (2018).
25.
go back to reference Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. International conference on machine learning 1597–1607 (2020). Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. International conference on machine learning 1597–1607 (2020).
26.
go back to reference He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020). He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020).
27.
go back to reference He, K. et al. Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 16000–16009 (2022). He, K. et al. Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 16000–16009 (2022).
28.
go back to reference Huang, H. et al. Contrastive learning-based computational histopathology predict differential expression of cancer driver genes. Briefings in Bioinformatics 23 (2022). Huang, H. et al. Contrastive learning-based computational histopathology predict differential expression of cancer driver genes. Briefings in Bioinformatics 23 (2022).
29.
go back to reference Zhou, Z. H. A brief introduction to weakly supervised learning. National Science Review 5, 44–53 (2017). Zhou, Z. H. A brief introduction to weakly supervised learning. National Science Review 5, 44–53 (2017).
30.
go back to reference Otsu, N. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9, 62–66 (1979). Otsu, N. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9, 62–66 (1979).
31.
go back to reference Cao, X. et al. Effect of cabazitaxel on macrophages improves cd47-targeted immunotherapy for triple-negative breast cancer. Journal for ImmunoTherapy of Cancer 9 (2021). Cao, X. et al. Effect of cabazitaxel on macrophages improves cd47-targeted immunotherapy for triple-negative breast cancer. Journal for ImmunoTherapy of Cancer 9 (2021).
32.
go back to reference Yunna, C., Mengru, H., Lei, W. & Weidong, C. Macrophage m1/m2 polarization. European Journal of Pharmacology 877, 173090 (2020). Yunna, C., Mengru, H., Lei, W. & Weidong, C. Macrophage m1/m2 polarization. European Journal of Pharmacology 877, 173090 (2020).
33.
go back to reference Hu, Q., Wang, X., Hu, W. & Qi, G.-J. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 1074–1083 (2021). Hu, Q., Wang, X., Hu, W. & Qi, G.-J. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 1074–1083 (2021).
34.
go back to reference Shao, Z. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems 34 (2021). Shao, Z. et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems 34 (2021).
35.
go back to reference Tekguc, M., Wing, J. B., Osaki, M., Long, J. & Sakaguchi, S. Treg-expressed ctla-4 depletes cd80/cd86 by trogocytosis, releasing free pd-l1 on antigen-presenting cells. Proceedings of the National Academy of Sciences 118, e2023739118 (2021). Tekguc, M., Wing, J. B., Osaki, M., Long, J. & Sakaguchi, S. Treg-expressed ctla-4 depletes cd80/cd86 by trogocytosis, releasing free pd-l1 on antigen-presenting cells. Proceedings of the National Academy of Sciences 118, e2023739118 (2021).
36.
go back to reference Xu, Q., Chen, S., Hu, Y. & Huang, W. Clinical m2 macrophages-related genes to aid therapy in pancreatic ductal adenocarcinoma. Cancer Cell International 21, 582 (2021). Xu, Q., Chen, S., Hu, Y. & Huang, W. Clinical m2 macrophages-related genes to aid therapy in pancreatic ductal adenocarcinoma. Cancer Cell International 21, 582 (2021).
37.
go back to reference Wang, Z. Q., Milne, K., Webb, J. R. & Watson, P. H. Cd74 and intratumoral immune response in breast cancer. Oncotarget 8, 12664–12674 (2017). Wang, Z. Q., Milne, K., Webb, J. R. & Watson, P. H. Cd74 and intratumoral immune response in breast cancer. Oncotarget 8, 12664–12674 (2017).
38.
go back to reference Liao, L. et al. A potent pgk1 antagonist reveals pgk1 regulates the production of il-1 and il-6. Acta Pharmaceutica Sinica B 12, 4180–4192 (2022). Liao, L. et al. A potent pgk1 antagonist reveals pgk1 regulates the production of il-1 and il-6. Acta Pharmaceutica Sinica B 12, 4180–4192 (2022).
39.
go back to reference Edwards, N. J. et al. The cptac data portal: A resource for cancer proteomics research. Journal of Proteome Research 14, 2707–2713 (2015). Edwards, N. J. et al. The cptac data portal: A resource for cancer proteomics research. Journal of Proteome Research 14, 2707–2713 (2015).
40.
go back to reference Mlynska, A. et al. A gene signature for immune subtyping of desert, excluded, and inflamed ovarian tumors. American Journal of Reproductive Immunology 84, e13244 (2020). Mlynska, A. et al. A gene signature for immune subtyping of desert, excluded, and inflamed ovarian tumors. American Journal of Reproductive Immunology 84, e13244 (2020).
41.
go back to reference Galon, J. & Bruni, D. Approaches to treat immune hot, altered and cold tumours with combination immunotherapies. Nature Reviews Drug Discovery 18, 197–218 (2019). Galon, J. & Bruni, D. Approaches to treat immune hot, altered and cold tumours with combination immunotherapies. Nature Reviews Drug Discovery 18, 197–218 (2019).
42.
go back to reference Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype-immunophenotype relationships and predictors of response to checkpoint blockade. Cell Reports 18, 248–262 (2017). Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype-immunophenotype relationships and predictors of response to checkpoint blockade. Cell Reports 18, 248–262 (2017).
Metadata
Title
Prediction of Tumor-Associated Macrophages and Immunotherapy Benefits Using Weakly Supervised Contrastive Learning in Breast Cancer Pathology Images
Authors
Guobang Yu
Yi Zuo
Bin Wang
Hui Liu
Publication date
17-06-2024
Publisher
Springer International Publishing
Published in
Journal of Imaging Informatics in Medicine
Print ISSN: 2948-2925
Electronic ISSN: 2948-2933
DOI
https://doi.org/10.1007/s10278-024-01166-y