Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy

Nature Biomedical Engineering volume 2, pages 741–748 (2018)Cite this article

Subjects

Abstract

The detection and removal of precancerous polyps via colonoscopy is the gold standard for the prevention of colon cancer. However, the detection rate of adenomatous polyps can vary significantly among endoscopists. Here, we show that a machine-learning algorithm can detect polyps in clinical colonoscopies, in real time and with high sensitivity and specificity. We developed the deep-learning algorithm by using data from 1,290 patients, and validated it on newly collected 27,113 colonoscopy images from 1,138 patients with at least one detected polyp (per-image-sensitivity, 94.38%; per-image-specificity, 95.92%; area under the receiver operating characteristic curve, 0.984), on a public database of 612 polyp-containing images (per-image-sensitivity, 88.24%), on 138 colonoscopy videos with histologically confirmed polyps (per-image-sensitivity of 91.64%; per-polyp-sensitivity, 100%), and on 54 unaltered full-range colonoscopy videos without polyps (per-image-specificity, 95.40%). By using a multi-threaded processing system, the algorithm can process at least 25 frames per second with a latency of 76.80 ± 5.60 ms in real-time video analysis. The software may aid endoscopists while performing colonoscopies, and help assess differences in polyp and adenoma detection performance among endoscopists.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Detailed validation process for the image analysis on dataset A.
Fig. 2: Receiver operating characteristic curve of the algorithm for the image analysis on dataset A.
Fig. 3: Examples of polyp detection for datasets A and C.

Similar content being viewed by others

Data availability

The authors declare that all data supporting the findings of this study are available within the paper and its supplementary information. Restrictions apply to the availability of the medical training/ validation data, which were used with permission for the current study, and so are not publicly available. Some data may be available from the authors upon reasonable request and with permission of the Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital.

References

  1. Lieberman, D. A. et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer. Gastroenterology 143, 844–857 (2012).

    Article  Google Scholar 

  2. Seeff, L. C. et al. How many endoscopies are performed for colorectal cancer screening? Results from CDC’s survey of endoscopic capacity. Gastroenterology 127, 1670–1677 (2004).

    Article  Google Scholar 

  3. Zauber, A. G. et al. Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. N. Engl. J. Med. 366, 687–696 (2012).

    Article  CAS  Google Scholar 

  4. Edwards, B. K. et al. Annual report to the nation on the status of cancer, 1975-2006, featuring colorectal cancer trends and impact of interventions (risk factors, screening, and treatment) to reduce future rates. Cancer 116, 544–573 (2010).

    Article  Google Scholar 

  5. Siegel, R., Naishadham, D. & Jemal, A. Cancer statistics, 2013. CA Cancer J. Clin. 63, 11–30 (2013).

    Article  Google Scholar 

  6. Chinese Society of Gastroenterology. Chinese consensus: screening, early diagnosis and treatment, comprehensive prevention of large bowel cancer. Chin. J. Gastroenterology Hepatology 20, 979–995 (2011).

    Google Scholar 

  7. Brenner, H., Chang-Claude, J., Seiler, C. M., Rickert, A. & Hoffmeister, M. Protection from colorectal cancer after colonoscopy: a population-based, case-control study. Ann. Intern. Med. 154, 22–30 (2011).

    Article  Google Scholar 

  8. Corley, D. A. et al. Adenoma detection rate and risk of colorectal cancer and death. N. Engl. J. Med. 370, 1298–1306 (2014).

    Article  CAS  Google Scholar 

  9. Ahn, S. B. et al. The miss rate for colorectal adenoma determined by quality-adjusted, back-to-back colonoscopies. Gut Liver 6, 64–70 (2012).

    Article  Google Scholar 

  10. Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (Henry Holt and Co. Inc., New York, 1982).

  11. Rex, D. K. et al. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies. Gastroenterology 112, 24–28 (1997).

    Article  CAS  Google Scholar 

  12. Heresbach, D. et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 40, 284–290 (2008).

    Article  CAS  Google Scholar 

  13. Leufkens, A. et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study. Endoscopy 44, 470–475 (2012).

    Article  CAS  Google Scholar 

  14. Mahmud, N., Cohen, J., Tsourides, K. & Berzin, T. M. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol. Rep. (Oxf.) 3, 179–184 (2015).

    Article  Google Scholar 

  15. Aslanian, H. R. et al. Nurse observation during colonoscopy increases polyp detection: a randomized prospective study. Am. J. Gastroenterol. 108, 166–172 (2013).

    Article  Google Scholar 

  16. Lee, C. K. et al. Participation by experienced endoscopy nurses increases the detection rate of colon polyps during a screening colonoscopy: a multicenter, prospective, randomized study. Gastrointest. Endosc. 74, 1094–1102 (2011).

    Article  Google Scholar 

  17. Buchner, A. M. et al. Trainee participation is associated with increased small adenoma detection. Gastrointest. Endosc. 73, 1223–1231 (2011).

    Article  Google Scholar 

  18. Mori, Y. et al. Computer-aided diagnosis for colonoscopy. Endoscopy 49, 813–819 (2017).

    Article  Google Scholar 

  19. Wang, Y. et al. Part-based multiderivative edge cross-sectional profiles for polyp detection in colonoscopy. IEEE J. Biomed. Health Inform. 18, 1379–1389 (2014).

    Article  Google Scholar 

  20. Brandao, P. et al. Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks. J. Medical Robotics Res. 3, 1840002 (2018).

    Article  Google Scholar 

  21. Chen, J. H. & Asch, S. M. Machine learning and prediction in medicine-beyond the peak of inflated expectations. N. Engl. J. Med. 376, 2507–2509 (2017).

    Article  Google Scholar 

  22. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

    Article  Google Scholar 

  23. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  CAS  Google Scholar 

  24. Misawa, M. et al. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience. Gastroenterology 154, 2027–2029 (2018).

    Article  Google Scholar 

  25. Gkolfakis, P. et al. New endoscopes and add-on devices to improve colonoscopy performance. World J. Gastroenterol. 23, 3784–3796 (2017).

    Article  Google Scholar 

  26. Pogorelov, K. et al. Efficient disease detection in gastrointestinal videos–global features versus neural network. Multimedia Tools Appl. 76, 22493–22525 (2017).

    Article  Google Scholar 

  27. Fernandez-Esparrach, G. et al. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy 48, 837–842 (2016).

    Article  Google Scholar 

  28. Wang, Y., Tavanapong, W., Wong, J., Oh, J. H. & de Groen, P. C. Polyp-Alert: near real-time feedback during colonoscopy. Comput. Methods Programs Biomed. 120, 164–179 (2015).

    Article  Google Scholar 

  29. Kuo, S. M., Lee, B. H. & Tian, W. Real-Time Digital Signal Processing: Implementations and Applications 2nd edn, Ch. 1 (John Wiley & Sons Ltd, Chichester, 2013).

  30. East, J. E. et al. Advanced endoscopic imaging: European Society of Gastrointestinal Endoscopy (ESGE) Technology Review. Endoscopy 48, 1029–1045 (2016).

    Article  Google Scholar 

  31. Mori, Y. et al. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Ann. Intern. Med. https://doi.org/10.7326/M18-0249 (2018).

    Article  Google Scholar 

  32. Takemura, Y. et al. Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video). Gastrointest. Endosc. 75, 179–185 (2012).

    Article  Google Scholar 

  33. Kominami, Y. et al. Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest. Endosc. 83, 643–649 (2016).

    Article  Google Scholar 

  34. Chen, P. J. et al. Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology 154, 568–575 (2018).

    Article  Google Scholar 

  35. Byrne, M. F. et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut https://doi.org/10.1136/gutjnl-2017-314547 (2017)

  36. Yao, K. The endoscopic diagnosis of early gastric cancer. Ann. Gastroenterol. 26, 11–22 (2013).

    PubMed  PubMed Central  Google Scholar 

  37. Winawer, S. J. et al. Colorectal cancer screening: clinical guidelines and rationale. Gastroenterology 112, 594–642 (1997).

    Article  CAS  Google Scholar 

  38. Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Analysis Machine Intelligence 39, 2481–2495 (2017).

    Article  Google Scholar 

  39. Dean, J. et al. Large scale distributed deep networks. Adv. Neural Inf. Processing Syst. 25, 1223–1231 (2012).

    Google Scholar 

  40. Bandos, A. I., Rockette, H. E., Song, T. & Gur, D. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65, 247–256 (2009).

    Article  Google Scholar 

  41. Bernal, J. et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans. Med. Imaging 36, 1231–1249 (2017).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the Machine Intelligence Laboratory of the University of Cambridge for developing SegNet and making it publicly available.

Author information

Authors and Affiliations

  1. Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital, Chengdu, China

    Pu Wang, Mengtian Tu, Fei Xiong, Xiao Hu, Peixi Liu, Yan Song, Di Zhang, Xue Yang, Liangping Li & Xiaogang Liu

  2. Shanghai Wision AI Co., Ltd, Shanghai, China

    Xiao Xiao, Jiong He, Xin Yi & Jingjia Liu

  3. Beth Israel Deaconess Medical Center and Harvard Medical School, Center for Advanced Endoscopy, Boston , MA, USA

    Jeremy R. Glissen Brown & Tyler M. Berzin

Authors
  1. Pu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeremy R. Glissen Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tyler M. Berzin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mengtian Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fei Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Peixi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Di Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Xue Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Liangping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Jiong He
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Xin Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Jingjia Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Xiaogang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.W. and X.L. contributed to study concept and design. P.W., X.L., L.L., M.T., F.X., X.H., P.L., Y.S., D.Z. and X.Yang. contributed to acquisition of data and statistical analysis. P.W., T.M.B. and J.R.G.B contributed to analysis, interpretation of data and drafting of the manuscript. X.X. contributed to algorithm development. J.L., J.H. and X.Yi. contributed to algorithm and software/hardware implementation. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xiaogang Liu.

Ethics declarations

Competing interests

X.X., J.L., J.H. and X.Y. are employees of Shanghai Wision AI Co., Ltd. The automatic polyp detection system was developed by the company and the software was provided free of charge for the purposes of this study. All other authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary figures and video captions.

Reporting Summary

Supplementary Video 1

Real-time visual assistance during colonoscopy on an adjacent monitor.

Supplementary Video 2

Additional video of real-time visual assistance during colonoscopy on an adjacent monitor.

Supplementary Video 3

Sample video from the simulated real-time video analysis.

Supplementary Video 4

Additional sample video from the simulated real-time video analysis.

Supplementary Video 5

Demonstration of simulated real-time video analysis on datasets C and D.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, P., Xiao, X., Glissen Brown, J.R. et al. Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nat Biomed Eng 2, 741–748 (2018). https://doi.org/10.1038/s41551-018-0301-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-018-0301-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer