Development and Demonstration of a Real-Time Machine Vision Platform for In Situ Microscopy

Machine vision protocols built around Convolutional Neural Networks (CNNs) are increasing in popularity for image analysis and quantification for microscopy-based workflows as they significantly increase quantification speeds and offer high repeatability rates^1–8. Here, we will outline the development of using a CNN, specifically the You Only Look Once (YOLO) real-time object detection algorithm⁹, for performing real-time quantification during in-situ transmission electron microscopy (TEM) ion irradiation experiments. In these experiments, radiation induced defects can be highly dynamic through nucleation, growth, movement, and loss. We will show the YOLO framework can track the lifecycle of single defects and defect groups during in-situ TEM ion irradiation experiments with near human-like detection levels (e.g., F1 scores of &gt;0.80)¹⁰. Through the established detection algorithm, we are now capable of new insights on dynamic defect processes. For example, we can now demonstrate, through YOLO tracking hundreds of defects during an irradiation of several FeCrAl alloy variants, that defect mobility scales with defect size and bulk matrix compositional complexity. In addition, we will demonstrate the applicable domain of the single-stage YOLO model for various S/TEM imaging methods/conditions and compare the useful domain of the model with two-stage detectors, such as the Mask Regional CNN (Mask R-CNN) and the Faster R-CNN¹¹. The value of these frameworks is readily realized, but nearly all applications of CNN-based detection occur in post in-situ experiment analysis. We have developed a new platform that combines the advantages of the YOLO object detection-based algorithm with an easy-to-use web application (webapp) hosted on edge computing devices. This platform enables <i>real-time</i> graphical overlays of object detections and quantitative information feedback through graphical displays during in-situ TEM experiments with no need for software installation on the image capturing computer(s) and is in-situ microscopy system agnostic. It will be shown that the platform enables immediate feedback of material responses and experimental quality leading to a paradigm shift in the way we conduct experiments and synthesize quantitative data at the time of data acquisition for in-situ microscopy.<br/>1. Li, W., Field, K. G. & Morgan, D. <i>Automated defect analysis in electron microscopic images</i>. npj Computational Materials <b>4,</b> 36 (2018).<br/>2. Roberts, G., et al. <i>Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels</i>. Sci. Reports <b>9,</b> 12744 (2019).<br/>3. Anderson, C. M., et al. <i>Automated Classification of Helium Ingress in Irradiated X-750</i>. arXiv:1912.04252 1–7 (2019).<br/>4. Förster, G. D., et al. <i>A deep learning approach for determining the chiral indices of carbon nanotubes from high-resolution transmission electron microscopy images</i>. Carbon <b>169,</b> 465–474 (2020).<br/>5. Zhang, C., et al. <i>Atomic resolution convergent beam electron diffraction analysis using convolutional neural networks</i>. Ultramicroscopy <b>210,</b> 112921 (2020).<br/>6. Yu, Z. X., et al. <i>High-throughput, algorithmic determination of pore parameters from electron microscopy</i>. Comp. Mat. Sci. <b>171,</b> 109216 (2020).<br/>7. Li, X., et al. <i>Statistical learning of governing equations of dynamics from in-situ electron microscopy imaging data</i>. Materials & Design <b>195,</b> 108973 (2020).<br/>8. Ziatdinov, M., et al. <i>Tracking atomic structure evolution during directed electron beam induced Si-atom motion in graphene via deep machine learning</i>. Nanotechnology (2020).<br/>9. Redmon, J. & Farhadi, A. <i>YOLOv3: An Incremental Improvement</i>. arXiv:1804.02767 (2018).<br/>10. Shen, M., et al. <i>A deep learning based automatic defect analysis framework for In-situ TEM ion irradiations</i>. Computational Materials Science <b>197,</b> 110560 (2021).<br/>11. Jacobs, R., et al. <i>Performance, Successes and Limitations of Deep Learning Semantic Segmentation of Multiple Defects in Transmission Electron Micrographs</i>. arXiv:2110.08244v1 (2021).