Optimizing and Understanding Neural Networks for Automated High-Resolution TEM Analysis

Machine learning and computer vision algorithms are promising solutions to quantify and analyze the ever-increasing amount of in situ transmission electron microscopy (TEM) videos, but it is unclear how to best customize these powerful tools for TEM data. The traditional computer vision mantra of “bigger is better” – more images and more complex networks – is infeasible with the lack of large, labeled TEM datasets, and potentially not even applicable due to the repeated, simple image motifs in TEM images. This problem is further compounded for in situ data, whose analysis generally spans a wider variety of material and imaging conditions than static TEM image datasets. Here, we examine what network, nanoparticle, and imaging features affect how neural networks pixel-wise separate crystalline nanoparticles from amorphous background in high-resolution TEM images, a task also known as image segmentation in computer vision. By systematically varying network architecture and image datasets, we identify how physical knowledge of the nanoparticle system can narrow down the optimal network architectures and necessary training datasets, making automated in situ TEM analysis more attainable.<br/><br/>First, we find that optimal network architectures take into account the nanoparticle size. By increasing the receptive field of the network, or how much of an input image the network utilizes for its final prediction, network segmentation performance improves, plateauing when the receptive field is greater than nanoparticle diameter. When the receptive field is smaller than the nanoparticle diameter, the performance is similar to Fourier filtering, which struggles with nanoparticles that are off zone-axis. Notably, our systematic analysis shows that receptive field is the key variable in network performance. Simpler networks with large enough receptive fields but fewer parameters can outperform more complex networks with small receptive fields, making it possible to train high-performing lightweight networks that do not need large training sets.<br/><br/>Next, to establish neural networks as a useful tool for quantitative studies and to reach the high precision and accuracy needed for in situ analysis, we need to understand what image features our networks are learning. However, neural networks are notably “black boxes”, or difficult to directly probe and understand due to their projection into increasingly abstract, complex spaces. Instead, we indirectly identify what networks are learning by training and testing on curated datasets of labeled experimental and simulated TEM images with systematically varied material and imaging parameters. By benchmarking segmentation performance on these simpler, controlled datasets, we distinguish the error that comes from varying imaging conditions against the general error in network performance. With our curated datasets, we then quantify the ranges at which varying the nanoparticle size and microscope defocus affect segmentation performance. Using this knowledge, we finally demonstrate how a network trained on a curated set of images can perform similarly to a network trained on a more traditional, larger training set in segmenting a nanoparticle growth series.<br/><br/>Altogether, our results provide guidance as to how neural networks can be optimized for in situ TEM datasets and identifies how to best use these powerful tools.