Deep Learning and Generative Networks for Atom-by-Atom Electron Microscopy on Million Atom Scales

Machine learning (ML) techniques are catalyzing an era of data-driven materials research, enabling new possibilities such as autonomous collection and processing of massive, atomic resolution electron microscopy data sets on the scale of millions or even hundreds of millions of atoms. However, developing ML models that can reliably handle new or changing experimental conditions in such large data sets remains challenging. We address this challenge by developing a cycle generative adversarial network (CycleGAN) for generating realistic simulated Scanning Transmission Electron Microscopy (STEM) images. The CycleGAN includes a novel reciprocal space discriminator, which learns the complicated, low and high spatial frequency information from experimental data and transfers this information to simulated images. We demonstrate that this CycleGAN can convert easily-generated, but unrealistic, simulated data into realistic images that are nearly indistinguishable from experiment. Such images are valuable because they represent a new method to rapidly generate labeled, experimentally realistic training data for ML-based image analysis—thereby addressing a major barrier that has previously limited the accuracy, ease-of-use, and generalizability of ML in materials characterization. These results represent an important step towards autonomous, large-scale data collection and processing of materials characterization data.