Semantic Segmentation for Analysis of Melting of Nanoscale Ice via Convolutional Neural Networks

Phase segmentation from electron microscopy datasets has emerged as a major subclass of computer vision problems for materials characterization. The melting of ice is one of the most common phase change phenomena in daily life. In-situ observation by Transmission Electron Microscopy (TEM) provides visual insights of dynamic progress and crystal structure. Nano water confined into a carbon film based liquid cell were transferred into the ultra-high vacuum TEM chamber by a cryo-TEM holder, which allowed us to freeze the nano-water droplets and precisely control the temperature during the melting progress. A high-speed K2 direct detection camera was used to record the melting process under low electron dose (&lt;0.1 e^-/nm²), at a rate of 400 frames per second. An ensemble of U-Nets, which have become a powerful tool for dense image segmentation, were trained to segment ice from the image frames and generate time profiles of its area fraction. The network encoders are pre-trained on a large dataset, following which the whole networks are fine-tuned on the current dataset. During inference, the individual image frames are grouped together into high quality composite images, and each composite image is passed as input to the trained neural networks to extract the area fraction of the ice particle. Overall, we demonstrate a workflow which can deliver high segmentation accuracy, along with a measure to quantify the uncertainty which can be used to isolate low-quality images. The approach allows rapid feedback and is a step in progress towards autonomous control of TEM experiments with high speed cameras.<br/><br/>This work was performed at the Center for Nanoscale Materials, a U.S. Department of Energy Office of Science User Facility, and supported by the U.S. Department of Energy, Office of Science, under Contract No. DE-AC02-06CH11357.