Accelerating the Closed-Loop Transmission Electron Microscope via Hardware-Software Codesign of Machine Learning Frameworks

Adjusting processing parameters on-the-fly in response to multi-modal datasets during an <i>operando</i> Transmission Electron Microscope (TEM) experiment promises precision control over structure and electronic states of a material. The continued development of multi-dimensional characterization techniques has created an exponential growth in data rates produced from the instrument. Machine learning (ML) has been proven to process, analyze, and respond to the large spatially and temporally resolved datasets, enabling closed-loop response of dynamics within the TEM. However, due to the timescales of the observed changes in an experiment, such as defect formation and crystallization, the latency and throughput of conventional machine learning frameworks lack the response time to act before the change has occurred. Traditional compute architectures for ML are optimized for large data tasks. Alternatively, edge compute devices from commercial vendors and custom designed accelerators on Field Programmable Gate Arrays (FPGAs) offer means to accelerate machine learning frameworks and optimize latencies for closed loop microscopy. We demonstrate how deploying ML frameworks on edge devices can reduce processing times and lessen the bottleneck of processing data within the closed-loop. To optimize the performance on edge and computation precision, hardware-software codesign is necessary for ML frameworks to use dedicated hardware accelerators in a System-on-a-Chip or FPGA. We also illustrate how codesign can be achieved by a domain scientist without extensive knowledge of computer architectures through tools and strategies available and capabilities developed in this work. Bridging the gap between microscopist, data scientist, and hardware engineer is a critical step towards achieving real time closed-loop control of materials within the TEM.