Reward Driven Image Analysis Workflows in Automated Electron Microscopy

Automated experiments in scanning transmission electron microscopy (STEM) require rapid image segmentation to optimize data representation for human interpretation, decision-making, site-selective spectroscopies, and atomic manipulation. Currently, segmentation tasks are typically performed using supervised machine learning methods, which require human-labeled data and are sensitive to out-of-distribution drift effects caused by changes in resolution, sampling, or beam shape. We develop an approach based on the concept of a reward function, intricately linked to the experimental objectives and the broader context, yet quantifiable upon experiment completion. Once defined, these reward function allow optimization of the workflow, including both combinatorial analysis selection and continuous parameter optimization via Bayesian Optimization, thereby ensuring the attainment of results that are both precise and aligned with the human-defined objectives. We demonstrate the applicability of reward-based workflows for tasks such as atom finding, identification of the amorphized regions due to the radiation damage on a single sublattice, and mapping of phases and ferroelectric domains. We further operationalize and benchmark reward-driven workflow for on-the fly image analysis in STEM. We establish the timing and effectiveness of this method, demonstrating its capability for real-time performance in high-throughput and dynamic automated STEM experiments. This unsupervised approach is much more robust, as it does not rely on human labels and is fully explainable. The explanatory feedback can help the human to verify the decision making and potentially tune the model by selecting the position along the Pareto frontier of reward functions. The reward driven approach allows to construct explainable robust analysis workflows and can be generalized to a broad range of image analysis tasks in electron and scanning probe microscopy and chemical imaging.