Human-In-The Loop Automated Experiment (hAE)—Explainable ML for Physics Discovery in Microscopy

Rapid advancements in machine learning over the last decade have attracted broad attention of scientific community towards the possibility of automated and autonomous experimentation (AE) in areas spanning synthesis, characterization, and microscopy. For microscopy, this interest is driven both by the opportunity to automate the often repetitive and tedious process of operation and data collection, and also capitalize on the fact that for many imaging modalities the intrinsic limits for microscope operation are considerably faster than human decision making and response times. Even for imaging, the achievable sampling sizes can exceed the depth of human perception by orders of magnitude, whereas addition of microscopy optimization and spectroscopy measurements opens infinite vistas for ML-driven workflow developments. In AE, the machine learning agent is interacting with the instrument driving the sequence of the operations, either autonomously (no human in the loop) or with human oversight. However, the incorporation of the AE requires significant changes in the human-machine interaction and in particular requires introduction or adoption of fundamentally new concepts describing the experiment planning and execution. In this presentation, I introduce the description of the AE workflows in microscopy and the fundamental concepts necessary for their development including rewards, policies, explainability, and strategies for human in the loop monitoring and intervention. I will illustrate several scenarios for automated microscopy for characterization of ferroelectric and electrochemically active materials including supervised learning, identification of microstrucutres responsible for desired responses, and autonomous physical discovery. I further pose that the emergence of the user facilities and cloud labs necessitate development of universal frameworks for workflow design, including universal hyper-languages describing laboratory operation, reward functions and their integration between domains, and policy development for workflow optimization. These tools will enable knowledge-based workflow optimization, enable lateral instrumental networks, sequential and parallel orchestration of characterization between dissimilar facilities, and empower distributed research.