Building Spatial Differentiability into Convolutional Neural Networks for Reliable and Accurate Pose Prediction of Autonomous Contact-Based Characterization Robotic Systems

Contact-based characterization techniques such as contact profilometry, four-point probes, and nanoindentation, among many others, are valuable tools in quantifying materials' surface and electrical properties. Integration of AI and autonomous robotics into contact-based characterization techniques has implications for both improved reliability and quality of measurements, relieving a significant burden from the researcher, who no longer must ensure the experiment is running optimally. However, integrating autonomy into these contact-based methods presents several challenges not found in optical characterization, such as the need for repeatable motion control and high-accuracy end effector contact positioning. To overcome these challenges, we could consider implementing a deep learning approach and then train a robotic positioning system to determine the contact-based end effector's optimal pose accurately. Unfortunately, training a conventional deep learning model, such as a convolutional neural network (CNN), to understand these patterns requires diversified and abundant datasets of optimal contact poses, presenting a significant data collection burden to researchers. Thus, we propose the design of a spatially differentiable loss function that acts as an objective function and can be integrated into any CNN architecture for optimal pose prediction of contact-based characterization tools rather than cross-checking predictions with a reference dataset to minimize loss during training.

Implementing spatial differentiability into the loss function of our CNN has a two-fold benefit. Firstly, it allows for direct computation on thin film images, enabling our predicted poses to be compared to the target image rather than abstracted to coordinates or geometric properties. Secondly, since our loss function can be expressed as an optimization objective, we can output optimal poses in image space by simply minimizing the loss during training. Therefore, we bypass the need for a reference dataset of optimal poses to train the CNN. Instead, we use an experimentally collected dataset of only 30 computer vision segmented thin films that are then augmented to 8,500 images with modified rotation and scale to ensure robustness in our predictions when running in a noisy experimental setting.

We evaluate the performance of the proposed spatially differentiable CNN on two different characterization tasks: (1) surface profilometry and (2) photoconductivity of printed MAPbBr_1-xI_x perovskite semiconductors. We compare the performance of our method to that of recent loss functions from literature designed for robust and spatial tasks (Reverse Huber, Wing, and Barron) and to conventional loss functions (mean squared error (MSE), mean absolute error (MAE), Huber, and Poisson negative log-likelihood). Our model achieves improvements of 16.2% and 20.0% in the median success rate of generating valid poses over conventional loss functions and robust loss functions, respectively, with only increasing median inference time by 2.4 nanoseconds across 100 independent trials. Lastly, we demonstrate the reliable utilization of the spatially differentiable CNN by running a continuous 24-hour autonomous robotic characterization campaign of perovskite photoconductivity, experimentally measuring 3,025 photocurrent curves without human intervention.