Multimodal Co-Orchestration to Uncover Structure-Property Relationships in Combinatorial Libraries

The rapid progress in automated synthesis techniques has enhanced cost-effectiveness and sped up the development of new materials. Combinatorial libraries are among the earliest examples of such high-throughput synthesis. However, a bottleneck remains in the characterization phase, which often requires an extensive analysis of various electric, mechanical, chemical, and structural material properties. Characterizing combinatorial libraries, for instance, involves exploring multiple aspects of their structures and functionalities using a wide range of local investigative methods, such as scanning probe microscopy, electron microscopy, Raman microscopy, and others.<br/>The ongoing revolution in autonomous instrumentation and novel artificial intelligence-based approaches is introducing groundbreaking capabilities for integrating multimodal tools, equipped with several sequential detection methods, or several characterization tools into automated workflows Here we present a <i>co-orchestration workflow</i> designed to facilitate the exploration of combinatorial libraries and similar systems through the simultaneous application of multiple methods (modalities). The central concept of this workflow is the real-time use of knowledge gained from one property to expedite the exploration of other properties measured by different methods. The proposed framework integrates dimensionality reduction through variational autoencoders (VAEs) with multi-task Gaussian Processes (MTGP) to harness correlations between modalities and use them for the characterization expedition.<br/>In multimodal co-orchestration, a process of characterization of a combinatorial library comprises sequential steps. At each of these steps, the orchestrating agent selects the measurement modality and location for the next exploration step based on the expected knowledge gain and associated measurement cost. The low-dimensional compositional space within a combinatorial library makes the implementation of Bayesian optimization (BO) a highly robust solution for governing the orchestrating agent. At the same time, raw measured data are often represented in high-dimensional datasets, such as spectra and images. In multimodal co-orchestration, the compositional dependencies of the VAE latent variables for the different modalities are employed to train the multi-task GPs (MTGP). The multimodal acquisition function is constructed based on the MTGP outputs. Leveraging VAEs for dimensionality reduction allows us to transfer the interpretation and feature extraction from raw data from humans to AI agents, enabling GPs to be trained on the compositional dependencies of complex features without direct human involvement.<br/>The proposed framework's effectiveness was evaluated by applying it to the co-orchestration of piezoresponse force microscopy hysteresis loops (BEPS) and micro-Raman spectra for exploring the Sm-BiFeO₃ (Sm-BFO) combinatorial library. This system exhibits a phase transition from the ferroelectric phase of pure rhombohedral BiFeO₃ to a non-ferroelectric orthorhombic phase in BiFeO₃ doped with 20% Sm, passing through the morphotropic phase boundary. The workflow demonstrated its effectiveness in optimizing autonomous exploration, particularly when the VAE latent variables showed smooth compositional dependencies. We believe that the multimodal co-orchestration workflow provides a flexible and robust solution for an expedition of the characterization of combinatorial libraries and similar material systems.