Can Chemical Property Prediction Models Extrapolate? Understanding How to Move Towards Generalizable Chemical Foundation Models

Alongside the push towards artificial general intelligence, there has been a surge of interest in the chemical community to develop large-scale chemical foundation models. This goal is motivated by the idea that a single foundation model trained on vast quantities of chemical data can be broadly applied to multiple applications without needing retraining. Although several chemical foundation models have already displayed their proficiency at predicting molecular properties, their capability to generalize across chemical space has yet to be systematically evaluated.<br/>In this talk, I will discuss our work on understanding the capabilities of current state-of-the-art models (both foundation models and otherwise) to extrapolate to new chemistries. From performing large-scale benchmarking of over seven models across ten unique datasets and millions of molecular samples, we find that no existing model has the ability to consistently extrapolate to new chemistry. As a result, we perform various ablation experiments to understand how to provide better generalization capabilities to models. Specifically, we probe how pre-training objectives, multi-task learning, model architecture and molecule representation affect the resulting extrapolation performance. The insights gained from these experiments provide actionable guidance on how the materials community can proceed towards achieving generalizable materials foundation models. In addition, the extrapolation tasks established in our benchmarking provide the materials community with a uniform benchmark for tracking progress in the extrapolation abilities of chemical models.<br/> <br/>This work was produced under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.