An equivariant pretrained transformer for unified 3D molecular representation learning

Abstract

Abstract Pretraining on a large number of unlabeled 3D molecules has showcased superiority in various scientific applications. However, prior efforts typically focus on pretraining models in a specific domain, missing the opportunity to leverage cross-domain knowledge. To mitigate this gap, we introduce Equivariant Pretrained Transformer, an all-atom foundation model that can be pretrained from multiple domain 3D molecules. Built upon an E(3)-equivariant transformer, the model learns both atom-level interactions and graph-level structural features ( e.g . residuals in proteins), allowing it to generalize across diverse tasks. The model achieves strong gains in ligand binding affinity prediction, while also performing competitively in predicting properties of proteins and small molecules. We further show that the model can help identify potential antiviral compounds against the main protease of the COVID-19 virus, and validate promising candidates through computational and experimental studies.