Transfer learning of high-dimensional features via attention-based embedding for Ni-based superalloys

Abstract

In feature engineering, one often reduces redundant dimensionality of raw feature space through selection. However, feature reduction poses a risk of removing critical yet unknown influence factors. In this work, we develop a general dimensionality augmentation approach for feature learning using attention-based embedding vectors constructed by transfer learning, to represent materials' compositions and elementary physiochemical descriptors beyond conventional concatenation to achieve more accurate prediction of materials properties. First, the high-dimensional embedded features [explicit Learnable Parameter Attention (LPA) and implicit Transformer-based Self-Attention (TSA)] were constructed by attention-based pretrained self-supervised learning using deep neural-network models [multi-layer perceptron (MLP)] with materials descriptor labels calculated analytically where attention scores can be used as feature importance. Next, the MLP models equipped with these embedded features were further fine-tuned by transfer learning to high-value property regions, which outperformed the MLP and tree-based baseline models with conventional features of concatenated compositions and descriptors in predicting melting temperatures of Ni-based superalloys as a case study. Finally, the inverse design via global optimization using a genetic algorithm (GA) combined with the embedded features (GA-MLP_LPA/TSA) was performed with the attention-based feature importance as a crossover probability. The integrated protocol surpassed the baseline Bayesian optimization in several statistical metrics. The proposed embedded feature models can capture correlations and couplings among features, providing a robust and efficient high-dimensional materials representation generally applicable to both forward property prediction and inverse materials design via machine learning.