Dependency-aware self-attention for robust neural machine translation

Abstract

Neural machine translation (NMT) has significantly benefited from integrating various forms of contextual information. However, conventional Transformer-based translation models primarily rely on self-attention mechanisms that are inherently position-invariant, making them inadequate for effectively capturing explicit syntactic dependencies, especially in low-resource scenarios or morphologically rich languages. To address this limitation, we propose a Dependency-Aware Self-Attention (DASA) mechanism that explicitly incorporates syntactic dependency structures into the attention computation. Our method first leverages a dependency parser to derive syntactic trees from source sentences, generating a dependency distance matrix representing pairwise syntactic proximity. This matrix is transformed into a normalized syntactic bias, which is seamlessly integrated into the attention mechanism through element-wise modulation of attention logits. By doing so, DASA guides attention weights towards syntactically relevant tokens, enhancing the Transformer encoder’s structural awareness and representation quality. Experimental results demonstrate that our approach substantially improves the translation performance, particularly in settings with limited training data. Experiments show that DASA enhances syntactic awareness and robustness, especially under data scarcity.