Orthogonal position representations for transformer in neural machine translation

Abstract

In recent years, the Transformer architecture has solidified its position as the dominant model in neural machine translation (NMT), thanks to its exceptional effectiveness in capturing long-range dependencies and remarkable scalability across diverse linguistic tasks. A key characteristic of the Transformer is its reliance on self-attention mechanisms, which, while powerful, are inherently position-insensitive—treating tokens as a set rather than an ordered sequence. This limitation makes positional encoding a critical component in Transformer-based models and their variants, as it provides the necessary sequential context to differentiate token positions within a sequence. In this paper, we address this challenge by proposing a novel orthogonal fixed-dimension positional representation (OPR). This design is meticulously engineered to maximize the discrimination of positions within a sequence, ensuring that each position is uniquely and distinctively encoded. Notably, OPR introduces no additional parameters to the model and incurs no extra computational overhead, making it highly efficient for real-world applications. Our experimental evaluations, conducted across multiple standard NMT datasets, demonstrate that OPR consistently outperforms several strong baselines, including traditional sine-cosine positional encoding and learnable positional embeddings. It achieves notable improvements in both BLEU and COMET scores, with gains observed across all tested language pairs. Furthermore, when combined with relative positional encoding (RPR), the OPR method’s performance is further enhanced, highlighting its ability to effectively model both absolute and relative positional relationships—a dual capability that is crucial for nuanced sequence understanding.