Focus shifts in contextual and lexical cue interactions in GPT models

Abstract

Transformer-based language models have demonstrated sensitivity to a range of linguistic dependencies, yet it remains unclear how they represent information-structural focus and integrate discourse and lexical focus cues during ellipsis resolution. We investigated GPT-style models’ interpretation of elliptical remnant continuations in double-object constructions by manipulating contextual focus via preceding interrogatives ( who vs. what ) and lexical focus via the particle only , whose surface position was varied. Using word-by-word surprisal as an index of processing difficulty, we conducted three experiments with GPT-2 models (Small–XL) and GPT-Neo. In Experiment 1 (no only ), models robustly tracked the wh -induced discourse focus, assigning higher surprisal to remnants that mismatched the contextually focused constituent. In Experiment 2 ( only preceding the indirect object), contextual focus continued to dominate, indicating that discourse cues were maintained despite the presence of a competing lexical marker. In Experiment 3 ( only preceding the direct object), lexical focus effects became stronger: models favored remnants aligned with the lexically biased direct object, consistent with locality-based cue weighting when only is adjacent to that object. Comparisons with human reaction-time data revealed broad convergence in contextual-focus sensitivity but divergence when the remnant was compatible with one cue but not the other, with GPT-style models exhibiting a stronger bias toward alignment with only than humans. Together, these findings suggest that the tested models maintain discourse-level focus representations while integrating multiple focus cues in a proximity-sensitive manner, revealing both overlap and limits in their alignment with human processing.