Comparing Pre-Norm and Post-Norm Transformers in Preserving Gender Information for Indonesian–English Translation through Attention-Based Signal Reinforcement

Abstract

Gender realization in Indonesian–English machine translation remains challenging due to the absence of grammatical gender in Indonesian, which often leads to unstable or ambiguous gender representations in English outputs. While Transformer-based models have demonstrated strong general translation performance, their ability to preserve gender information across encoding layers remains inconsistent and poorly understood, particularly with respect to architectural normalization strategies.

This study presents a comparative analysis of Pre-Norm and Post-Norm Transformer architectures in preserving gender information, and examines the role of attention-based signal reinforcement in mitigating representational degradation. The reinforcement mechanism is introduced prior to standard encoder processing to strengthen gender-relevant token interactions without modifying the overall model structure.

Four controlled configurations—Post-Norm, Pre-Norm, Post-Norm with attention-based reinforcement, and Pre-Norm with attention-based reinforcement—are trained under identical random seeds on both unbalanced and balanced datasets. Evaluation is performed on gender-ambiguous test sentences without explicit gender annotations to assess generalization. Gender preservation is assessed at the output level using gender-specific accuracy and BLEU score, and at the representation level using cosine similarity between gender cue embeddings and English gendered pronouns.

The results show that Post-Norm Transformers fail to maintain stable gender representations, yielding near-random gender accuracy (~50%) and negligible BLEU scores. Pre-Norm architectures improve training stability but achieve limited gender accuracy (around 30%). Incorporating attention-based signal reinforcement substantially enhances gender preservation, with accuracy rising to over 50% and reaching up to 56% under balanced training conditions, accompanied by a consistent increase in cosine similarity values (exceeding 0.35) between gender cues and corresponding pronouns. These findings indicate that normalization strategy and attention-based reinforcement jointly determine the stability of gender representations in Transformer-based machine translation.