All models produce fewer errors when trained on joint split compared to zero split. Pretraining and upsampling idiom-train data helps all models. Masking increases errors on the joint split, and decoder-side word replacements yield a similar behavior in terms of LitTer. Adding word replacements in the encoder reduces LitTER. 


Literal Translation Errors 


To estimate machine translation model accuracy for idioms, reference and hypothesis matches aligning with source idiom words are compared. Training on a joint data split improves accuracy, and pretraining outperforms random initialization. Upsampling idiom training data doesn't yield consistent improvements, and source-side word replacements reduce literal translation errors while degrading overall idiom translation accuracy.

Idiom Translation Accuracy

Targeted idiom-level evaluation macro-averages scores over individual idioms to reduce the influence of frequency imbalance. LiTER measures literal translation errors, whereas APT-Eval measures idiom translation accuracy. Models trained on the joint split produce fewer literal translation errors, and pretraining improves idiom translation accuracy. Encoder-side word replacement lowers LiTER but degrades overall idiom translation accuracy.

University of California, Berkeley

Google

Results for English-to-French idiom translation were consistent with those for English-to-Spanish. Evaluation included global BLEU scores across diverse test sets and two targeted idiom-level methods. In the upsampling split, idiom-training data were upsampled 20-fold; increasing this to 100-fold caused the models to overfit.

English-to-French and English-to-Spanish Idiom Translation Model Results

mBART models are significantly better than random models, and there is no measurable difference between splits. Noisy finetuning methods do not improve results and encoder-side word replacements degrade overall performance. It is hypothesized that they induce hallucinations. 


General Purpose Results 


Global evaluation considers full sentences, which overshadows idiom translation impact. Models perform better when trained on joint split. 


Learn Before

Related

Learn After