mBART models are significantly better than random models, and there is no measurable difference between splits. Noisy finetuning methods do not improve results and encoder-side word replacements degrade overall performance. It is hypothesized that they induce hallucinations. 


University of California, Berkeley

Results for English-to-French idiom translation were consistent with those for English-to-Spanish. Evaluation included global BLEU scores across diverse test sets and two targeted idiom-level methods. In the upsampling split, idiom-training data were upsampled 20-fold; increasing this to 100-fold caused the models to overfit.

English-to-French and English-to-Spanish Idiom Translation Model Results

General Purpose Results 


Global evaluation considers full sentences, which overshadows idiom translation impact. Models perform better when trained on joint split. 


Idiom-Test Results


Targeted idiom-level evaluation macro-averages scores over individual idioms to reduce the influence of frequency imbalance. LiTER measures literal translation errors, whereas APT-Eval measures idiom translation accuracy. Models trained on the joint split produce fewer literal translation errors, and pretraining improves idiom translation accuracy. Encoder-side word replacement lowers LiTER but degrades overall idiom translation accuracy.

Learn Before

Related