mBART models are significantly better than random models, and there is no measurable difference between splits. Noisy finetuning methods do not improve results and encoder-side word replacements degrade overall performance. It is hypothesized that they induce hallucinations. 


General Purpose Results 


Global evaluation considers full sentences, which overshadows idiom translation impact. Models perform better when trained on joint split. 


Idiom-Test Results


Targeted idiom-level evaluation macro-averages scores over individual idioms to reduce the influence of frequency imbalance. LiTER measures literal translation errors, whereas APT-Eval measures idiom translation accuracy. Models trained on the joint split produce fewer literal translation errors, and pretraining improves idiom translation accuracy. Encoder-side word replacement lowers LiTER but degrades overall idiom translation accuracy.

Targeted Idiom Translation Evaluation Results

Results for English-to-French idiom translation were consistent with those for English-to-Spanish. Evaluation included global BLEU scores across diverse test sets and two targeted idiom-level methods. In the upsampling split, idiom-training data were upsampled 20-fold; increasing this to 100-fold caused the models to overfit.

University of California, Berkeley

Google

Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


There are three idiom testing conditions.


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


For idiom translation experiments, sentence pairs with more than 80 words and length ratios greater than 1.5 are filtered out. The remaining sentences are tokenized using SentencePiece (SPM) (Kudo & Richardson, 2018).

Idiom Data Preprocessing

To analyze how neural machine translation (NMT) models represent and translate idioms, researchers vary the amount of surrounding context available to the encoder using three context-masking conditions:

1. **Full Context**: The idiom phrase is encoded normally within the entire input sentence, allowing the model to use surrounding syntactic and semantic clues.
2. **Phrase-Level Context**: The idiom phrase is encoded in isolation (with the surrounding sentence context masked or removed), forcing the model to represent the idiom solely based on its constituent words.
3. **Word-Level Context**: Each word of the idiom is encoded independently, breaking phrase-level cohesion and forcing a compositional, word-by-word representation.

By feeding these encoder representations to the decoder, researchers can measure how varying context affects decoder distributions and translation output.

Context Levels for Probing NMT Idiom Translation

Parallel data is split into regular and idiom subsets using a pattern-matching tool built on Spacy's (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom's occurrence in a sentence is annotated.

Separating Regular and Idiom Data

Pretraining on monolingual data can improve both general translation quality and idiom translation quality, even though monolingual corpora do not contain explicit examples of translating idioms between languages.

Learn Before

Related

Learn After