Authors use both randomly initialized and mBART initialized models in experiments. 


Monolingual Data Models


Pretraining on monolingual data can improve both general translation quality and idiom translation quality, even though monolingual corpora do not contain explicit examples of translating idioms between languages.

University of California, Berkeley

Google

Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

The data (list of 225 English idioms) is fed into the pattern-matching tool, and translation pairs containing an idiom on the source side are extracted and annotated. Sentences with idioms occurring only once are discarded. Regular data is used for training, while the idiom data are evenly divided into idiom-train and idiom-test sets, i.e. half of each idiom’s sentence pairs go in the training set. 


Processing Idiom Data 


There are three idiom testing conditions.


Idiom Testing Conditions


The IWSLT dev-set for each language pair is used for development. For general translation evaluation, results are reported in the WMT newstest14 and IWSLT’17 test sets for en->fr and WMT newstest13 and IWSLT’17 test sets for en->es. The extracted idiom-test data per language pair is used for targeted idiom evaluation. A fast-align model trained on language-pairgenerates word alignments for APT-Eval, and beam search (beams size 5) and BLEU computed with SacreBLEU evaluates all models. 


Experiment Evaluation 


For idiom translation experiments, sentence pairs with more than 80 words and length ratios greater than 1.5 are filtered out. The remaining sentences are tokenized using SentencePiece (SPM) (Kudo & Richardson, 2018).

Idiom Data Preprocessing

To analyze how neural machine translation (NMT) models represent and translate idioms, researchers vary the amount of surrounding context available to the encoder using three context-masking conditions:

1. **Full Context**: The idiom phrase is encoded normally within the entire input sentence, allowing the model to use surrounding syntactic and semantic clues.
2. **Phrase-Level Context**: The idiom phrase is encoded in isolation (with the surrounding sentence context masked or removed), forcing the model to represent the idiom solely based on its constituent words.
3. **Word-Level Context**: Each word of the idiom is encoded independently, breaking phrase-level cohesion and forcing a compositional, word-by-word representation.

By feeding these encoder representations to the decoder, researchers can measure how varying context affects decoder distributions and translation output.

Context Levels for Probing NMT Idiom Translation

Parallel data is split into regular and idiom subsets using a pattern-matching tool built on Spacy's (Hannibal and Montani, 2017) rule-based matching engine. The span of each idiom's occurrence in a sentence is annotated.

Separating Regular and Idiom Data

Effect of Monolingual Pretraining on Idiom Translation

Results for English-to-French idiom translation were consistent with those for English-to-Spanish. Evaluation included global BLEU scores across diverse test sets and two targeted idiom-level methods. In the upsampling split, idiom-training data were upsampled 20-fold; increasing this to 100-fold caused the models to overfit.

Learn Before

Related

Learn After