A blocklist is a list of candidate literal translation errors. 


Blocklist Definition  


1. Select from the source text the list of words $$s = \langle s_1, s_2, \dots, s_N \rangle$$ belonging to the annotated idiom.
2. For each word $$s_i$$, obtain its word translations in the target language using a bilingual dictionary and add them to a blocklist $$b_i = \langle t_1, t_2, \dots, t_M \rangle$$ and create a candidate list of blocklists $$B_s = \langle b_1, b_2, \dots, b_N \rangle$$.
3. For each word in the reference (R), search if it occurs in any of the blocklists $$b_i$$. If so, remove the corresponding blocklist $$b_i$$ from $$B_s$$.
4. Check if the hypothesis contains any blocklisted words. If so, mark this hypothesis as having a literal translation error.

The final score is the percentage of translations that trigger the blocklist. Since LiTER requires source-side annotations, test data with idioms on the source side are collected, and the spans where they occur are annotated.

LiTER Algorithm

While Alignment-based Phrase Translation Evaluation (APT-Eval) measures idiom translation accuracy, it cannot detect literal translation errors in inaccurate translations. In contrast, the Literal Translation Error Rate (LiTER) metric is specifically designed to identify and measure these literal translation errors.

LiTER vs. APT-Eval

Literal Translation Error Rate (LiTER) is a targeted evaluation metric for literal errors in translations of idiomatic expressions. It flags a translation hypothesis when the hypothesis contains a blocklisted target-language word derived from the annotated source idiom, after candidates appearing in the reference translation are excluded. The LiTER score is the percentage of evaluated translations that trigger a blocklist.

University of California, Berkeley

Google

Targeted evaluation isolates the idiom translation with word alignments or word edit distance, which measures idiom translation accuracy but doesn’t account for literal translation errors. Shao et al. (2018) proposed a method for estimating these errors’ frequency but it requires hand-crafted blocklists with words corresponding to literal translation errors. 


Targeted Evaluation for NMT of idiomatic expression 


Alignment-based Phrase Translation Evaluation (APT-Eval) extends Fadee et al. (2018) and subword-level metrics to measure idiom translation accuracy. It uses word alignments to find words in the hypothesis and reference sentences which align with the annotated idiom source span and compares the two. 

Alignment-based Phrase Translation Evaluation (APT-Eval) 


Experiments are done on en-fr and en-es data. For each language pair, the data are concatenated from Europarl v7 (Koehn, 2005), part of the WMT news translation task (Bojar et al., 2014), and TED talk transcripts (Cettolo et al., 2017).


Idiom Translation Experiments 

Standard evaluation methods in neural machine translation (NMT) often overlook the varying frequencies of different idioms, potentially favoring models overfitted on over-represented expressions. To address this issue in targeted evaluation, performance scores are macro-averaged over individual idioms rather than micro-averaged over all occurrences.

Learn Before

Related

Learn After