Learn Before
Visual Example of Discriminator Operation in Replaced Token Detection
The discriminator's task in replaced token detection is to classify each token in a sequence that has been altered by a generator. It receives an input sequence and produces a label of either or for each position.
For example, if the sequence generated is "The boy spent decades working on toys .", the process can be visualized as follows:
\text{replaced:} & [\mathrm{CLS}] & \text{The} & \text{boy} & \text{spent} & \text{decades} & \text{working} & \text{on} & \text{toys} & . & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \multicolumn{9}{c}{\text{Discriminator (the model we want)}} & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow & \downarrow \text{label:} & \mathrm{original} & \mathrm{original} & \mathrm{original} & \mathrm{original} & \mathrm{replaced} & \mathrm{original} & \mathrm{original} & \mathrm{original} & \mathrm{original} \end{array}$$ As shown, the model accurately identifies "decades" as $$\mathrm{replaced}$$ and "toys" as $$\mathrm{original}$$, even though "toys" was masked and regenerated by the generator, because it matches the original text.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Visual Example of Discriminator Operation in Replaced Token Detection
In a particular self-supervised learning setup, a 'generator' model first processes an input sentence and replaces some of its words with plausible alternatives. A second, more powerful 'discriminator' model then receives this altered sentence. The discriminator's task is to examine each word and determine if it is identical to the word in the original, unaltered sentence.
Consider this example:
- Original Sentence: "The scientist discovered the new element."
- Altered Sentence from Generator: "The scientist found the new element."
Given the discriminator's task, how should it classify the words 'found' and 'element' from the altered sentence?
In a replaced token detection task, a generator model is given the sentence 'The chef cooked the meal' with the word 'cooked' masked. The generator predicts the word 'cooked' to fill the mask. The discriminator then receives the sentence 'The chef cooked the meal'. According to the discriminator's objective, it should classify the word 'cooked' as 'replaced'.
Discriminator Performance Analysis
Learn After
A text-processing system is designed to identify altered words in a sentence. It uses a 'generator' to produce a new version of an original sentence, and then a 'discriminator' to label each word in the new sentence as either 'original' (if it's the same as the word in the original sentence) or 'replaced' (if the generator changed it).
Original Sentence: "The scientist discovered a new planet." Generator's Output Sentence: "The scientist found a new planet."
Based on its task, how should the discriminator label the words 'found' and 'planet' in the generator's output sentence?
Discriminator Labeling Logic
Evaluating a Token Classification System