Learn Before
Auto-Encoding Pre-trained Language Models
During pre-training, auto-encoding language models corrupt part of the input, such as by masking tokens, and learn to reconstruct the missing content from bidirectional context. They typically use the encoder portion of a Transformer, allowing each token representation to attend to available context on both sides. After fine-tuning, these models are well suited to natural-language-understanding tasks such as sentence classification and sequence labeling. Examples include BERT, RoBERTa, and ERNIE.
0
1
Contributors are:
Who are from:
Tags
Deep Learning (in Machine learning)
Data Science
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Auto-Regressive (AR) Models
Seq2seq Models for Text Generation
An engineering team is tasked with creating a system to analyze customer reviews and automatically classify them as 'positive', 'negative', or 'neutral'. The most critical requirement is for the model to have a deep, holistic understanding of the entire review's context to make an accurate classification. Which of the following architectural approaches for a pre-trained model would be best suited for this task?
You are an NLP engineer selecting a pre-trained model architecture for three different projects. Match each project description to the most suitable underlying model training objective.
Model Architecture Selection Flaw
Auto-Encoding Pre-trained Language Models