Learn Before
GPT (Generative Pre-Training)
The GPT (Generative Pre-Training) model represents an effort to design a general, task-agnostic architecture for context-sensitive representations. Built on a Transformer decoder, it pretrains a language model to represent text sequences. When adapted for downstream applications, the model's output feeds directly into an added linear layer to predict task labels. In sharp contrast to earlier models that freeze their pretrained weights, GPT fine-tunes all parameters in the pretrained Transformer decoder during supervised learning. Evaluated on twelve tasks encompassing natural language inference, question answering, sentence similarity, and classification, GPT improved the state of the art in nine of them with minimal changes to its core architecture.
0
1
Contributors are:
Who are from:
Tags
Data Science
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
D2L
Dive into Deep Learning @ D2L
Related
GPT-2
GPT-3
The GPT series of models is renowned for its strong performance on text generation tasks. Considering the typical components of a transformer, which statement best analyzes why a 'decoder-only' architecture is particularly effective for this purpose?
Match each transformer architecture type with its primary application and a representative model family.
A developer is building a chatbot designed for open-ended, creative conversation. The primary requirement is that the chatbot can generate fluent, coherent, and contextually relevant continuations of the user's input. Which architectural principle, central to the design of the GPT series, makes it particularly well-suited for this task?
GPT (Generative Pre-Training)
GPT (Generative Pre-Training)
PaLM (Pathway Language Model)
Learn After
A foundational generative language model introduced in 2018 significantly improved the ability to capture relationships between words far apart in a text, a major challenge for previous sequential models. Which of the following best analyzes the core architectural innovation responsible for this leap in performance?
Critique of an Early Transformer-Based Language Model
Training Objective of an Early Transformer Model
GPT-2
Similarities Between BERT and GPT in Fine-Tuning
Autoregressive Limitation of GPT
Comparison of ELMo and GPT on Downstream Adaptation
Comparison of ELMo, GPT, and BERT