A breakthrough in NLP and deep learning, pre-trained language models (PLMs) are trained on large-scale unlabeled data, which allows them to grasp diverse knowledge. These models serve as 'foundation models' that can be adapted to a wide range of downstream tasks, often through fine-tuning on a small amount of supervised data, leading to state-of-the-art results. This approach has fundamentally changed the NLP development paradigm.

Pre-trained Language Models

In controllable text generation (CTG), a control condition (or controlled condition) refers to a task-specific constraint, attribute, or metadata (such as sentiment, style, topic, or format) used to guide and restrict the output of a generative language model.

Control Condition in Controllable Text Generation

A controllable text generation (CTG) system comprises three components: (1) a controlled input, consisting of source text and a task-specific control condition; (2) a generative model, represented as process $$P$$; and (3) generated text, represented as output $$O$$, that satisfies the control condition.

University of California, Berkeley

Google

An area within Natural Language Generation, Controllable Text Generation is integral in developing text generation technologies that are more natural and customizable in practical applications. CTG is the task of generating text given a controlled element. 

Controllable Text Generation (CTG) 


An NLG system should generate text that meet task-specific, controllable constraints. In dialogue response generation, one would control for politeness / offensive content. 

Controllability in Text Generation 


Early approaches were based on sequential methods and style embedding, which was followed by deep generative model-based methods such as Variational Autoencoders (VAEs), Generative Adversarial Nets (GANs), and Energy-based Models. 

Deep Learning-Based CTG Methods 


Controllable Text Generation (CTG) can be described as $$P(Y|C) = p(y_1, y_2, …, y_n|C)$$, where $$C$$ varies according to the task. Additionally, the generated text $$Y$$ should satisfy the constraint conditions while being reasonably human-readable.

Learn Before

Related

Learn After