we increase the depth and width of standard Transformer architectures.

Two direction:

The first direction focuses on fitting a larger model on single device by reducing the memory required by activations and optimizer states during the training process. The second direction focuses on efficient training of even larger models through model parallelism e.g. splitting a model across multiple devices.

Dense Scaling

several strategies to increase the capacity of a sequence-to-sequence Transformer model in the context of multilingual machine translation.

Training Large Dense Models

Scaling model capacity with language-specific parameters is a technique where certain layer parameters are split by language or language group based on vocabulary similarity. Because each translation direction only accesses a subset of these parameters, model capacity can scale without significantly increasing training and inference times.

Scaling Model Capacity with Language-Specific Parameters

A single many-to-many translation model covering 100 languages may support 9,900 directed translation pairs. This scale can exceed a standard model's capacity to represent many languages and scripts adequately. Two approaches are dense scaling, which increases Transformer depth or width, and language-specific parameter scaling, which allocates portions of the model to particular languages or language groups.

Google

University of Michigan - Ann Arbor

In a Many-to-Many translation setting, the amount of parallel data grows quadratically with the number of languages, so a standard-capacity neural network underfits rapidly. To supply enough capacity, model size is scaled far beyond typical bilingual systems: models over 50 times larger than current bilingual models are trained using model parallelism, building on scaling results from Kaplan et al. (2020) and Arora et al. (2018).

Scaling Model Size for Many-to-Many Multilingual Translation

Able to scale Transformers and directly translate between 100 languages without pivoting through English at a performance that is competitive with bilingual models on many competitive benchmarks, including WMT

Learn Before

Related

Learn After