A significant practical advantage of residual networks is the ability to dynamically add layers during the training process. Because the default behavior of a residual block is to let data pass through unchanged via the identity mapping, new layers initialized as identity functions can be inserted without disrupting the already learned representations. This technique can actively accelerate the training phase of very large neural networks.

Adding Layers During Training in Residual Networks

Residual connections allow inputs to forward propagate faster across layers by providing a direct path that bypasses intermediate transformations. This accelerated propagation of information facilitates the training of significantly deeper neural networks without degradation. As a consequence, architectures can scale to extensive depths, such as the $$152$$-layer model introduced in the original ResNet research.

Accelerated Forward Propagation in Residual Networks

Residual connections provide an identity path across layers. If added residual layers learn a zero residual mapping, the block preserves the identity mapping, so increasing depth does not force the network to learn a worse function. This makes much deeper ResNets easier to train, helps avoid the degradation observed in plain networks, and lets information propagate directly across layers.

University of Michigan - Ann Arbor

Google

- Artificial neural network that builds on constructs from pyramid cells in the cerebral cortex
- Residual neural network do this by using shortcuts to jump over some layers 


ResNets Convolutional Neural Network

In a residual network, the desired underlying mapping $$f(\mathbf{x})$$—the function the network ultimately aims to approximate—is not learned directly by a stack of layers. Instead, those layers are reformulated to learn only the residual mapping $$g(\mathbf{x}) = f(\mathbf{x}) - \mathbf{x}$$, and the target function is recovered as $$f(\mathbf{x}) = g(\mathbf{x}) + \mathbf{x}$$. This reformulation is motivated by the degradation problem: as plain networks grow deeper, their training accuracy can paradoxically worsen, suggesting that the added layers struggle to approximate even the identity function. By recasting the problem in terms of $$g(\mathbf{x})$$, the identity case $$f(\mathbf{x}) = \mathbf{x}$$ reduces to $$g(\mathbf{x}) = 0$$, which is significantly easier for a network to learn because it only requires driving the weights and biases of the constituent layers toward zero.

Residual Mapping

1. ResNeXt
2. Densely Connected CNN
3. Deep Network with Stochastic Depth


Recent Variants of ResNets

Plain vs. ResNets Convolutional Neural Network Architectures

- In general, accuracy gains are obtained from increased depth.
-  the 34-layer ResNet has lower error rate than the 18-layer ResNet architecture.
- The 50/101/152-layer ResNets have lower error rate than the 34-layer by considerable margins.

| model | top-1 error (%) |
| ----------- | ----------- |
| ResNet-34 A    | 25.03       |
| ResNet-34 B   | 24.52      |
| ResNet-34 C   | 24.19       |
| ResNet-50  | 22.85      |
| ResNet-101   | 21.75     |
| ResNet-152  | 21.43       |

Evaluate ResNet at different depths for ImageNet Classification

For single model comparison, the 34-layer ResNets was able achieve competitive accuracy, so as other deeper ResNets.

6 ResNet models of different depths with only two 152-layer ones were used to form an ensemble model. It was recorded 3.57% top-5 error on the test set, which was the first place in ILSVRC 2015.

| method     | top-5 err. (test) |
| ----------- | ----------- |
| VGG (ILSVRC’14)    | 7.32%       |
| GoogLeNet   (ILSVRC’14) | 6.66%       |
| VGG (v5)   | 6.8%       |
| PReLU-net    | 4.94%       |
| BN-inception   | 4.82%       |
| ResNet    | 3.57%       |

Evaluate ResNet models with other state-of-the-art models for ImageNet Classification

•  The "identity shortcuts" are referring to performing the element wise addition of x with the output of the residual layers.

 •  We consider a building block defined as: y = F (x, {Wi}) + x, where x and y are the input and output vectors of the layers considered, the function F (x, {Wi}) represents the residual mapping to be learned.

 •  The residual mapping (F (x, {Wi})) becomes y= W2σ2(W1x)+x
 •  To sum up,  we take the output x of a layer skip it forward and element wise sum it with the output of the residual mapping and thus produce a residual block.



Shortcut’s technique for identity mapping

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778). https://doi.org/10.1109/cvpr.2016.90

Deep Residual Learning for Image Recognition

In the original ResNet architecture designed for ImageNet-scale images, the initial layers consist of a $$7 	imes 7$$ convolutional layer with $$64$$ output channels and a stride of $$2$$, followed by batch normalization, a ReLU activation, and a $$3 	imes 3$$ max-pooling layer with a stride of $$2$$. These large receptive fields and aggressive downsampling are appropriate for the $$224 	imes 224$$ pixel inputs common in ImageNet. However, when working with significantly smaller images (e.g., $$28 	imes 28$$ or $$96 	imes 96$$ pixels from datasets like Fashion-MNIST or CIFAR), these initial layers would reduce spatial dimensions too aggressively, leaving insufficient resolution for the subsequent residual blocks to extract meaningful features.

ResNet Initial Layers

Prior to the widespread adoption of residual connections, highway networks were introduced to enable the effective training of deep architectures with over 100 layers. Highway networks achieved this by using bypassing paths controlled by parameterized gating units. In contrast, Residual Networks (ResNets) simplified this approach by using parameter-free identity functions as their bypassing paths, a design choice that performed remarkably well on various computer vision tasks.

Highway Networks vs. Residual Networks

The introduction of residual connections had a major influence on the design of subsequent deep neural networks, spanning both convolutional and sequential domains. A prominent example is the Transformer architecture, which adopts residual connections alongside other design choices. This combination has become pervasive across diverse fields, including natural language processing, computer vision, speech recognition, and reinforcement learning.

Influence of Residual Connections on Subsequent Architectures

Residual Connections Enable Deeper ResNet Training

In deep learning, adding capacity by nesting function classes allows for strictly more powerful, rather than just subtly different, function classes. Residual connections accomplish this by allowing additional layers to pass the input directly to the output. Consequently, this architectural choice shifts the network's inductive bias: instead of assuming that simple functions take the form $$f(\mathbf{x}) = 0$$, the network assumes that simple functions look like $$f(\mathbf{x}) = \mathbf{x}$$. This makes it significantly easier for the residual mapping to learn the identity function by pushing the parameters in the weight layer toward zero.

Inductive Bias of Residual Connections

ResNet decomposes a target function into a simple linear term and a more complex nonlinear one, mathematically expressed as $$f(\mathbf{x}) = \mathbf{x} + g(\mathbf{x})$$. This approach is conceptually similar to a Taylor expansion, which decomposes a function into terms of increasingly higher order at a given point (e.g., $$f(x) = f(0) + x \cdot \left[f'(0) + x \cdot \left[\frac{f''(0)}{2!} + \cdots ight]ight]$$). By isolating the identity mapping ($$\mathbf{x}$$), ResNet allows the neural network to focus on learning the more complex nonlinear residual ($$g(\mathbf{x})$$).

Learn Before

Related

Learn After