In the overcomplete case, the hidden code has a greater dimension than the input. 
The problem with this case is the linear encoder and linear decoder only learn to copy the input to the output. They do not learn anything about the data distribution.

Ohio State University, Columbus

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Autoencoders try to train the network so that the weights for the encoder and decoder make the loss between the original data and its reconstruction as small as it is possible. 

What does the Autoencoder try to do ?

```python         
        #<1>The input to the autoencoder is the same as the input to the encoder.
        model_input = encoder_input
        #<2> The output from the autoencoder is the output from the encoder passed 
        # through the decoder.
        model_output = self.decoder(encoder_output)
        
        #<3> The Keras model that defines the full autoencoder—a model that takes 
        # an image, and passes it through the encoder and back out through the 
        # decoder to generate a reconstruction of the original image.
        self.model = Model(model_input, model_output)```

Putting it all together - AutoEncoder Code

1. Sampling for latent space can be very problematic
2. With autoencoders it is not guaranteed that we would obtain the same  quality when sampling from latent space
3. Some generated outputs are  poorly formed

Problems With Autoencoders

https://github.com/davidADSP/GDL_code/blob/master/models/AE.py

Reference to the AutoEncoder Code

- Traditionally, autoencoders minimize some function $L(x,g(f(x)))$ where $L$ is a loss function penalizing $g(f(x))$ for being dissimilar from $x$, such as the $L^2$ norm of their difference. 
- A denoising auto encoder (DAE) instead minimizes $L(x,g(f(\widetilde{x})))$ where $\widetilde{x}$ is a copy of $x$ that has been corrupted by some form of noise. Denoising autoencoders must therefore undo this corruption rather than simply copying theirinput.
- Two assumptions are inherent to this approach: 1)Higher level representations are relatively stable and robust to the corruption of the input; 2) To perform denoising well, the model needs to extract features that capture useful structure in the input distribution.
- In other words, denoising is advocated as a training criterion for learning to extract useful features that will constitute better higher level representations of the input.

Denoising Autoencoders

A sparse autoencoder is an autoencoder whose training criterion involves a sparsity penalty $$\Omega(h)$$ on the code layer $$h$$. The loss function is $$L(x, g(f(x))) + \Omega(h)$$, where $$g(h)$$ is the decoder output, and $$h = f(x)$$ is the encoder output.

It approximates the maximum likelihood training of a generative model with visible variables $$x$$ and hidden variables $$h$$, with an explicit joint distribution $$p_{model}(x, h) = p_{model}(h)p_{model}(x | h)$$. The log-likelihood can be decomposed as:
$$log(p_{model}(x)) = log \sum\limits_h p_{model} (h,x)$$

We can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for $$h$$. With this chosen $$h$$, we maximize:
$$log(p_{model}(h, x)) = log(p_{model}(h)) + log(p_{model}(x | h))$$

Expressing the log-prior as an absolute value penalty, we obtain:
$$\Omega (h) = \lambda \sum\limits_i |h_i|$$
$$- log(p_{model}(h)) = \sum\limits_i (\lambda |h_i| - log \frac{\lambda}{2}) = \Omega (h) + const$$

Sparse Autoencoders

Undercomplete Autoencoders are an attempt to solve the problem of encoding and decoding being learned so successfully that they circumvent the true goal of autoencoders, which is to learn which features of the data are important. 

"Undercomplete" means that for an autoencoder with basic structure
 input->code->output, with encoding function g and decoding function h, we constrain h to have a lower dimension than the input. Because of this reduced granularity, it is hoped that h will capture only the most "salient features" of the input data during training.

The learning process just involves minimizing the cost function L(x, g(f(x)). where L is a loss function which delivers a panalty for dissimilarity between x and f(x). (example: mean squared error).

Undercomplete Autoencoders

Overcomplete Autoencoders

One way to regularize autoencoder is to add a penalty term. The loss function is thus $L(x, g(f(x))) + \Omega(h, x)$, where $\Omega(h, x) = \lambda \sum_{i}\Vert\nabla x h_i\Vert^2$. It is also called a contractive autoencoder. 

Regularizing Autoencoder

Making the autoencoder network deep has several benefits. 1. It reduces the computational cost of a single hidden layer. 2. It decreases the amount of data needed for training. 3. Deep autoencoder network yields better results. 

Autoencoder Depth

All autoencoder training procedures involve a compromise between two forces:
                                                                                                                                        
1. Learning a representation h of a training example x such that x can be approximately recovered from h through a decoder.
                                                                                                                                                                                                
2. Satisfying the constraint or regularization penalty. 

Neither force alone would be useful.
The two forces together are useful because they force the hidden representation to capture information about the structure of the data-generating distribution.
                                                                                                                               


Learning Manifolds Using Autoencoder

Some autoencoders explicitly represent a probability distribution and admit straightforward ancestral sampling. Most other kinds of autoencoders require MCMC sampling. 
There is also a general Markov chain that can sample from any denoising autoencoder.

Drawing Samples From Autoencoders

In a standard **autoencoder**, each input (such as an image) is mapped to a single point in the latent space. In contrast, in a **variational autoencoder** (VAE), each input is mapped to a multivariate normal distribution around a mean point in that space. The following diagram illustrates this difference:

Autoencoder Encoder vs. Variational Autoencoder Encoder

```python
# <1> Define the input to the encoder (the image).
encoder_input = Input(shape=self.input_dim, name='encoder_input')
x = encoder_input

for i in range(self.n_layers_encoder):
    conv_layer = Conv2D(
        filters=self.encoder_conv_filters[i],
        kernel_size=self.encoder_conv_kernel_size[i],
        strides=self.encoder_conv_strides[i],
        padding='same',
        name='encoder_conv_' + str(i)
    )
    # <2> Stack convolutional layers sequentially on top of each other.
    x = conv_layer(x)
    x = LeakyReLU()(x)
    if self.use_batch_norm:
        x = BatchNormalization()(x)
    if self.use_dropout:
        x = Dropout(rate=0.25)(x)

shape_before_flattening = K.int_shape(x)[1:]
x = Flatten()(x)

# <3> Dense layer that connects this vector to the 2D latent space.
encoder_output = Dense(self.z_dim, name='encoder_output')(x)

# <4> The Keras model that defines the encoder -- a model that takes an
# input image and encodes it into the 2D latent space.
self.encoder = Model(encoder_input, encoder_output)
```

Autoencoder Encoder Sample Code

This Keras decoder maps a latent-space vector to a reconstructed image. A dense layer and reshape operation restore the encoder's pre-flattening tensor shape, and transposed-convolution layers perform the reconstruction. The decoder need not mirror the encoder exactly, but its final output shape must match the reconstruction target.

This snippet assumes that `shape_before_flattening` and the decoder configuration attributes have already been defined.

```python
decoder_input = Input(shape=(self.z_dim,), name='decoder_input')

x = Dense(np.prod(shape_before_flattening))(decoder_input)
x = Reshape(shape_before_flattening)(x)

for i in range(self.n_layers_decoder):
    conv_t_layer = Conv2DTranspose(
        filters=self.decoder_conv_t_filters[i],
        kernel_size=self.decoder_conv_t_kernel_size[i],
        strides=self.decoder_conv_t_strides[i],
        padding='same',
        name='decoder_conv_t_' + str(i),
    )
    x = conv_t_layer(x)

    if i < self.n_layers_decoder - 1:
        x = LeakyReLU()(x)
        if self.use_batch_norm:
            x = BatchNormalization()(x)
        if self.use_dropout:
            x = Dropout(rate=0.25)(x)
    else:
        x = Activation('sigmoid')(x)

decoder_output = x
self.decoder = Model(decoder_input, decoder_output)
```

Source implementation: https://github.com/davidADSP/GDL_code/blob/master/models/AE.py

Learn Before

Related