PyTorch supports several categories of neural-network layers, including convolution, pooling, padding, normalization, recurrent, transformer, linear, dropout, sparse, vision, shuffle, and DataParallel (multi-GPU) layers.

University of Michigan - Ann Arbor

Google

An open source deep learning library that specializes in tensor computations and GPU acceleration. It allows you to build neural networks and has plenty of features such as implementing custom layers and network architectures.

PyTorch

One of the major features of PyTorch is how it represents data. It uses a multi-dimensional array like in NumPy but they are called Tensors. To a programmer the the initialization and manipulation of the array is the exact same or very similar to NumPy. But under the hood, they are used to dynamically generate a graph and apply automatic differentiation.

Tensors in PyTorch

CUDA helps keep track of which GPU is being used and ensures the tensors allocated will be on that device. It allows you to leverage GPU computational speed for certain tensor operations but also has functionality that allows you to move the tensor between the CPU and GPU.

CUDA in PyTorch

PyTorch package for implementing a variety of optimizations alogithms. Interface is abstracted well enough that any optimization method can be used without changing surrounding code. There's also the option to implement a custom optimization method. Some examples of ones already implemented by PyTorch are:
- Adadelta
- Adagrad
- Adam
- ASGD
- RMSprop
- SGD

torch.optim

When working with large datasets (which you usually do with neural networks), it takes a large amount of time to load it all into memory before training. Also since we generally use a GPU for neural networks, there can also be memory limitations of the system. PyTorch's dataloader parallelizes the data loading process with automatic batching which in turn boosts the speed and saves memory.

DataLoader

One of the most popular applications of deep learning is computer vision and torchvision is a library that goes hand in hand with PyTorch. It has efficient utilities for image and video transformations, model architectures, and some of the most popular datasets already loaded.

Torchvision

Layer Types in PyTorch

PyTorch Autograd performs automatic differentiation on computations represented by a dynamically generated graph. After .backward() is called on a loss tensor, Autograd traverses the graph backward, computes gradients for participating model parameters, and stores them in their .grad attributes. An optimizer can then use these gradients to update the parameters.

Learn Before

Related