An artificial neuron receives inputs $$x_i$$, multiplies each by a weight $$w_i$$, adds a bias $$b$$, and computes the weighted sum $$y = \sum_i x_i w_i + b$$. The neuron may then apply a nonlinear activation function $$\sigma(y)$$ to produce its output.

University of Michigan - Ann Arbor

Google

A neural network is typically structured with an input layer, one or more hidden layers, and an output layer. Its design is historically inspired by biological neurons: dendrites are modeled as inputs $$x_i$$, synapses as synaptic weights $$w_i$$, and the nucleus as an aggregator that computes a weighted sum $$y = \sum_i x_i w_i + b$$, potentially followed by a nonlinear activation function $$\sigma(y)$$. The resulting signal is then transmitted to the output, analogous to a biological axon. A positive weight reflects an excitatory connection, while a negative value signifies an inhibitory connection.

Neural Network Representation

A computational graph is a directed dependency graph where nodes represent mathematical operations or variables, mapping the functional dependency from inputs to outputs. When evaluating a vector-valued function, this graph is traversed in a forward direction. Conversely, to compute gradients automatically, the graph is traced in a backwards direction from output to input, which involves multiplying matrices of partial derivatives along the paths.

Computational Graph

Linear regression can be conceptualized as a single-layer, fully connected neural network. In this representation, each given input feature $$x_1, \ldots, x_d$$ corresponds to an input neuron. Since the goal is to predict a single numerical value, all input neurons are directly connected to a single computed output neuron $$o_1$$. The total number of inputs $$d$$ is referred to as the feature dimensionality in the input layer.

Learn Before

Related