What is Neural Network?

The video opens by highlighting how the human brain can recognize numbers even when they are written in low resolution, poorly drawn, or vary significantly in pixel patterns. This ability, which we do without conscious effort, becomes very challenging when designing a program to do the same.
Neural networks are inspired by this natural capability, mimicking, at a basic level, how neurons in the human brain process and identify visual information.

Neurons: These are the basic units in a neural network, each holding a number between 0 and 1, representing “activation” levels. The network starts with an input layer of neurons that correspond to the 784 pixels (28x28) of a grayscale image. Each pixel’s brightness level is converted to a value between 0 (black) and 1 (white).
Hidden Layers: Between the input and output layers, there are hidden layers—these are layers that transform inputs in steps before arriving at an output. Here, the video uses two hidden layers, each with 16 neurons.
Output Layer: The final layer has 10 neurons, each representing a digit from 0 to 9. The “activation” of these neurons indicates how strongly the network thinks the input resembles each possible digit.

Learning Components: When humans recognize a number, we identify unique components (for example, loops and lines). Neural networks attempt to replicate this by assigning certain neurons to detect specific visual patterns or subcomponents (like edges or curves).
Edges and Patterns: The initial hidden layer neurons might identify edges, while later layers combine these edges into higher-level patterns (e.g., loops or lines) that form digits. This process is hierarchical, meaning simple patterns combine into complex ones across layers.

Weights: These are parameters that adjust the influence of one neuron on another. Each connection between neurons has a weight that defines the strength and type of influence. For example, weights help neurons in the hidden layers focus on specific pixel patterns.
Biases: Each neuron also has a bias, which controls how easily it gets “activated.” Biases are like thresholds that decide when neurons should respond to certain patterns.
Weighted Sum and Activation Function: For each neuron in a layer, the network calculates a weighted sum of the inputs (from the previous layer) and applies an activation function, like the sigmoid function, which limits the output between 0 and 1. This transformation makes the network’s decision-making process smoother.

Training a neural network is the process of adjusting its weights and biases to correctly classify inputs, such as recognizing digits from images.
Adjusting the values of weights and biases to minimize errors over many training examples allows the network to “learn” which patterns correlate with each digit.

Sigmoid Function: An older function used to “squish” values to a 0-1 range, making it biologically analogous to the way neurons activate (either firing or not).
ReLU Function: This activation function outputs the maximum of 0 or the input value, making it simpler and often faster for training deep networks. ReLU is widely preferred in modern neural networks because it reduces problems like the vanishing gradient, where gradients become too small to update weights effectively in deeper layers.

Neural networks heavily rely on matrix operations to compute activations efficiently. Each layer’s neuron activations can be represented as vectors, and the weights as matrices. This way, all connections from one layer to the next can be processed with matrix-vector multiplication.
Biases are added as vectors to these matrices to create a single, compact expression for each layer’s transformation, allowing for faster computation with optimized libraries.

The entire neural network functions as one large, complex mathematical function that takes 784 inputs (pixel values) and outputs 10 values (one for each digit probability). Adjusting the 13,000 weights and biases allows the network to map various pixel configurations to the correct digits.
The process is sophisticated, involving many interconnected components, making it capable of handling complex tasks like digit recognition.

In practice, neural networks learn by adjusting weights and biases based on training data. By feeding in labeled images and comparing outputs to known labels, the network “learns” to minimize errors and improve its classification ability.