Neural networks: perceptron, MLP and backpropagation

The single neuron

Definition. A neuron computes a weighted sum of its inputs plus a bias, then applies an activation function $g$:

$z = w^\top x + b = \sum_i w_i x_i + b, \qquad a = g(z).$

• With $g(z) = z$ the neuron is linear regression.
• With $g = \sigma$ (sigmoid) and cross-entropy loss it is logistic regression.
• With a step function it is the perceptron.

So the familiar linear models are one-neuron networks. Neural networks gain power by stacking many neurons with non-linear activations.

Geometry. The set $w^\top x + b = 0$ is a hyperplane. The vector $w$ is perpendicular to it, and the distance from the origin to the plane is $\lvert b\rvert / \lVert w\rVert$. A single neuron with a step or sigmoid activation can therefore only separate classes with a straight boundary.

The perceptron and its learning rule

Model. $\hat y = 1$ if $w^\top x + b > 0$, otherwise $\hat y = 0$.

Learning rule (for one example at a time, learning rate $\eta$):

$w \leftarrow w + \eta\,(y - \hat y)\,x, \qquad b \leftarrow b + \eta\,(y - \hat y).$

When the prediction is right, $y - \hat y = 0$ and nothing changes. A missed positive adds $x$ to $w$; a false positive subtracts it.

Perceptron convergence theorem. If the data is linearly separable, the rule reaches a separating hyperplane after finitely many updates. Suppose every point satisfies $\lVert x\rVert \le R$ and some unit vector separates the data with margin $\gamma$. Then the number of mistakes is at most $(R/\gamma)^2$. If the data is not separable, the rule never settles.

Solved example: learning AND

Example 1. Train on AND with $w = (0, 0)$, $b = 0$ and $\eta = 1$, visiting the points in the order $(0,0), (0,1), (1,0), (1,1)$. The target is 1 only for $(1,1)$. Each row shows the point, the score $z$, the prediction, and the weights after the step.

Epoch	Point	$z$	$\hat y$	$y$	New $w$	New $b$
1	(1,1)	0	0	1	(1, 1)	1
2	(0,0)	1	1	0	(1, 1)	0
2	(0,1)	1	1	0	(1, 0)	−1
2	(1,1)	0	0	1	(2, 1)	0
3	(0,1)	1	1	0	(2, 0)	−1
3	(1,0)	1	1	0	(1, 0)	−2
3	(1,1)	−1	0	1	(2, 1)	−1
4	(1,0)	1	1	0	(1, 1)	−2
4	(1,1)	0	0	1	(2, 2)	−1
5	(0,1)	1	1	0	(2, 1)	−2

The table lists only the steps that made a mistake. In epoch 6 every point is correct: the scores are $-2, -1, 0, 1$, so only $(1,1)$ has $z > 0$. The perceptron has learned AND with $w = (2, 1)$ and $b = -2$.

What one perceptron cannot do

XOR outputs 1 for $(0,1)$ and $(1,0)$ only. Suppose a line separated it. Then:

• $b \le 0$, from $(0,0)$;
• $w_2 + b > 0$ and $w_1 + b > 0$;
• $w_1 + w_2 + b \le 0$.

Adding the two middle inequalities gives $w_1 + w_2 + 2b > 0$. So $w_1 + w_2 + b > -b \ge 0$, which contradicts the last line. No single perceptron computes XOR. This was the classic argument (Minsky and Papert, 1969) that motivated hidden layers.

Activation functions

Name	$g(z)$	$g'(z)$	Range	Notes
Step	1 if $z > 0$, else 0	0 (undefined at 0)	{0, 1}	no useful gradient
Sigmoid	$\sigma(z) = \frac{1}{1 + e^{-z}}$	$\sigma(z)(1 - \sigma(z))$	(0, 1)	max slope 0.25 at $z = 0$
Tanh	$\frac{e^z - e^{-z}}{e^z + e^{-z}}$	$1 - \tanh^2 z$	(−1, 1)	zero-centred; slope 1 at 0
ReLU	$\max(0, z)$	1 if $z > 0$, 0 if $z < 0$	$[0, \infty)$	cheap; can "die"
Leaky ReLU	$z$ if $z > 0$, else $\alpha z$	1 or $\alpha$	$\mathbb{R}$	keeps a small gradient
Linear	$z$	1	$\mathbb{R}$	used at regression outputs

Useful identities.

• $\sigma(-z) = 1 - \sigma(z)$.
• $\tanh(z) = 2\sigma(2z) - 1$.
• $\sigma(0) = 0.5$ and $\sigma'(0) = 0.25$.
• $\tanh(0) = 0$ and $\tanh'(0) = 1$.

Example 2: derivative values. At $z = 0.5$: $\sigma(0.5) \approx 0.6225$, so $\sigma'(0.5) \approx 0.6225 \times 0.3775 \approx 0.235$. At $z = 4$: $\sigma(4) \approx 0.982$, so $\sigma'(4) \approx 0.018$. A saturated sigmoid passes back almost no gradient.

Dying ReLU. If a ReLU unit's input is negative for every training example, its gradient is always 0 and it never recovers. Leaky ReLU and careful learning rates prevent this.

Output layers and losses

Task	Output activation	Loss
Regression	linear	squared error $\frac12(\hat y - y)^2$
Binary classification	sigmoid	binary cross-entropy $-[y\ln\hat y + (1 - y)\ln(1 - \hat y)]$
$K$-class classification	softmax	cross-entropy $-\ln \hat y_{\text{true}}$

Softmax. $\operatorname{softmax}(z)_k = \dfrac{e^{z_k}}{\sum_j e^{z_j}}$. The outputs are positive and sum to 1. Adding the same constant to every $z_k$ leaves them unchanged, and implementations subtract $\max_k z_k$ for numerical safety.