// HACKER NEWS — CYBERSECURITY
Why back propagation goes backward
The usual explanation of backpropagation (Rumelhart et al., 1986), the algorithm used to train neural networks, is that it is propagating errors for each node backwards. But when I first learned about the algorithm, I had a question that I could not find answered directly: why does it have to go backwards? A neural network is just a composite function, and we know how to compute the derivatives of composite functions using the chain rule. Why don’t we just compute the gradient in a forward pass? I found that answering this question strengthened my understanding of backprop.
I will assume the reader broadly understands neural networks and gradient descent and even has some familiarity with backprop. I’ll first setup backprop with some useful concepts and notation and then explain why a forward propagation algorithm is supoptimal.
Recall that the goal of backprop is to efficiently compute ∂f/∂θi\partial f / \partial \theta_i∂f/∂θi for every weight θi\theta_iθi in a neural network fff. To frame the problem, let’s reason about an arbitrary weight θ1\theta_1θ1 and node vvv somewhere in fff:
To be clear, the node vvv refers to the output value of the node after passing the weighted sum of its inputs through an activation function σ\sigmaσ, i.e.:
u=θ1t1+θ2t2+⋯+θntnv=σ(u)
\begin{aligned}
u &= \theta_1 t_1 + \theta_2 t_2 + \dots + \theta_n t_n \\
v &= \sigma(u)
\end{aligned}
uv=θ1t1+θ2t2+⋯+θntn=σ(u)
Note that in a typical diagram, uuu, σ\sigmaσ, and vvv would all be a single node, denoted by the dashed line. In my mind, the most important observation needed to understand backprop is this: most of computing ∂f/∂θ1\partial f / \partial \theta_1∂f/∂θ1 can be done locally at every node because of the chain rule:
∂f∂θ1=∂f∂v∂v∂u∂u∂θ1
\frac{\partial f}{\partial \theta_1} =
\frac{\partial f}{\partial v}
\frac{\partial v}{\partial u}
\frac{\partial u}{\partial \theta_1}
∂θ1∂f=∂v∂f∂u∂v∂θ1∂u
We can compute ∂v/∂u\partial v / \partial u∂v/∂u analytically; it just depends on the definition of σ\sigmaσ. And we know that ∂u/∂θ1=t1\partial u / \partial \theta_1 = t_1∂u/∂θ1=t1. So at every node vvv, if we knew ∂f/∂v\partial f / \partial v∂f/∂v, we could compute ∂f/∂θ1\partial f / \partial \theta_1∂f/∂θ1.
The challenge with computing ∂f/∂v\partial f / \partial v∂f/∂v is that downstream nodes depend on the value of vvv. Thankfully, the multivariable chain rule has the answer. Given a multivariable function g(w1,w2,…,wm)g(w_1, w_2, \dots, w_m)g(w1,w2,…,wm) in which each wiw_iwi is a single variable function wi(v)w_i(v)wi(v), the multivariable chain rule says:
∂g∂v=∂∂vg(w1(v),w2(v),…,wm(v))=∑j∂g∂wj∂wj∂v
\frac{\partial g}{\partial v} = \frac{\partial}{\partial v} g(w_1(v), w_2(v), \dots, w_m(v)) = \sum_{j} \frac{\partial g}{\partial w_j} \frac{\partial w_j}{\partial v}
∂v∂g=∂v∂g(w1(v),w2(v),…,wm(v))=j∑∂wj∂g∂v∂wj