Neural Tangent Kernel: A theoretical view of Deep Learning Theory
What the Neural Tangent Kernel says about gradient descent in very wide networks: where the kernel comes from, why it stays fixed, and what the theory cannot explain.
In 2018 (a bit old by now), a paper by Arthur Jacot, Franck Gabriel, and Clément Hongler dropped a theoretical result on the deep learning community. Their work on Neural Tangent Kernels (NTKs) showed that neural networks, under certain conditions, behave exactly like classical kernel methods, a connection that was both surprising and profound.
In this post, I want to walk through where this kernel comes from, why it matters, and also what it doesn’t explain.
The Setup: Watching the Function, Not the Weights
Let’s start with a network $f(x; \theta)$, a training set $\lbrace (x_i, y_i) \rbrace_{i=1}^n$, and the usual squared loss:
\[L(\theta) = \frac{1}{2} \sum_{i=1}^n \big(f(x_i; \theta) - y_i\big)^2\]To keep things simple, imagine gradient descent with a tiny learning rate, so the weights move smoothly over time:
\[\frac{d\theta}{dt} = -\nabla_\theta L(\theta) = -\sum_{i=1}^n \nabla_\theta f(x_i; \theta)\,\big(f(x_i; \theta) - y_i\big)\]Usually we look at how the weights move. The NTK idea is to look at how the predictions move instead. With the chain rule, for any input $x$:
\[\frac{d}{dt} f(x; \theta_t) = -\sum_{i=1}^n \Theta_t(x, x_i)\,\big(f(x_i; \theta_t) - y_i\big)\]where
\[\Theta_t(x, x') = \nabla_\theta f(x; \theta_t)^\intercal \, \nabla_\theta f(x'; \theta_t)\]What’s happening here?
- $\nabla_\theta f(x; \theta_t)$ is the gradient of the output with respect to all the weights
- $\Theta_t(x, x’)$ is the dot product of two of these gradients - this is the Neural Tangent Kernel
- $f(x_i; \theta_t) - y_i$ is the error on training point $x_i$
I like to think of the kernel as a “how connected are these two inputs” score. If I take a gradient step to fix the error on $x_i$, $\Theta(x, x_i)$ tells me how much my prediction on $x$ moves too. Inputs with similar gradients get pulled together, inputs with orthogonal gradients don’t affect each other at all.
Nothing here is an approximation yet, this holds for any differentiable model. The problem is that $\Theta_t$ depends on the weights, so it keeps changing during training, and we’re back to something hard to analyze.
The Infinite-Width Limit
This is where the paper comes in. Jacot and his coauthors looked at fully connected networks where every hidden layer becomes infinitely wide, with the weights scaled in a specific way (each layer’s output is divided by $\sqrt{n}$, where $n$ is the width of the previous layer).
They proved two things:
- At initialization, the kernel stops being random. It converges to a fixed kernel $\Theta$ that depends only on the architecture (depth, activation, initialization), not on the random weights you happened to draw.
- During training, the kernel doesn’t move: $\Theta_t = \Theta$ the whole time.
The second point is the surprising one. Why would the kernel stay frozen?
The intuition is that with a huge number of weights sharing the work, each weight only needs to change by a tiny amount (on the order of $1/\sqrt{n}$) for the output to change a lot. The gradients barely notice these small moves, so the kernel stays the same while the predictions still go all the way to a solution.
Lee and coauthors made this more concrete: a very wide network trains exactly like its first-order Taylor expansion around the initial weights:
\[f_{\text{lin}}(x; \theta) = f(x; \theta_0) + \nabla_\theta f(x; \theta_0)^\intercal (\theta - \theta_0)\]This is just a linear model in $\theta$, where the features are the gradients at initialization. Chizat, Oyallon, and Bach called this the lazy training regime.
Training Becomes a Linear Equation
Once the kernel is fixed, things get much simpler. Let’s stack the predictions on the training set into a vector $u_t$ and write $\Theta$ for the $n \times n$ kernel matrix on the training inputs. The training dynamics become:
\[\frac{du_t}{dt} = -\Theta\,(u_t - y) \quad\Longrightarrow\quad u_t - y = e^{-t\Theta}\,(u_0 - y)\]This is a linear differential equation, and we can solve it in closed form. If we let training run until the end, the prediction on a new input $x$ is:
\[f_\infty(x) = f_0(x) + \Theta(x, X)\,\Theta^{-1}\,(y - u_0)\]If you’ve seen kernel regression before, this is exactly it, just with the NTK as the kernel. So an infinitely wide network, fully trained, is a kernel machine, and we can compute its predictions without running a single step of gradient descent.
What the Eigenvalues Tell Us
Let’s write the kernel matrix with its eigenvalues, $\Theta = \sum_k \lambda_k v_k v_k^\intercal$. Along each eigenvector, the error shrinks on its own:
\[v_k^\intercal (u_t - y) = e^{-\lambda_k t}\, v_k^\intercal (u_0 - y)\]Two things come out of this:
- Convergence: if all $\lambda_k > 0$, the training error goes to zero, and fast (exponentially). This was one of the first clear answers to why gradient descent manages to reach zero training loss on huge non-convex networks.
- Order of learning: directions with large eigenvalues are learned first, small ones much later. For common architectures, the large eigenvalues correspond to smooth, simple patterns. This matches what Rahaman and coauthors observed and called spectral bias: networks learn the simple structure before the fine details.
The Limits of the NTK
This is the part I think matters most, because it’s easy to get too excited about the NTK.
No feature learning: in the NTK regime, the features (the gradients at initialization) never change. The network only learns a linear combination of features it started with. But learning good representations is probably the main reason deep learning works so well! Yang and Hu showed that this comes from the way the weights are scaled, not from the width itself: with a different scaling, infinitely wide networks do learn features.
Kernels are worse than real networks: Arora and coauthors computed the exact convolutional NTK on CIFAR-10. It did well for a kernel method, but still about 6% below the equivalent finite network, and that was without tricks like batch normalization. Real networks learn something the kernel misses.
Cost: kernel regression needs the full $n \times n$ kernel matrix and its inverse, which scales cubically with the number of training points. The NTK is a tool to understand networks, not a replacement for training them on big datasets.
Real networks aren’t infinite: they’re wide but finite, and trained with large learning rates that push them out of the lazy regime. Understanding what happens at finite width is still an active research topic.
Why It’s Still Worth Knowing
Even with these limits, the NTK changed how people think about deep learning:
- It gives a setting where we can actually prove that training works, which made the success of huge networks a bit less mysterious
- It connects architecture choices (depth, activation, initialization) to something we can compute and compare: the kernel
- It gives strong baselines on small datasets, where kernel methods are competitive
If you want to play with it, Google’s Neural Tangents library computes these kernels for many architectures, so you can compare a network with its infinite-width version directly.
What’s Next?
The way I see it, the NTK describes what a network does when it’s not allowed to learn new features. Real networks are interesting precisely because they leave that regime, and a lot of current theory is about understanding what happens when they do. That’s probably a topic for a future post.
If you want to go further, these are the papers I mentioned:
- Jacot, Gabriel, and Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018.
- Lee et al., Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent, NeurIPS 2019.
- Chizat, Oyallon, and Bach, On Lazy Training in Differentiable Programming, NeurIPS 2019.
- Arora et al., On Exact Computation with an Infinitely Wide Neural Net, NeurIPS 2019.
- Rahaman et al., On the Spectral Bias of Neural Networks, ICML 2019.
- Yang and Hu, Feature Learning in Infinite-Width Neural Networks, ICML 2021.