If you've been following the AI/ML notes series on DEV.to, you know that vanishing gradients have been the arch-nemesis of deep neural network training for decades—gradients shrinking to nearly nothing as they propagate backward through layers, leaving early layers essentially frozen. But there's a darker mirror image to that problem: the exploding gradient. Instead of dying out, gradients balloon uncontrollably during backpropagation, rendering training unstable and often completely broken from iteration one.

What Exactly Is an Exploding Gradient?

During backpropagation, gradients are computed by multiplying error signals through each layer using the chain rule. If those weight matrices have eigenvalues greater than 1, each multiplication amplifies rather than attenuates the signal. In a network with many layers, these repeated multiplications compound exponentially—a gradient that starts at 0.1 might become 10, then 100, then NaN after just a handful of layers. The model weights update by enormous magnitudes, overshooting any meaningful minimum entirely.

Why Deep Networks Are Particularly Vulnerable

Recurrent neural networks (RNNs) are the classic victims here because they process sequential data through the same weights repeatedly at each time step. A sequence length of 100 means your gradients get multiplied together up to 100 times during backpropagation through time (BPTT). Even small weightization errors become catastrophic over long sequences. But vanilla deep feedforward networks aren't immune either—poorly conditioned weight matrices or aggressive learning rates can trigger the same explosive behavior.

The Warning Signs Are Obvious Once You Know Them

Practically, you'll see model loss spike to infinity or NaN within the first few training epochs. Weight values will balloon into astronomically large numbers (think 10^15 and beyond). If you're monitoring gradient norms during training, a healthy network might show gradients in the range of 0.01-1.0; an exploding network will display values climbing by orders of magnitude each batch.

Gradient Clipping: The Circuit Breaker

The most common mitigation is gradient clipping—capping gradients to a maximum value before the weight update. If ||g|| > threshold, rescale it: g_clipped = g * (threshold / ||g||). PyTorch implements this as torch.nn.utils.clip_grad_norm_() and clip_grad_value_(). This doesn't fix the underlying numerical instability, but it prevents training from completely diverging.

Better Initialization and Normalization

Xavier/Glorot initialization scales weights based on layer fan-in and fan-out to keep variance stable across layers. He initialization adapts this for ReLU activations specifically. Layer normalization and batch normalization also help by keeping intermediate activations in reasonable ranges, which stabilizes the gradients flowing backward.

Key Takeaways

  • Exploding gradients occur when weight matrices amplify signals during backpropagation instead of attenuating them
  • Deep networks and RNNs processing long sequences are most vulnerable due to repeated multiplications
  • Symptoms include NaN losses, astronomically large weights, and gradient norms climbing unboundedly
  • Gradient clipping acts as a circuit breaker; proper initialization and normalization address root causes

The Bottom Line

The exploding gradient problem is fundamentally a numerical stability issue that exposes how fragile naive backpropagation becomes in deep architectures. Understanding why it happens isn't just academic—it directly informs every architectural decision from RNN design to weight initialization strategy. If you're building anything deeper than a few layers, clip your gradients by default and monitor those norm logs religiously.