If your embedding model collapses to a single point, your alignment metric will report perfection. A new analysis of NT-Xent, the InfoNCE loss powering SimCLR and most modern embedding models, demonstrates that a completely collapsed encoder achieves a cosine similarity of 1.0000. This exposes a critical blind spot in standard evaluation pipelines: the metric most engineers trust to measure representation quality fails to detect the worst possible failure mode.

Batch Size Breaks Loss Comparability

The NT-Xent objective relies on the batch as the negative pool, meaning batch size fundamentally alters the task difficulty rather than just introducing gradient noise. The author provides a live browser-based demo showing that loss values are incomparable across different batch sizes. At a batch size of 2, the collapse floor is log(3), or approximately 1.0986. At a batch size of 64, the floor rises to log(127), approximately 4.8442. If you are comparing loss curves between runs with different batch sizes, you are comparing apples to oranges; a lower loss at N=256 versus N=64 does not indicate better learning, only a harder task with more negatives.

Uniformity Is the Only Honest Signal

When an encoder collapses, every embedding becomes the exact same vector. This results in an alignment score of 1.0000, which looks like the ideal outcome on a dashboard. However, uniformity, which measures how spread out the embeddings are on the hypersphere, correctly flags this failure. In the collapsed state, uniformity hits 1.0000, indicating zero spread. A well-separated encoder shows uniformity near -0.0229, while a collapsed one sits at 1.0000. The loss value itself confirms this: at collapse, the loss equals exactly log(2N-1), because the softmax becomes uniform over all options, rendering temperature irrelevant.

Temperature Is a Negative Mining Knob

Lowering the temperature parameter (ฯ„) is often described as 'sharpening' the distribution, but it effectively acts as a hard-negative miner. As ฯ„ decreases, the gradient share concentrated on the hardest negative climbs dramatically. At ฯ„=1.00, only 5.0% of the negative gradient goes to the hardest negative. At ฯ„=0.01, that share jumps to 63.1%, and independent checks show it can reach 98.8% at ฯ„=0.02. This means large batches become inefficient at low temperatures because the model ignores most negatives, discarding the computational effort used to generate them.

Numerical Stability Is Not Optional

Training with very low temperatures introduces severe numerical risks. At ฯ„=0.001, a cosine similarity of 1.0 becomes a logit of 1000. Standard floating-point operations will evaluate Math.exp(1000) as Infinity, leading to NaN losses. A NaN loss is not a large loss; it is the absence of a gradient, causing training to silently stop learning. The author recommends using a stable logsumexp implementation that shifts the maximum value out to prevent overflow. This simple numerical guard is essential for any production-grade contrastive learning pipeline using low temperatures.

Key Takeaways

  • A collapsed encoder scores 1.0000 on alignment, making alignment alone a dangerous metric for detecting representation collapse.
  • NT-Xent loss values are not comparable across different batch sizes due to the changing collapse floor log(2N-1).
  • Uniformity is the necessary companion metric to alignment; it correctly identifies collapse where alignment fails.
  • Low temperatures (ฯ„ < 0.1) concentrate gradients on hardest negatives, effectively wasting the signal from the rest of the batch.
  • Always use logsumexp for numerical stability when training with low temperatures to prevent silent NaN failures.

The Bottom Line

Stop trusting alignment scores in isolation. If you aren't monitoring uniformity alongside it, you might be celebrating a model that has learned absolutely nothing.