NCCL is not magic. It is a protocol.

NCCL is a protocol, not a library. Once you see it that way, the failure modes make sense. This is the note we wish someone had handed us before our first multi-node run.

## What NCCL actually is

A collective communication protocol over RDMA or TCP. All-reduce, all-gather, broadcast, reduce-scatter — these are the primitives your trainer depends on. When NCCL hangs, it's because one of these primitives is waiting for a peer that will never respond.

## The four failure modes

1. **Topology mismatch** — NCCL's ring is built from hostnames, not physical layout.
2. **MTU shenanigans** — different values across NICs silently halves bandwidth.
3. **GPU affinity drift** — CUDA_VISIBLE_DEVICES lies if you aren't careful.
4. **The silent SIGPIPE** — one peer crashes, the rest sit forever.

## Instrumentation that matters

Enable `NCCL_DEBUG=INFO` in pre-prod. Log every collective start and end. When you see a collective that started on N-1 peers, you know exactly who the missing one is.