Modules D02, D03, B07 · concept player

Ring all-reduce, one step at a time

All-reduce equals reduce-scatter plus all-gather. Every step, every rank sends to its ring neighbor at once — step through both phases and watch the byte volume per rank converge to 2S no matter how many ranks you add, while the step count keeps climbing.

Ranks P

Payload size S

Step: 0 of 14

14total steps, 2(P−1)

1.75 MBbytes moved per rank, this run

2.00 MBthe limit as P → ∞, 2S

The ring, at this step

Reduce-scatter, step

Exact arithmetic on the ring formula, not a measurement. Per-rank traffic is 2(P−1)/P × S; step count is 2(P−1). At P=8 that is 14 steps (D03 section 3 gives 2,046 steps at P=1024 — volume converges, latency does not). The latency/bandwidth crossover is S* = P × α × B; at P=8, α=8μs, B=50 GB/s that is 3.05 MiB (B07 section 2) — below it you are paying for launches, not wire.

This page needs JavaScript, because the control is the point. The static version of the same argument is the infographic below, which module B07 embeds.