Every tensor is a
flat run of bytes.
Rank 0 to rank n, the storage never changes shape: one contiguous buffer plus the strides that pretend it is a rectangle. Watch a 6×8 matrix unroll onto its tape, then read the same bytes as its own transpose.
Same number,
six budgets.
Sign, exponent, fraction. Exponent bits buy range, fraction bits buy precision, and every format is a different way of spending a fixed number of bits. Move the slider and watch the same value round six ways.
The bus is the
slowest thing here.
A tensor built on the CPU has to be pinned, pushed down sixteen PCIe lanes and parked in HBM before a single SM can touch it. The dtype you chose decides how long that takes — and whether it fits at all.
Bytes are the bill,
FLOPs are the clock.
Count a training run in FLOPs, divide by what one card can actually deliver in a week, then ask the question that decides which of the two numbers you are paying: how much arithmetic does this kernel do per byte it moves?