FP8-Capable Hardware Does Not Make Every Training Quantity FP8

Ask which part of training the FP8 claim describes

Mixed precision means using different numerical formats within one computation. A training proposal therefore needs a policy that assigns formats to operations and stored values. Saying that a GPU supports FP8 supplies a hardware capability; it leaves that policy unspecified.

Start with four questions. What formats reach the arithmetic as operands? What format holds the running accumulation? Which weight representations remain stored? What formats carry and retain gradients? An operand is an input value used by an operation. A weight is a learned model parameter. A gradient describes how the training loss changes with a value or parameter.

The concrete FP8 examples here come from NVIDIA's archived Transformer Engine 2.14.0 documentation. Its primer discusses H100 inputs, and its optimization discussion names Hopper Tensor Cores. These examples establish particular documented choices, rather than a default policy for every framework or FP8-capable GPU.

References: [M1][T1][T2]

Eight bits do not identify one floating-point format

A format defines how bits encode a value. Dynamic range concerns the magnitudes it can represent, from small nonzero values to large ones. Precision concerns how finely values are distinguished within that range. Equal storage width can allocate bits differently between these purposes.

The 2.14.0 primer distinguishes E4M3, with one sign bit, four exponent bits and three fraction bits, from E5M2, with one sign bit, five exponent bits and two fraction bits. Their largest finite magnitudes are 448 and 57,344 respectively. E5M2 trades finer precision for greater range.

Conversion chooses a representable value in the destination format and may lose information. Scaling changes the relationship between stored numbers and represented values; its use is another implementation choice. The primer also warns that some operations are unsuitable for FP8. A proposal must identify the operations covered and the actual representation, including its scaling policy, rather than assuming every value becomes the same eight-bit type.

References: [T1][F1]

Separate arithmetic accumulation from gradient scheduling

Matrix multiplication combines many products into output values. The accumulator holds the running combination; its format is a separate question from the formats of the input operands. NVIDIA's FP16 mixed-precision guide illustrates this distinction with lower-precision inputs and separately specified accumulation outputs. That historical FP16 example explains the roles without establishing an FP8 implementation.

For FP8 training, the 2.14.0 optimization discussion describes a Hopper option to accumulate matrix products into FP32 and generate FP32 weight gradients into a separate gradient tensor. It is an available path in that discussion, not evidence that all FP8 training produces FP32 gradients.

Keep the internal arithmetic accumulator, the operation's output format and the retained gradient buffer separately identified. They describe different points in the numerical path, even when an implementation connects them directly.

Gradient accumulation across microbatches is another concept: several smaller computations contribute gradients before an optimizer updates parameters. Its scheduling question is how much work contributes to an update. The numerical-format question is how those contributions are represented and combined. The related microbatch article owns the scheduling explanation; an accumulation-round count does not answer the format question.

References: [M1][T2]

Follow the retained weight through conversion

The 2.14.0 optimization guide also discusses converting higher-precision weights for FP8 computation and retaining converted weights for reuse across microbatches. This exposes another missing field in an FP8 label: the representation supplied to an operation need not be the only weight representation retained.

Ask which copy receives parameter updates, which copy supplies a particular operation and when a converted representation is refreshed. Then ask which copies coexist. A description of the operand alone answers none of those ownership or lifetime questions.

The documentation warns that cached and uncached paths need not produce bitwise-identical outputs. Treat conversion and reuse as part of the identified implementation when comparing results. This discussion supplies no caching recommendation or estimate of the allocations in a particular training system.

References: [T2]

MP-216: one set of positions, three simultaneous arrays

Fictional example MP-216 stipulates 1,024 corresponding learned parameter positions and three arrays retained at the same instant. Every array, width and lifetime in this example is invented.

The first array is a weight copy using two bytes per value. Its raw payload is 1,024 x 2 = 2,048 B.

The second is a converted operand copy using one byte per value. Its raw payload is 1,024 x 1 = 1,024 B.

The third is a gradient accumulation array using four bytes per value. Its raw payload is 1,024 x 4 = 4,096 B.

Together, these stipulated payloads occupy 2,048 + 1,024 + 4,096 = 7,168 B. With 1 KiB defined as 1,024 bytes, the total is 7 KiB.

There are 3,072 stored scalar entries across the arrays, but they do not constitute 3,072 independent learned parameters. The arrays serve different representation or gradient roles associated with the same 1,024 parameter positions. The one-byte operand copy accounts for only 1 KiB of the stipulated total.

MP-216 is raw-payload accounting, not a Transformer Engine allocation forecast. It excludes scales, metadata, other tensors, optimizer state and allocator behavior. Its conclusion depends on the invented simultaneous retention. A real memory comparison needs the actual arrays and their lifetimes; neither this seven-byte-per-position sum nor the one-byte operand width is a universal training-memory formula.

A wider accumulator cannot undo operand rounding

Finite floating-point formats cannot represent every real value exactly. NVIDIA's floating-point white paper explains rounding and shows that operation ordering can change a computed result.

Consider the information consequence: if conversion maps two distinct original values to the same encoded operand, later arithmetic receiving only that operand cannot tell which original value was supplied. Increasing the accumulator's precision does not restore the distinction. This is a deduction about lost information, not a claim that a particular model will fail.

Accumulator precision still matters to subsequent arithmetic. It simply answers a different question from how faithfully the inputs were represented. Likewise, storing an output in a wider format does not prove that all earlier work used that format.

Trained quality concerns the resulting model on the agreed evaluation. A bit-width label cannot supply that result, and bitwise identity is a different requirement from acceptable task quality. Agree the relevant numerical comparison and quality criteria with the appointed model team.

References: [F1]

Compare two claimed FP8 implementations by what is specified

Imagine two proposals for the same requested training workload. Proposal A names an FP8-capable GPU and labels one tensor FP8. Proposal B identifies the exact hardware, software versions, operations using reduced-precision operands, operand formats and scaling, arithmetic accumulation formats, output formats, retained weight copies and gradient buffers.

These are fictional descriptions of proposal contents, with no test results. B makes the numerical policy inspectable. It has not, merely by supplying more detail, demonstrated memory fit, trained quality or speed.

For either proposal, connect the identified policy to evidence for the agreed model revision, data and objective, input shapes, batch and update schedule, system size and training outcome. Ask the evaluating team to retain the actual configuration with its results. If a format or conversion choice changes between the described implementation and the evaluated one, resolve that difference before relying on the comparison.

Equal hardware does not prove equal kernels. Equal nominal input precision does not prove equal numerical results, output quality or throughput. Compare the complete work and its accepted outcome using the related benchmark guide. No universal speed, memory or energy ratio follows from replacing one width label with another.

Carry the selected equipment into a physical deployment inquiry

Once the software and platform owners have selected equipment, Leviathan's Complete Physical Infrastructure Deployment service provides a separate next step for agreed receiving, rack integration, onsite installation, IT cabling and physical handoff records. Use the related service page and project inquiry when those physical stages are needed.

Share a nonconfidential equipment list, quantities, destination, rack layout, included cable endpoints and target window. Keep the numerical-policy evaluation with the appointed software team. This educational comparison offers no model operation, tuning or training service; physical installation does not approve an FP8 implementation or establish its trained quality.

References: [L1]