FP8 training -- where matrix multiplications run in 8-bit floating point instead of the usual 16-bit -- has been a reliable way to squeeze more throughput out of NVIDIA hardware for a while. AMD's Instinct GPUs support FP8 too, but getting it to actually work well in PyTorch required a non-trivial amount of plumbing. That plumbing is now done. AMD and Meta engineers have upstreamed a full suite of FP8 optimizations into mainline TorchAO and TorchTitan, so AMD Instinct GPUs get competitive FP8 performance out of the box with no AMD-specific installs required.

The hidden correctness bug that had to come first

Before any performance work could land, there was a silent correctness problem to fix. AMD Instinct GPUs use a variant of FP8 called FNUZ (Finite, No NaN, Unsigned Zero), specifically e4m3fnuz, which has a maximum representable value of 240. NVIDIA's FP8 format, e4m3fn, has a max of 448. TorchAO was hardcoded to NVIDIA's format, so on AMD hardware it was computing scaling factors against the wrong ceiling.

Because e4m3fnuz has no NaN/Inf encodings, the overflow did not raise an error -- it degraded model quality instead. Selecting the correct format is therefore a correctness requirement, not a tuning option. The fix was hardware auto-detection, so TorchAO now selects the correct format automatically. Alongside this, the team fixed MFU (Model FLOP Utilization) reporting to use the correct MI300X peak FLOPS, and added platform-specific loss baselines for FNUZ numerics.

Dense models: a clean 13.4% win

Rowwise FP8 with a high-precision weight-gradient recipe -- where the weight-update GEMM stays in BF16 while the forward and gradient-input GEMMs use FP8 -- delivers a 13.4% throughput gain over BF16, with peak memory nearly identical (~39 GB). The win comes from faster FP8 matrix cores, not memory savings. This was measured on Llama3-8B across 8×MI300X GPUs with FSDP2 and torch.compile.

Bar chart comparing FP8 training throughput on 8×MI300X GPUs across BF16 and FP8 configurations, showing up to 14.7% improvement

The "rowwise" part matters here. Rowwise scaling is better at handling outliers than tensorwise scaling, so these recipes are different points on the accuracy vs performance curve. Tensorwise is fastest but coarsest; rowwise trades a small amount of compute for better numerical fidelity. For most training workloads, rowwise is the right default.