---
title: "Which GPU Is Fastest? How to Compare AI Performance"
description: "Leviathan Systems — Technical content Published 2026-10-15; updated 2026-10-19."
canonical: "https://www.leviathansystems.co/articles/which-gpu-is-fastest-workload-benchmarks"
last-updated: "2026-10-19"
---

# Which GPU Is Fastest? How to Compare AI Performance

Leviathan Systems — Technical content
Published 2026-10-15; updated 2026-10-19.

The fastest GPU depends on the workload and the complete system. For training, compare time to a stated quality target. For inference, compare throughput under the required response-time and quality conditions. Keep the model, test version, precision, accelerator count and software visible. Peak FLOPS alone cannot establish which system will finish your job first.

## Define the job before the ranking

A ranking needs a finish line. A training team may care about reaching a quality target before a scheduled evaluation. An interactive application may care about how quickly the first token appears and how smoothly the rest arrives. A batch-processing team may instead care about completed work across a long queue. One system can be a useful candidate for one job without leading the others.

Write the requirement as an observable outcome: the model or application, input size, output size, accepted quality, number of simultaneous requests and the response-time limit. Add whether the comparison concerns one accelerator, one server or a larger cluster. If those fields are missing, a list headed top GPUs by speed is a list of product names rather than evidence for the reader's task.

This guide explains how to build that comparison. It does not claim an overall winner, predict application performance or report a Leviathan-operated benchmark.

## Separate the speed measures

Use the metric that matches the decision. MLCommons describes its training benchmark in terms of elapsed time to a defined quality target. Its inference rules distinguish load scenarios and response-time conditions. That means a training-time result and an inference-throughput result cannot be merged into one speed column.

For language-model serving, time to first token describes the initial wait; time per output token describes the average interval during generation under the benchmark's definition. Aggregate token throughput concerns the system's output across its workload. A throughput figure needs its request pattern and response-time requirements alongside it.

References: [M1](https://mlcommons.org/benchmarks/training/), [M2](https://github.com/mlcommons/inference_policies/blob/ff7edba545fded369e7e7e3d5a2f0bab4a95eece/inference_rules.adoc)

Choose the measurement before choosing a winner
| Reader task | Record | Question still open | Source |
| --- | --- | --- | --- |
| Train or fine-tune a stated model | Elapsed time to the specified quality target, plus system scale | Does our own training recipe reach the required quality? | Planning analysis |
| Serve an interactive model | Initial response and generation latency at the required load | Does the response-time distribution meet the application requirement? | Planning analysis |
| Process a batch queue | Completed work per unit time under the stated test conditions | Is that batch pattern representative of our queue? | Planning analysis |
| Compare arithmetic capability | Data type, dense/sparse basis, operation type and hardware scope | Does the application use that operation and precision? | Planning analysis |
| Evaluate interconnect or memory | The measured or specified interface, unit and direction | Is that interface the limiting part of the application? | Planning analysis |
Original comparison worksheet. It is not a published benchmark result or a claim that any one interface predicts application speed.

## Read peak FLOPS with the footnotes

NVIDIA's current HGX specification table gives a useful example. For the eight-GPU HGX B300 and HGX B200 platforms, its FP4 Tensor Core row lists 144 and 108 PFLOPS for B300, and 144 and 72 PFLOPS for B200, with each pair explicitly labeled sparse and dense. The sparse entries are equal; the dense entries differ. Retaining the footnote changes what the comparison actually says.

The arithmetic 108 / 72 = 1.5 describes the ratio between those two published dense FP4 specifications. It is not evidence that an application runs 1.5 times faster. Nor is either platform's total a specification for one GPU. These are platform specifications from the OEM page retrieved on October 15, 2026, not measured model-serving results.

When a graph loses the precision, sparsity or hardware-count label, return to the original table before drawing a conclusion. Do the same when a vendor projection appears beside a measured result: preserve the projection label and its stated conditions.

References: [N1](https://www.nvidia.com/en-us/data-center/hgx/)

## Understand what a system result includes

An accelerator name is only part of the tested configuration. MLCommons publishes system and software information alongside results, including the accelerator count. Keep that information attached when building a shortlist. A result from a larger system may be valuable evidence for a larger deployment, but it does not become a measurement of one accelerator after dividing by the count.

For a fair reader comparison, collect the CPU and host-memory configuration, accelerator variant, number of nodes, interconnect, software stack and version. Ask which fields differ, rather than pretending that matching the GPU family makes the rest identical. If software is different, describe the result as a comparison of the submitted configurations. Do not attribute the whole difference to the silicon.

Keep the original result and unit. A calculated per-accelerator value can help describe resource use if its calculation is explicit; it must not be presented as a separately measured single-device result or used to predict how the workload scales.

References: [M3](https://mlcommons.org/benchmarks/inference-datacenter/)

## Use one benchmark slice at a time

Start with one workload and one scenario. Then retain its quality requirement, division and availability category. In the MLCommons framework, Closed and Open divisions permit different implementation choices; Available, Preview and Research/Development/Internal categories also describe different availability conditions. Those labels belong in a comparison, not in a footnote that disappears when someone copies the table.

For inference, an Offline scenario sends the work as a batch; a Server scenario applies a request pattern with latency requirements. Keep those results in separate comparisons. A larger Offline number does not show that the same system can serve an interactive application at the desired response time.

Use the benchmark version recorded with the result. If you want to compare different versions, check the publisher's compatibility guidance first. A newer release date alone does not establish that its score is comparable with the preceding release.

References: [M2](https://github.com/mlcommons/inference_policies/blob/ff7edba545fded369e7e7e3d5a2f0bab4a95eece/inference_rules.adoc), [M3](https://mlcommons.org/benchmarks/inference-datacenter/), [M4](https://github.com/mlcommons/policies/blob/a4520be877271064ceff8339b837edb67027537b/MLPerf_Results_Messaging_Guidelines.adoc)

## Create a comparison record someone else can check

A useful shortlist is reproducible as a document before anyone repeats an expensive test. Each row should point back to the original result, its configuration and the source retrieval date. Keep the filter that selected the row. If a subsequent correction invalidates a result, that record lets the reviewer identify the affected conclusion without rebuilding the entire comparison.

Use the following fields as a working record. Empty fields are unresolved questions, not permission to fill in the most likely answer. If a vendor summary and the underlying submission disagree, retain both references and resolve the mismatch before promoting the row into a recommendation.

Evidence fields for a GPU performance comparison
| Field | What to retain | Why the reviewer needs it | Source |
| --- | --- | --- | --- |
| Workload and test identity | Publisher, suite, version, exact model/task and scenario | Defines what the score measures | Planning analysis |
| Quality and implementation | Required quality level, division, precision and relevant optimization notes | Shows which implementation changes the comparison permits | Planning analysis |
| System scope | Complete system name, variant, nodes and accelerator count | Prevents rack, server and device totals being mixed | Planning analysis |
| Software | Framework, runtime, driver and other submitted version details | Keeps a system result from being relabeled as a chip-only result | Planning analysis |
| Result provenance | Published entry identifier, direct source, status and retrieval date | Allows another reviewer to locate the same evidence | Planning analysis |
| Interpretation | Metric direction, unit, stated limitations and any explicit calculation | Separates the published measurement from the buyer conclusion | Planning analysis |
Original evidence record. Use the publisher-required identifiers and citation format when reproducing a score.

## Keep memory capacity and utilization in their own columns

A larger memory-capacity figure answers how much memory is specified. It does not measure how quickly an application completes. Likewise, an activity percentage is not a model-serving throughput result. Keep capacity, activity and completed work in separate columns, even when a dashboard places them side by side.

For a shortlist, ask the application owner to establish whether the model and operating settings fit the proposed configuration, then ask for performance evidence under those settings. Do not replace that work with a product's largest memory figure. A system that fits the workload and a system that serves it at the required speed are related decisions with different evidence.

## Check energy and availability without inventing a proxy

A GPU power limit, a server power-supply rating and measured system energy describe different things. MLCommons states that its power measurement covers the entire system and is valid for the accompanying benchmark. If a submission has no corresponding measured power result, leave that comparison unavailable rather than substituting a nameplate value.

An Available category in a benchmark also does not establish stock, cloud quota or capacity for your region and dates. Resolve the actual purchase or compute-access question with the selected provider. Keep the date, location, quantity and exact product in that separate check; a public performance result cannot answer an account-specific availability question.

References: [M3](https://mlcommons.org/benchmarks/inference-datacenter/)

## Compare the billed system cost of the same completed work

A cheaper GPU-hour is not automatically a cheaper completed job. First establish a comparable outcome using the workload, quality, scenario and system boundaries above. MLCommons' training definition measures time to a target quality; its inference scenarios attach conditions to the measured output. Neither supplies the price a buyer will actually pay. Keep the performance evidence and the dated commercial offer as separate inputs, and record where their configurations differ. [M1, M3]

Match the billed unit before doing the arithmetic. An offer per accelerator-hour differs from one per complete server-hour; a multi-accelerator benchmark does not become a single-device measurement because the price is quoted that way. Identify how many billed units are reserved, the billable interval and which parts of the job that interval includes. A benchmark's timed portion may omit setup, waiting or other time that the actual offer bills. Keep minimum charges, commitment conditions and any separately charged storage, data transfer, support or software visible where they apply. An unspecified charge remains unknown rather than silently becoming zero.

Fictional EX-COST24 compares two complete-system offers labelled generation A and generation B. These are invented labels, not named GPU products or published benchmark submissions. Both are stipulated to complete the same job at the same quality; A uses two billable system-hours at 12 invented cost units per system-hour, while B uses one at 20. The resulting compute charges are 24 and 20 cost units. B's higher hourly number therefore produces a lower charge for this stipulated completed job. Nothing in the example establishes the relative runtime or cost of real GPU generations.

That is a limited comparison, not a total ownership or profitability calculation. It assumes the stated whole-system billing and completed-work evidence and excludes all other charges. If those inputs cannot be reconciled, show an incomplete cost comparison rather than projecting a result from peak FLOPS, a different benchmark scenario or a generation's marketing speedup. For inference, compare a common amount of accepted output under the same response-time and quality requirement; output that misses the requirement is not an equivalent completed unit.

References: [M1](https://mlcommons.org/benchmarks/training/), [M3](https://mlcommons.org/benchmarks/inference-datacenter/)

EX-COST24: fictional whole-system charges, with the job and billing boundary held explicit
| Invented complete-system offer | Stipulated billable work | Compute charge and remaining boundary | Source |
| --- | --- | --- | --- |
| Generation A: 12 cost units per complete-system hour | Same completed job/quality; two billable system-hours | 12 × 2 = 24 cost units. No actual GPU performance, real rate or other fee is represented. | Planning analysis |
| Generation B: 20 cost units per complete-system hour | Same completed job/quality; one billable system-hour | 20 × 1 = 20 cost units. The higher hourly figure is lower for this stipulated job, not a forecast for a real generation. | Planning analysis |
| A quote with a different unit or missing billable-time basis | For example, accelerator-hour price against an unreconciled multi-accelerator system result | Comparison incomplete. Resolve the actual billed population and interval; do not divide a benchmark by GPU count and call it a measured single-GPU cost. | Planning analysis |
Wholly fictional cost units, runtimes and offers; original arithmetic, not MLCommons results, a provider quote, current market price, financial advice, acquisition/TCO model or Leviathan compute service.

## Turn a shortlist into a validation request

Send the platform owner a compact request built around the intended job. State the model and revision, representative input/output sizes, required quality, load pattern and success criteria. Attach the two or three relevant public result records rather than a screenshot of a general leaderboard. Ask what changes between those submitted systems and the proposed configuration.

Before accepting a new test, agree who runs it, which software and data may be used, how results will be retained and who decides whether the result answers the requirement. Keep sensitive workloads and customer data out of a public benchmark request. This article does not prescribe benchmark commands, tuning settings or a production stress test.

If no comparable public result exists, record that gap. It may justify a configuration-specific evaluation; it does not justify turning an OEM peak specification into an application-performance promise.

## Hand the selected system to the physical project team

Once the platform decision is made, move the exact configuration into the physical planning package. The installer needs the actual server or rack model, customer-furnished equipment and materials, released rack and cabling plans, access authority, permitted work and the agreed handoff records. A benchmark result does not release a facility location or authorize work on power, cooling or customer networks.

Leviathan can discuss an agreed scope for physical mounting, rack-and-stack installation, structured cabling, labeling and physical handoff documentation. Application benchmarking, workload tuning and production performance acceptance remain with the appointed platform and acceptance owners unless a separately reviewed scope establishes otherwise. Contact us when the selected equipment has a physical installation requirement; include the site and the work you need completed.

References: [L1](https://www.leviathansystems.co/services/rack-and-stack)

## Common planning questions

### Can I choose the fastest AI GPU by peak FLOPS?

Peak FLOPS describes a specified arithmetic capability. Preserve the precision, dense/sparse basis and hardware scope, then compare application evidence for the workload and quality requirement.

### Does dividing a server score by its GPU count give a single-GPU benchmark?

No. It gives a calculated average for that submitted configuration. It is not a separately measured single-GPU result and does not prove linear scaling.

### Can I use an Offline throughput result to promise interactive response time?

No. Compare the appropriate scenario and its response-time requirements. Retain the model, quality target, system configuration and software details.

### Does the lowest price per GPU-hour identify the lowest-cost completed job?

No. Match the actual billed unit, population and interval to comparable completed-work evidence. Include applicable charges and conditions separately; a cheaper hourly figure can require more billable time, and a benchmark timed interval may differ from the bill.

### Can I price a newer GPU generation by applying its peak-FLOPS ratio to an older job?

That ratio does not establish the actual job runtime, accepted output or billing. Use configuration-specific comparable evidence and the actual dated offer. If those inputs are missing, leave the cost conclusion unresolved rather than inventing a generation discount.

## Primary sources

- [M1 — MLCommons training benchmark: metric and result definitions](https://mlcommons.org/benchmarks/training/). Checked 2026-10-15. Training time to a defined quality target. No training score or product ranking reproduced.
- [M2 — MLCommons inference rules: scenarios and latency definitions](https://github.com/mlcommons/inference_policies/blob/ff7edba545fded369e7e7e3d5a2f0bab4a95eece/inference_rules.adoc). Checked 2026-10-15. Pinned primary rules snapshot last changed August 20, 2026. General scenario, TTFT and TPOT definitions only; no numeric threshold is generalized across models or versions.
- [M3 — MLCommons datacenter inference: system, division, availability and measured-power context](https://mlcommons.org/benchmarks/inference-datacenter/). Checked 2026-10-15. Current page retrieved with v6.1 results context. General interpretation only, no numerical result or implied endorsement.
- [M4 — MLCommons result messaging guidance](https://github.com/mlcommons/policies/blob/a4520be877271064ceff8339b837edb67027537b/MLPerf_Results_Messaging_Guidelines.adoc). Checked 2026-10-15. Primary publication guidance for score identification, compatible comparisons and disclosures. No benchmark score is reproduced in this guide; trademark/reference wording still requires final editorial rights check.
- [N1 — NVIDIA HGX platform: B300 and B200 specifications and footnotes](https://www.nvidia.com/en-us/data-center/hgx/). Checked 2026-10-15. Eight-GPU HGX FP4 platform specification only: B300 sparse/dense 144/108 PFLOPS; B200 sparse/dense 144/72 PFLOPS. Derived dense ratio 1.5 is arithmetic on OEM specifications, not measured application speed. The page also includes projected Rubin comparisons; those projections are not used as rankings.
- [L1 — Leviathan rack-and-stack service scope](https://www.leviathansystems.co/services/rack-and-stack). Checked 2026-10-15. Physical installation scope only. No benchmark operation, tuning, cloud hosting, commissioning, electrical or cooling service representation.

## Related Leviathan resources

- [Data Center GPUs by Memory Capacity: Compare the Exact Variant](/articles/data-center-gpu-memory-capacity-comparison)
- [GPU utilization and facility occupancy: different denominators](/articles/gpu-utilization-vs-facility-occupancy)
- [GPU cloud availability: quotas, reservations and capacity windows](/articles/gpu-cloud-capacity-availability)
- [Compare the exact Dell, Supermicro or HPE GPU server](/articles/dell-vs-supermicro-vs-hpe-gpu-servers)
- [Rack-and-stack installation scope](/services/rack-and-stack)

## Selected equipment that needs physical installation?

Share the actual configuration, facility, released plans and authorized mounting or cabling scope. Our team can discuss requirements and availability.

[Discuss the scope](/contact?service=rack-and-stack&from=/articles/which-gpu-is-fastest-workload-benchmarks)