PCIe AER Reports in a GPU Server: Read the Error Before Replacing the Card

Follow the report through the PCIe hierarchy

Advanced Error Reporting, or AER, concerns PCIe hierarchy, link and transaction errors. Device-specific errors have separate handling. In the Linux 6.18 AER model, a reporting device sends an error message to the Root Port above it. The Root Port records status and the reporting agent's Requester ID; enabled reporting can trigger an interrupt for handling.

This is a chain of detection, reporting and handling. The place where information becomes visible is not automatically the place where a replaceable part has failed. In a GPU server, start with the actual internal attachment relationship rather than treating every PCIe observation as an observation of the GPU itself. The existing GPU-to-NIC topology guide explains that relationship in more detail.

This article interprets supplied evidence using the version-pinned Linux 6.18 reporting and recovery model. That reference does not establish the software installed on a particular server. The discussion supplies no operating procedure or authorization for hardware work.

References: [K1][K4]

Compare corrected, nonfatal and fatal evidence

The severity categories distinguish interface impact. They do not assign a repair disposition. Consider three fictional report summaries, C, N and F. Each stipulates only the stated category for a separate event; they are not a sequence on one server. No real logs, measurements or test outcomes are represented.

References: [K1]

Locate the reporter before judging the part

Requester ID identifies the error-message sender. It answers who reported the event, not which physical item must be replaced.

Linux 6.18's AER statistics definition makes the distinction concrete: a problem associated with an endpoint can be seen and counted by its link partner while the endpoint's counters remain zero. The observed counter location can therefore differ from the component suspected of causing the problem.

For a replacement offer, ask the supplier to connect the supplied reporting identity to the exact server hierarchy and explain the evidence implicating the proposed part. Naming the GPU because the interrupted workload used a GPU skips that causal step. Naming the Root Port because it reported the event skips the same step in another direction. A topology drawing can locate the relationship; it cannot supply the missing diagnosis.

References: [K3][K4]

Interpret silence and message counts at their actual scope

The Linux AER driver depends on firmware granting the operating system control; console filtering also affects visible output. A quiet supplied excerpt therefore needs a reporting context before it can support a claim about absence of errors.

The Linux v6.18 AER implementation throttles correctable and nonfatal uncorrectable messages separately for each device. Fatal messages are not throttled by that mechanism. It also updates device statistics before the decision to print detailed output. A printed-message count is consequently not interchangeable with the corresponding device statistics.

The statistics definitions distinguish errors seen or reported by a device from messages reported to a Root Port. They also warn that one message can report multiple errors, so a message total need not equal the sum of individual error entries. Adding those quantities would erase their different meanings.

This matters when comparing supplier evidence. A smaller excerpt does not establish a healthier server, and dividing its line count by elapsed time does not repair incomplete reporting coverage. A meaningful comparison needs the actual counted quantity, observation interval, continuous counting history and reporting scope. Even a valid count rate does not become a universal health score or acceptance limit.

Likewise, the category in row C supplies no general assurance that recurring reports can be ignored. Their recurrence needs interpretation against the actual platform and agreed requirement. Neither silence nor repetition alone decides the equipment's disposition.

References: [K1][K3][K4]

Keep PCIe AER, GPU HBM ECC and Ethernet FEC separate

AER is not an umbrella count of everything that can go wrong in a GPU server. NVIDIA's GPU Memory Error Management 595 overview explicitly scopes its main ECC discussion to uncorrectable high bandwidth memory, or HBM, errors. That is a different observation population from the PCIe hierarchy reports considered here.

Cisco's Ethernet optics explanation describes forward error correction, or FEC, encoding data with redundancy and decoding it at the receiver. Those external Ethernet correction observations are different again. The common word corrected does not give an AER report, an HBM event and a FEC counter the same unit or meaning.

Keep these findings alongside one another when they concern the same system, with their own definitions intact. Adding them into a single GPU error total would conceal the interface and mechanism each observation concerns. The linked ECC and Ethernet FEC articles develop their separate interpretation questions.

References: [K1][N1][C1]

Separate platform recovery from recovery of the job

Linux 6.18's PCI error-recovery model coordinates the platform and affected device drivers. Their recovery responses can affect the next platform action; recovery is not a decision made by one severity word alone. The model's final resume notification tells participating drivers to restart normal activity after the required recovery stages succeed.

That notification concerns device operation. It supplies no saved application state and no proof that an interrupted GPU job resumed correctly. A supplier statement that recovery completed should identify whether it means the platform sequence completed, device I/O resumed, or the application continued from an identified state. Those are different claims requiring evidence at their respective layers.

The recovery document also explains that a platform's permanent-failure outcome does not mean every underlying cause is permanent physical damage. A software recovery outcome is therefore insufficient as a component-level replacement diagnosis.

Return to the fictional rows: none includes a recovery result. Row N cannot be used to assert successful transaction completion merely because the link remains functional. Row F cannot be used to assert either permanent GPU damage or successful job restoration. Application continuation remains a separate unanswered question in all three rows.

References: [K2]

Make the offer answer the unresolved hardware question

A supplier offering a replacement card should explain what evidence connects that card to the reported problem and why the proposed change addresses the cause. A severity-only excerpt leaves that explanation missing. It can support a request for investigation without selecting the repair in advance.

For a support offer, establish which technical assessment it will provide and which equipment and software context it covers. For a hardware-QA offer, establish the hardware question its observations are intended to answer. A report describing the checks performed is useful evidence within that scope; the buyer should not silently expand it into a promise of fault isolation, application recovery or future reliability.

Leviathan's Hardware Burn-In & Validation service is a relevant route when the open question calls for agreed hardware QA before integration, deployment or reuse. Its scope depends on the identified configuration, approved plan, confirmed resources and acceptance parameters. Production software recovery and OEM repair decisions remain with their appointed providers.

Use the service and project-inquiry links below with a nonconfidential equipment summary, the existing observation and the hardware decision still unresolved. That lets the discussion address a concrete evidence gap before anyone treats a replacement label as the answer.

References: [L1]