GPU Core Dumps: Which Failure State Did the Artifact Capture?

Start with the state question

A supplier may return a diagnostic package described as complete. Before relying on that description, identify what the reader needs to establish: where an exception was reported, which data values were retained, what another GPU context was doing, or whether an application can continue. Those questions require different evidence.

Completeness is meaningful only against that requirement. A package can answer a narrow question well while leaving a broader one unresolved. Compare the claimed conclusion with the actual contents and provenance, rather than treating the package's name, file count or byte size as a coverage statement.

Read the documented coverage boundaries

NVIDIA's CUDA-GDB 13.4 manual, section 4.5, distinguishes exception logging without a core file from core dumps. It documents possible omission of global/constant, shared and local GPU memory, and capture restricted to contexts with exceptions. Its inspection description concerns recorded device state, or host and device state when both kinds of core are supplied.

These distinctions make provenance consequential. The reader needs an account of the supplied artifact's coverage, not an assumption that every documented category must be present. The manual describes capabilities and limits; it does not verify the contents of a particular supplier's package.

References: [N1]

FLEET-DUMP-02 contains three files and one GPU core

Fictional case FLEET-DUMP-02 concerns an application with GPU contexts C1 and C2. C1 encounters an exception. No outcome for C2 is stipulated.

Package A contains exactly three supplied files: one exception log, one GPU core file and one CPU core file. Its supplied provenance expressly states that the GPU core covers C1 only and omits GPU global/constant memory segments. C1 exception program-counter, or PC, and stack information is separately stipulated present. The CPU core contains the supplied host-state capture; no additional contents are established.

The inventory therefore establishes one GPU core, not three GPU captures. The log and CPU core have their own stated roles. Their presence does not expand the GPU core's declared context or memory coverage.

This is an invented artifact comparison. No real exception, capture, file inspection or application continuation occurred.

Carry each omission into the conclusion

For FLEET-DUMP-02, the supplied C1 PC and stack information can support a discussion of that reported exception location. They do not supply the values in the omitted global/constant segments. A conclusion requiring those values remains unsupported by the stipulated package.

C2 device state is also unestablished. C1-only provenance does not show that C2 was healthy, failed, continued or stopped. The example supplies no C2 outcome, so none should appear in its conclusion. Likewise, it makes no further claim about shared or local memory contents; silence about a category does not establish its inclusion.

The CPU core adds the stipulated host-state capture. Nothing in the case establishes corresponding copied or mapped versions of the omitted GPU data within that capture. The existence of a CPU core cannot supply that missing content claim automatically.

These are limits on what this package establishes. Missing state is not evidence that the state was healthy, corrupt, irrelevant or responsible for the exception.

Separate image association from fault-time instruction evidence

NVIDIA states that GPU core dumps omit the device instruction memory backing executing functions. Disassembly uses the original cubin image rather than instruction bytes captured from device memory at fault time.

Identifying the corresponding original binary build helps connect the reported location with the program image used to interpret it. A familiar filename alone does not establish that build relationship. Image identity supplies association; it establishes neither the cause of an exception nor corruption of instructions in device memory. A disassembly view must retain its stated evidence basis when someone uses it to support a conclusion.

FLEET-DUMP-02 supplies no independent fault-time instruction-byte evidence. Referring to the corresponding application image would not fill that gap or add another captured state to Package A. A report should distinguish the image used to interpret a location from any claim about the bytes actually present during the event.

References: [N1]

A missing core and a readable core answer different questions

The manual documents generation failures, including a kernel exiting before an exception is recognized. It also says core inspection requires no installed GPU.

A missing core file therefore cannot, by itself, prove that no exception occurred. Nor does the existence of a file settle its coverage. These are separate questions about whether an artifact exists and what evidence it contains.

If a supplied core is successfully inspected, that result concerns the artifact. It does not establish that omitted state became available, that the affected GPU is usable or that a workload resumed. Inspection performed without a GPU is particularly clear evidence of this distinction: reading the record is not an observation of the affected device's subsequent operation.

References: [N1]

Compare the proposed return with the question it must answer

If the requested return in FLEET-DUMP-02 is the reported C1 exception location and stack account, the stipulated information addresses that narrow request. If the question instead requires values from the omitted global memory or C2 device state, Package A leaves the needed evidence unestablished. Calling either conclusion complete without naming the question would conceal the difference.

Ask the supplier to connect its proposed return to the required state, the contexts included, the omitted categories and the corresponding binary build. A larger archive or a higher file count cannot establish inclusion of a particular missing region. Compare whether the return answers the question before comparing its size. The useful disposition can be precise: the reported C1 location is supplied; the omitted values and C2 state remain unanswered. That is an evidence assessment, not a fault diagnosis.

Keep continuation and support questions with their owners

FLEET-DUMP-02 establishes neither a resumable application checkpoint nor successful continuation. The appropriate conclusion is that these results remain unproven by the supplied material. It does not follow that every core format or recovery arrangement makes restart impossible.

The related Training Checkpoint or Storage Snapshot guide owns the question of preserved application continuation state. The GPU ECC, Error Containment and Job Recovery guide distinguishes hardware error handling from the affected job's recovery. Recovery Point and Recovery Time explains how an observed application return is compared with its objectives.

The GPU Rack Support Coverage guide addresses the responsible receiving and supporting parties. A package's evidence gaps should reach the appointed application, platform or product-support owner. Possession of diagnostic files does not establish that a physical installer holds the software-support responsibility or that any provider has established the fault's cause.

Select physical hardware work only when it is a separate requirement

A reader evaluating software diagnostic evidence may need no physical service. The unresolved package questions can remain with the appointed software and platform providers.

If the wider project separately calls for physical hardware QA before integration or reuse, Leviathan's Hardware Burn-In & Validation service and project inquiry provide a relevant next step. Identify the equipment, requested hardware QA result and receiving team so the configuration, plan, resources and acceptance criteria can be discussed.

That is a separately selected physical package. The service's hardware observations and exceptions have their own scope; they do not supply missing GPU payload, establish an exception's cause or demonstrate application recovery.

References: [L1]