RDMA Write Completion Is Not an Application Durability Receipt

Understand what the one-sided transfer does

RDMA Write moves a payload from local memory into a permitted remote memory region. In the one-sided operation, the remote processor need not execute an application receive-and-copy step for each transfer. That mechanism explains why data movement can finish without the destination application having completed its own work.

The historical NVIDIA-hosted v1.7 manual distinguishes ordinary Write, which provides no remote-host notification for the operation, from Write With Immediate, which notifies the remote host of an immediate value. This is a notification distinction, not a definition of application processing. The manual supplies historical operation semantics, not current hardware-support evidence.

References: [R1][N1]

A completion record reports a work request's outcome

The upstream rdma-core manual separates the number of returned work-completion records from each record's operation status. It identifies the completed work request and, where valid, its operation type. A successful retrieval of three records therefore does not mean that three operations succeeded.

The manual also limits valid fields when status is not IBV_WC_SUCCESS: only the work-request identifier, status, local queue-pair number and vendor error remain valid. An error record cannot be read as though every success-path field were populated meaningfully.

For a supplier's report, distinguish the identified request ending from that request ending successfully. Then ask what the applicable successful result actually covers. Neither a record count nor the word completion identifies an application job or its recovery state.

References: [V1]

RC-233: three successful 8 MiB writes, one logical delivery

Consider wholly fictional delivery RC-233 over a supported reliable-connected, or RC, transport. It consists of three distinct 8 MiB chunks belonging to one logical application delivery. Stipulate three individually signaled ordinary Write operations, each reported successful. Also stipulate that all three chunks have been placed in nonoverlapping portions of a supported remote volatile-DRAM buffer. No application-level acknowledgement has arrived yet.

The stipulated placed payload is 3 × 8 = 24 MiB. With 1 MiB equal to 1,048,576 bytes, that is 25,165,824 payload bytes. Placement is an explicit premise here; it is not inferred for every transport or device from a count of records.

There are three successful work requests and one logical delivery. The chunks are different portions of that delivery, not three copies of it. Neither the number of operations nor their total byte quantity establishes three application jobs or three durable copies.

At this point, application processing remains unknown to the sender. The application might have started or even finished its work. Absence of its acknowledgement establishes neither completion nor failure of that work.

A notification and an application result answer different questions

In a separately supported version of the fictional case using Write With Immediate, the added remote notification would not, by itself, establish that the application had examined the complete delivery or passed its processing rule. A notice that work is available and a result that application work is complete have different meanings.

Now add a separate stipulated observation to RC-233: an application acknowledgement identifies the complete 24 MiB logical delivery and confirms that it has passed the application's declared validation rule. This is evidence about one application-level result, beyond the three transport operations. It does not turn those operations into three completed AI jobs, or extend validation into an unstated downstream processing result.

The useful comparison is between what the sender knew before this acknowledgement and what its stated meaning adds. The acknowledgement needs the logical delivery identity and the result it represents; an unexplained received label leaves that distinction unresolved.

References: [N1]

A processing acknowledgement needs a separate retention promise

RC-233 expressly uses volatile DRAM. Its later application acknowledgement confirms the stipulated validation result; no retention or recovery behavior has been stipulated. Calling that acknowledgement durable would add a promise absent from the example.

A durability claim needs to identify the retained object, the acknowledged point, the failure conditions covered and the result promised after those failures. Continued availability while a process runs, survival of a process restart and survival of loss of power to the receiving host are different requirements. A supplier should explain which one its acknowledgement covers and what evidence connects that acknowledgement to the promised retention.

Where a product supplies a higher-level durable service or documented persistent-memory behavior, assess that actual service and supported implementation. The error is assigning its additional promise to an ordinary transport result without evidence.

Make the supplier explain the completion boundary

Do not assume identical completion semantics across RDMA transports. The October 2007 IETF RDMAP specification, RFC 5040, explicitly permits local Write completion before the message has reached the remote application's buffer or been examined by it. That RDMAP rule is not a replacement definition for the RC example. It illustrates why the exact transport and implementation matter.

Ask the supplier which operation, transport, device, memory type and software release its result concerns. Have it distinguish the transport result, any receiver notification, the application's declared processing result and the retention promise, if one is offered. Identify the logical object to which each observation belongs. For RC-233, that prevents three chunk records from becoming three delivery receipts.

The linked NVMe atomicity article addresses the separate storage-command layer. The checkpoint article addresses which application state is captured. Those questions become relevant if the proposed service adds storage or training recovery; neither can be answered by relabeling an RDMA completion.

References: [R1]

Bring physical connection requirements to the cabling inquiry

If the selected system needs physical connections installed or documented, Leviathan's structured cabling service covers agreed fiber and copper installation, labels, connection records and physical checks. Use the linked service and project inquiry with a nonconfidential equipment summary, approved endpoints and routes, specified assemblies, appointed materials supplier and required handoff records.

That physical package has its own useful completion evidence. RDMA configuration, software integration and application durability remain with the separately appointed technical providers. This article explains how to interpret transport and application claims; it supplies no field procedure or authorization to perform work.

References: [L1]