---
title: "RDMA Message Reception Needs More Than Spare Buffer Bytes"
description: "Leviathan Systems — Technical content Published 2026-10-21; updated 2026-10-21."
canonical: "https://www.leviathansystems.co/articles/rdma-send-posted-receive-resource-availability"
last-updated: "2026-10-21"
---

# RDMA Message Reception Needs More Than Spare Buffer Bytes

Leviathan Systems — Technical content
Published 2026-10-21; updated 2026-10-21.

RDMA Send and Write With Immediate require available posted receive requests; ordinary RDMA Write does not. A consumed receive request contributes zero availability even when its buffers retain unused bytes.

## A posted receive request describes a reception

An RDMA receive request, or RR, is a work request made available on a receive queue. The rdma-core v60.0 interface describes its storage through a scatter/gather list containing buffer addresses, lengths and local memory-region keys. One request can describe several storage segments; segment count and request count are different quantities.

For Send, the receiver supplies destination storage in advance. A successfully received message consumes one request. The remaining capacity within its buffers does not become another available request.

The distinction matters when a supplier reports receive capacity in bytes. That quantity describes storage. To explain how many messages the receiver can accept under a stated scenario, the explanation also needs the population of available receive requests.

References: [RR1](https://networking-docs.nvidia.com/doca/archive/3-5-0/rdma-aware-networks-programming-guide), [RR2](https://raw.githubusercontent.com/linux-rdma/rdma-core/v60.0/libibverbs/man/ibv_post_recv.3)

## RR-70: three requests for four messages

Consider wholly fictional case RR-70. Stipulate a supported reliable-connected, or RC, queue pair using its own private receive queue, with no shared receive queue. Initially, three receive requests are posted. Each describes a distinct, nonoverlapping 4 KiB buffer, with appropriate registration and access. Here 1 KiB equals 1,024 bytes.

Four ordinary Send messages, each carrying 1 KiB, are considered in sequence. Stipulate that the first three are successfully received, each consuming one of the initial requests, before considering the fourth. No additional receive requests are posted and no other traffic consumes the queue's requests during this interval. Other resources are assumed sufficient for those first three receptions.

These premises isolate a finite resource question. They specify neither a real adapter nor a measured workload, and they make no assumption about what the receiving application subsequently does with the data.

## Count available requests separately from unused bytes

Initially, RR-70 has 3 x 4 KiB = 12 KiB of storage described by its three available requests. After the first successful message, two requests remain available. The consumed request's buffer contains 1 KiB of payload and has 3 KiB of unused capacity.

After the second successful message, one request remains available. After the third, none remain. The three messages have delivered 3 x 1 KiB = 3 KiB, or 3,072 bytes. Across the three consumed requests' buffers, 12 KiB - 3 KiB = 9 KiB, or 9,216 bytes, were not filled by these payloads.

Those 9 KiB belong to buffers whose requests have already been consumed. They are not storage described by a still-available request. The fourth 1 KiB message therefore cannot be justified by comparing its size with the aggregate 9 KiB of unused capacity.

The precise state is zero available requests and 9 KiB unused within the consumed requests' buffers. It is not an assertion that the host has run out of RAM, or that the unused bytes have become free for arbitrary reuse. The example counts payload occupancy and receive-request availability separately.

## Larger buffers change one quantity in the comparison

Now change only the fictional buffer capacity from 4 KiB to 8 KiB per request. Keep three requests, three successful 1 KiB messages and the same no-refill premise. Initial described storage becomes 24 KiB. After the third message, 21 KiB is unused within the consumed buffers, and zero requests remain available.

A proposal promising twice as many buffer bytes has therefore left this particular fourth-message question unchanged. That is a consequence of the stipulated request count, rather than evidence that larger buffers are generally useless. Buffer capacity answers whether the storage is large enough for a message; available requests answer the separate reception-resource question.

Neither version establishes a sustainable message rate. The case supplies no time interval, replenishment behavior or application-consumption rate from which such a rate could be calculated.

## Write With Immediate has an additional receive requirement

An RDMA Write request identifies its remote memory destination: the v60.0 post-send interface includes a remote address and remote memory-region key. Ordinary Write uses no RR. Write With Immediate still targets the addressed memory but also consumes an RR for receiver notification; its payload does not use that RR's Send buffers.

Thus RR-70's 4 KiB receive-buffer size is not a Write destination limit. An explanation of an addressed Write needs the applicable remote region, complete range and permissions. The linked memory-region article develops that access question.

Changing the operation changes which evidence answers the claim. A description that establishes destination access for ordinary Write leaves the additional receive resource for Write With Immediate unresolved.

References: [RR1](https://networking-docs.nvidia.com/doca/archive/3-5-0/rdma-aware-networks-programming-guide), [RR3](https://raw.githubusercontent.com/linux-rdma/rdma-core/v60.0/libibverbs/man/ibv_post_send.3)

## RNR identifies a resource condition at reception

NVIDIA's archived DOCA 3.5.0 guide describes receiver-not-ready, or RNR, on an RC connection when a receive request is absent. If RR-70's fourth Send arrives while no request is available, it encounters that condition.

The case supplies no later observation. It therefore establishes no final completion status, permanent message loss, retry duration or eventual success. Likewise, the stipulated reception of the first three messages says nothing about their application processing or retention. Use the linked completion-and-durability article for that separate decision.

A physical-link record cannot supply the missing request count. Conversely, this hypothetical receive-resource shortage supplies no evidence that a cable is defective or that a physical link lacks capacity. Its conclusion concerns the stipulated queue and interval.

References: [RR1](https://networking-docs.nvidia.com/doca/archive/3-5-0/rdma-aware-networks-programming-guide)

## Ask which receive population the supplier means

An assessable supplier explanation identifies the operation, the relevant receive queue, its available request population at the stated point and what happens to that population during the interval being discussed. It also distinguishes per-request storage from total host memory. In RR-70, those facts make the fourth-message limitation visible without a performance test.

Private and shared queues need different accounting boundaries. The v60.0 receive manual states that a queue pair associated with a shared receive queue uses that shared queue instead of its own receive queue. RR-70 expressly excludes this arrangement. Its initial count cannot be carried into a shared population without identifying that population and its other consumers.

The same manual makes buffer reuse conditional on request execution and retrieval of the corresponding work completion. Unused space alone is therefore insufficient evidence for a reuse claim. These interface descriptions support interpretation of the supplied explanation; the offered hardware, software version and actual behavior still need their own evidence.

References: [RR2](https://raw.githubusercontent.com/linux-rdma/rdma-core/v60.0/libibverbs/man/ibv_post_recv.3)

## Use physical connection records for the cabling package

If the deployment separately requires physical cabling, Leviathan's Structured Cabling & Fiber Installation service covers agreed fiber and copper installation, endpoint labels, connection records and physical checks. Use the linked service and project inquiry with approved endpoints, routes, specified assemblies, materials responsibilities and required handoff records.

Those records identify the installed connections for the appointed system and network providers. RDMA software configuration and receive-resource behavior belong to their scope. This article explains a resource comparison; actual physical work requires its own agreed scope.

References: [L1](https://www.leviathansystems.co/services/structured-cabling)

## Primary sources

- [RR1 — NVIDIA DOCA archive 3.5.0: RDMA-aware Networks Programming Guide](https://networking-docs.nvidia.com/doca/archive/3-5-0/rdma-aware-networks-programming-guide). Checked 2026-10-20. RC RNR and Send/Write receive-resource semantics; archived documentation does not establish support for an offered device.
- [RR2 — rdma-core v60.0: ibv_post_recv manual](https://raw.githubusercontent.com/linux-rdma/rdma-core/v60.0/libibverbs/man/ibv_post_recv.3). Checked 2026-10-20. Receive-request storage fields, private versus shared receive-queue applicability and the execution/completion requirement for buffer reuse. The tag identifies the interface reference, not a hardware qualification.
- [RR3 — rdma-core v60.0: ibv_post_send manual](https://raw.githubusercontent.com/linux-rdma/rdma-core/v60.0/libibverbs/man/ibv_post_send.3). Checked 2026-10-20. Addressed RDMA destination fields, including remote address and remote memory-region key; these fields do not report a completed operation.
- [L1 — Leviathan Systems: Structured Cabling & Fiber Installation](https://www.leviathansystems.co/services/structured-cabling). Checked 2026-10-20. Published physical installation, labeling, connection-record and physical-check scope, with appointed materials responsibilities and separately owned network configuration.

## Related Leviathan resources

- [RDMA memory-region keys, complete ranges and operation permissions](/articles/rdma-memory-region-keys-range-and-access)
- [RDMA Write completion and application durability](/articles/rdma-write-completion-versus-application-durability)
- [NIC receive queues, RSS and physical port capacity](/articles/nic-rss-receive-queues-versus-port-capacity)
- [Structured Cabling & Fiber Installation](/services/structured-cabling)
- [Discuss a separately scoped physical cabling project](/contact?service=structured-cabling&from=%2Farticles%2Frdma-send-posted-receive-resource-availability)

## Define the physical connections and their handoff

If physical cabling is required, share approved endpoints, routes, specified assemblies, materials responsibilities and required connection records to discuss scope and availability.

[Discuss the scope](/contact?service=structured-cabling&from=/articles/rdma-send-posted-receive-resource-availability)