A Connected RDMA Port Does Not Grant Access to Every Byte

Registration defines the memory scope

In rdma-core v60.0, ordinary explicit memory registration associates a memory region, or MR, with a protection domain and specifies its starting address, length and access attributes. The relevant object is that region, rather than the server's entire installed memory.

For a buyer, this turns a broad direct-access claim into a relationship that can be examined. Which region does the supplier mean, and which bytes within it does the proposed operation concern? A diagram showing the server connected to a fabric leaves both answers open.

Address convention matters too: the manual also describes zero-based and alternative-base variants. The worked example below uses an ordinary address-based region. It identifies no CPU physical backing and supplies no universal convention for other registration variants.

References: [MR1]

Separate the local buffer from the remote region

The tagged post-send manual places a local buffer's address, length and lkey in its scatter/gather description. The RDMA remote fields instead contain remote_addr and rkey. These pairings connect each buffer address to its own memory-registration relationship on the appropriate side of the work request. Local and remote are relative to the requester; a generic field called memory key obscures which relationship has been described.

For example, a supplier identifying only the requester's local buffer and its lkey has left the remote-region relationship unexplained. It has not supplied an interchangeable substitute for the remote address and rkey. A remote key also needs its applicable region and resource context; a bare number is an incomplete explanation.

This local-buffer discussion concerns ordinary non-inline requests. The manual expressly says the inline-send case does not check lkey. That exception is another reason to identify the operation and buffer handling before treating a field list as universal.

References: [MR2]

Permission belongs to the named operation

The registration manual distinguishes remote-read, remote-write and remote-atomic access, with atomic access conditional on support. Read permission therefore does not establish write or atomic permission.

For a proposed remote read, the useful answer is that the identified region permits that operation in the applicable implementation. Calling the server RDMA enabled leaves this unanswered. Likewise, naming the correct remote-key role does not explain which operations the region permits.

Region permission is also distinct from transport capability. The post-send manual identifies different supported operations for different queue-pair transport types. A region's read permission does not make every transport support an RDMA Read. Nor do these memory-access fields establish cryptographic peer identity or encryption.

References: [MR1][MR2]

MR-66: compare the complete 64-byte range

Consider wholly fictional region MR-66. Stipulate an ordinary explicit, address-based region starting at 0x100000 and spanning 16 KiB. Here 1 KiB is 1,024 bytes, so its length is 16,384 bytes, or hexadecimal 0x4000. Its exclusive end is 0x104000; its last included byte is 0x103FFF. Also stipulate the correct remote key in its applicable context and permission for remote reads. Transport and hardware are otherwise unspecified.

A proposed 64-byte read starting at 0x103FC0 has exclusive end 0x103FC0 + 0x40 = 0x104000. It occupies bytes 0x103FC0 through 0x103FFF. Ending exactly at the region's exclusive end fits: the request includes the preceding byte, not the byte at the end address.

A second proposed 64-byte read starts at 0x103FE0. Its exclusive end is 0x103FE0 + 0x40 = 0x104020. Only 0x104000 - 0x103FE0 = 0x20, or 32 bytes, remain inside the region. The other 32 bytes extend beyond it. The starting address fits, but the complete requested range does not.

The region, read permission, key assumption and physical connection are unchanged between these comparisons. The different result comes from the requested range. This is arithmetic applied to a supplied claim, not an attempted access or an observed device result.

A range that fits still needs the right permission

The first MR-66 range satisfies the illustrated range comparison. Its stipulated read permission supplies a different part of the explanation. These are two distinct premises, neither of which reports an actual transfer outcome.

Now consider a buyer changing the requirement from reading those bytes to writing them. The address arithmetic for the same 64-byte range is unchanged. The supplied permission premise, however, covers reads. The supplier must explain the proposed write permission separately before the earlier assessment can answer the changed requirement.

That gives the buyer two useful distinctions. The second read claim contains a range contradiction under the stated assumptions. The changed write requirement instead lacks a supplied permission basis. Both require a supplier response, but they are different gaps; neither is resolved by a connected-port photograph.

Compare the access claim before interpreting its result

An assessable supplier explanation connects the named region and address convention to the requested starting address and byte count, identifies the local and remote key roles, and states the permission for the operation being discussed. For MR-66, that explanation exposes the 32-byte overrun without relying on a performance measurement. It also shows exactly what is missing when the requested operation changes.

Relate that explanation to the actual offered device, transport and software implementation. The v60.0 manuals describe a software interface; they do not establish hardware support for a supplier's system.

Use the linked GPU-to-NIC topology article for the internal attachment and platform-support question. Use the linked RDMA completion article for what a reported transfer outcome establishes. Those questions follow their own evidence: an attachment drawing supplies no missing region permission, and an access description supplies no completion or application-durability receipt.

References: [MR1][MR2]

Carry the physical connections into the cabling scope

If the selected deployment requires external connections installed or documented, Leviathan's Structured Cabling & Fiber Installation service covers agreed fiber and copper installation, endpoint labels, physical checks and connection records. Use the linked service and project inquiry with a nonconfidential equipment summary, approved endpoints and routes, specified assemblies, appointed materials supplier and required physical handoff.

The appointed system and software providers retain memory registration, key handling and RDMA behavior. Physical connection records give those providers identifiable endpoints; they do not establish memory-access permission. Use this article to interpret a supplier's explanation, with any actual work governed by its separately authorized scope.

References: [L1]