Cabling
GPU Storage Network Handoff: Trace the Physical Connections
A storage network physical handoff should identify the installed connection from the server’s storage-facing interface to its approved destination. Keep adapter identity, physical port, cable/channel references and network-role evidence together. A completed cable path does not establish storage protocol configuration, data access or workload performance.
Locate the storage boundary in the actual design
Obtain the server and network design references that identify storage-facing connections. NVIDIA’s DGX GB rack-scale documentation names storagenet separately from its compute and management networks. That is a specific implementation example, not a universal network blueprint for every GPU server.
Record the exact model and approved role assignment. Some projects may share physical infrastructure or use different adapter arrangements; the network/storage owner must define the intended design. This guide does not choose a storage protocol, prescribe RDMA settings or specify a performance topology.
References: [N5]
Follow the connection to its real destination
Start with the asset, adapter and physically locatable port. Trace the identified assembly or channel through any documented patch positions to the approved remote physical endpoint. If the destination is a network switch, say so; do not label the physical cable as though it directly terminates at a storage controller.
Keep the role reference supplied by the network/storage owner separate from physical observations. Operating-system interface names and adapter mappings can be model/configuration-specific. The field team should not derive them from connector position alone or change the plan to make a familiar naming convention fit.
Build a physical storage connection register
Use one row per connection or defined branch and retain the complete channel relationship where components are shared. The hypothetical fields below avoid private hostnames, addresses and storage volume names. Store sensitive operational detail only in the customer’s authorized records.
Scroll the table horizontally on smaller screens.
| Field | What to record | Acceptance boundary | Source |
|---|---|---|---|
| Source | Asset, approved adapter reference and physical port | Identifies the installed endpoint | Planning analysis |
| Role basis | Released network/storage design and revision | Design owner confirms intended role | Planning analysis |
| Channel | Assembly IDs; panel/trunk positions where used | Physical component traceability | Planning analysis |
| Destination | Actual remote chassis and physical port | Does not infer downstream storage access | Planning analysis |
| Physical evidence | Observed labels, deviations and agreed verification reference | States exactly what was inspected or measured | Planning analysis |
| System evidence | Separate configuration/service acceptance reference or pending owner | Storage/network owner assesses service readiness | Planning analysis |
Original illustrative planning record. Values and identifiers are examples, not a customer record, engineering design or acceptance standard.
Define what one pre-install matrix row represents
A usable matrix starts with the connection set it is supposed to describe. Ask the network/storage owner for that set and its governing issue. State whether each row represents one physical connection or a defined branch, then give it a stable connection identifier. Qualify a local port name with its server asset and adapter reference: port P1 on one adapter is not a complete key for every server in the room. Keep the expected source, cable/channel and remote endpoint beside the owner-supplied role reference.
A role view and a connection list can have different row counts. If two views refer to the same physical connection under a shared design, link them through the common connection ID rather than counting two cable requirements. If they describe different connections, keep their identities distinct even when the role wording is similar. The owner’s actual design must establish that relationship; this article neither merges nor splits a connection on the buyer’s behalf. An unqualified spreadsheet total cannot tell the buyer which case applies.
NVIDIA’s cited DGX GB networking page distinguishes external connectivity, provisioning/management, out-of-band access, compute communication and storage access. This is a model-specific naming example, not a required network set or port map for another installation. Keep the customer’s exact role terms and their defining record issue. In particular, customer data is not automatically compute fabric, and a role name is not a physical endpoint key. The following example uses an owner-supplied four-connection set rather than inventing a connection count from those role names.
References: [N5]
Work a pre-install server-to-storage matrix against the received inputs
Before using the matrix as an installation input, establish the expected connection list and its governing issue. Compare received endpoint attachments against that list by stable connection ID, not by spreadsheet row number or the total number of cables. Keep planned endpoints separate from later observations. A complete-looking received table can still contain a different destination or omit the evidence that assigns the role.
Fictional example EX-DOM19 concerns four expected connections for asset EX-S01. The customer-supplied connection package CP-R4 identifies two storage-facing links, one management link and one customer-data link. Domain record DR-R2 defines those local names. These invented assets, adapters and ports do not represent an OEM port layout or an approved topology. The two storage rows illustrate separate connection identities only; they make no redundancy, path-independence or availability claim.
The received package contains four rows. EX-ST01, EX-MG01 and EX-DT01 agree with their expected endpoint references. EX-ST02 instead carries a destination attachment naming EX-MGSW1/P12, although its expected destination is EX-STSW2/P11. EX-MG01 also names EX-MGSW1/P12. The table therefore has four expected rows, four received rows, three documented endpoint agreements and one unresolved destination discrepancy. It has no installed or tested connection results. The matching total cannot resolve the conflicting reference.
Scroll the table horizontally on smaller screens.
| Connection / owner-declared role | Planned server endpoint and cable | Expected remote endpoint | Received remote reference | Documentary result / responsible return | Source |
|---|---|---|---|---|---|
| EX-ST01 / storage | EX-S01; adapter EX-AD-ST1; port EX-SP1; cable EX-C101 | EX-STSW1 / P11, CP-R4 | EX-STSW1 / P11, attachment EA-ST01-R4 | Endpoint references agree; storage/network owner retains the role basis DR-R2. No physical observation supplied. | Planning analysis |
| EX-ST02 / storage | EX-S01; adapter EX-AD-ST2; port EX-SP1; cable EX-C102 | EX-STSW2 / P11, CP-R4 | EX-MGSW1 / P12, attachment EA-ST02-R3 | Unresolved destination discrepancy; network/storage owner must identify the applicable destination and attachment issue. Do not silently replace either value. | Planning analysis |
| EX-MG01 / management | EX-S01; BMC reference EX-BM1; cable EX-C201 | EX-MGSW1 / P12, CP-R4 | EX-MGSW1 / P12, attachment EA-MG01-R4 | Endpoint references agree for this row; identify its relationship to the conflicting EX-ST02 reference in the owner return. No management-access result supplied. | Planning analysis |
| EX-DT01 / customer data | EX-S01; adapter EX-AD-CD1; port EX-CP1; cable EX-C301 | EX-CDSW1 / P21, CP-R4 | EX-CDSW1 / P21, attachment EA-DT01-R4 | Endpoint references agree; the customer-data owner retains the local role definition DR-R2. No data-service result supplied. | Planning analysis |
Every identifier and issue is invented. The physical destination of each storage row is a switch port, not a claimed direct cable to a storage controller. This worksheet neither specifies a topology nor validates adapter fit, media, port modes, separation, security, redundancy or performance. Expected, received and later observed values are different evidence states.
Return the discrepancy without manufacturing a completed connection
Send the designated network/storage owner the affected connection ID, expected source/role/destination and issue, conflicting attachment and issue, related row references, and the specific decision needed. In EX-DOM19 the question is which destination and source issue govern EX-ST02, and how the EX-MG01 reference is accounted for. It is not a request for an installer to select a convenient spare port. A familiar management-switch name is a reason to resolve the supplied records, not proof of the actual port configuration or a diagnosis of a misconnected cable.
An owner return could supply a corrected attachment retaining CP-R4, or identify a separately authorized design change and its applicability. If shared infrastructure or another relationship is intentional, the return needs the record defining it rather than a claim inferred from similar labels. Keep the superseded attachment and link the new disposition to the original discrepancy. Limit that disposition to its named rows and source issue; do not treat it as approval of every connection on the server.
Carry the resulting matrix into handoff with separate fields for planned identity, actual endpoint observations, any agreed physical verification reference and the responsible system/service acceptance reference. A document correction closes only the identified record conflict. If EX-ST02 has no physical observation or verification evidence, those fields remain not supplied even after the destination is clarified. The network/storage team can then see what is documented, what was actually observed and what still needs its own agreed evidence without mistaking a reconciled spreadsheet for storage readiness.
Scroll the table horizontally on smaller screens.
| Record item | Required return or evidence | What remains separate | Source |
|---|---|---|---|
| EX-ST02 governing endpoint | Named owner’s disposition identifying the applicable CP/attachment issue, destination and affected connection ID | No field change, port selection, configuration or work authorization is supplied by this example. | Planning analysis |
| EX-MG01 relationship | Explicit treatment of the destination reference also appearing in EX-ST02, with the applicable design/record relationship | No inferred physical sharing, isolation, security or topology validation. | Planning analysis |
| As-installed endpoint evidence | Separately recorded observations tied to the actual connection, observer, date and evidence reference, if supplied | A corrected planned record is not proof that a cable was installed there. | Planning analysis |
| Physical verification and service acceptance | Distinct agreed evidence references and the parties responsible for their defined scope, or an explicit not-supplied state | Neither a document agreement nor a visible link establishes authenticated access, storage access, configuration or workload performance. | Planning analysis |
Original documentary decision aid. It does not prescribe technical verification, security or configuration procedures, supply actual evidence, assign contractual acceptance authority or guarantee an operational result.
State the limits of a successful physical check
An endpoint reconciliation can establish agreement with the released schedule. An agreed measurement supports its stated physical criterion. Neither proves that a host can discover a storage target, access the correct dataset or meet an application requirement. Those are separate configuration and service questions.
If formal optical measurement is required, define the actual equipment, personnel, method and criteria. Do not use a generic instrument list as the test specification. Fluke distinguishes insertion-loss measurement from OTDR event characterization; the required evidence must fit the question being accepted.
Resolve a missing storage-role reference before handoff
Illustrative example: a cable reaches the scheduled switch port, but the record never identifies which host adapter it serves. Capture the physical endpoints and raise the missing role/adapter mapping with the design owner. Do not mark storage service accepted simply because the link is present or an indicator is lit.
Deliver the physical register, approved changes, open items and evidence references to the network/storage owner. Leviathan can discuss the agreed physical cabling and documentation package. Storage architecture, protocol tuning, configuration, data migration and performance commissioning are not established by that package.
References: [L1]
Identify the NVMe transport before specifying the channel
NVMe over Fabrics describes access to storage across a networked fabric; it is not the name of one cable or connector. SNIA distinguishes Fibre Channel, RDMA and TCP transports. Its RDMA category includes InfiniBand, RoCE and iWARP. NVM Express separately describes NVMe over TCP as mapping NVMe command and data transport over TCP/IP. These distinctions matter before a physical team interprets an instruction labelled NVMe storage.
Ask the storage/network owner which transport and fabric the approved design actually uses, which host adapter and remote port provide it, and which supported assembly or complete channel connects them. Keep the full equipment/port and media references with the connection record. A copper or optical appearance cannot establish the transport, and a connector form cannot establish host, switch or storage-controller compatibility. Do not derive a channel requirement from a protocol name or assume that a cable reaches a storage device when its physical far end is a switch.
The table is a way to expose the missing input. It does not choose a protocol, prescribe a network architecture, state reach or loss limits, or supply configuration and test steps. A fabric can support several services, and its physical population need not equal the number of logical service names. The design owner must identify the actual shared or separate relationship; physical documentation should preserve that relationship rather than create an extra cable for every label.
Scroll the table horizontally on smaller screens.
| Declared transport | Physical question still to answer | Separate next-team responsibility | Source |
|---|---|---|---|
| NVMe over TCP | Which actual adapters, ports and supported channel form the identified IP-network connection? | The storage/network owner defines configuration, discovery and service requirements. | [V6] |
| NVMe over RDMA | Which actual RDMA fabric and approved endpoint/channel implementation apply? InfiniBand, RoCE and iWARP are not one interchangeable equipment specification. | The responsible designer selects and qualifies the actual fabric and system implementation. | [S6] |
| NVMe over Fibre Channel | Which Fibre Channel equipment, termination and supported media references govern the connection? | The responsible system/network team establishes the applicable service and acceptance requirements. | [S6] |
Educational transport/interface distinction. No arbitrary optic, adapter, protocol, field method or product combination is approved by this table.
Compare path independence against a named failure boundary
Two connection IDs establish two recorded connections. They do not, by themselves, establish redundancy. Ask what failure the design is intended to tolerate and which parts of each path must remain independent for that requirement. Trace the identified adapters, physical terminations, switches, patch positions and shared route references to the extent covered by the approved inputs and permitted evidence. A physical record cannot establish software failover or end-to-end storage availability.
There is a useful source-specific distinction in NVIDIA’s cited NVL72 enterprise reference architecture: the CPU-converged network description connects a node to two separate switches. That is a named reference design feature, not a universal port count or a complete redundancy specification for another installation. It demonstrates why two host-port entries and two different remote-switch identities answer different questions. Actual site requirements and implementation evidence remain separate.
Fictional EX-ST127 compares two separately stipulated two-connection plans. In Plan A, both host connections terminate on switch EX-SW1 and the supplied route records name the same trunk. If the stated review question is loss of that one switch, the plan does not demonstrate switch independence. No failure event or service outcome has been observed. In Plan B, the two destination switches differ, but the second route record is missing. The different switch IDs answer the stated switch-identity question; they do not prove route independence, separate power sources or an operational failover result.
Return the specific missing relationship or evidence to the nominated network/storage owner. Keep a declared design objective, an identified dependency, an actual physical observation and a witnessed service result as different fields. Closing a route-document question should not automatically close a storage-service acceptance item. The physical handoff can make those boundaries inspectable without claiming that a two-row schedule proves an available storage system.
References: [N6]
Scroll the table horizontally on smaller screens.
| Separate fictional plan | Supplied relationship | Supported documentary conclusion | Source |
|---|---|---|---|
| Plan A: two host connections | Both terminate on EX-SW1; both route references name trunk EX-T1. | Two connection records share a stated switch and trunk. Switch independence is not demonstrated; no actual fault or availability result is supplied. | Planning analysis |
| Plan B: two host connections | Destinations EX-SW2 and EX-SW3 differ; only the first path has a route record. | Different destination switch identities are documented. The missing second route prevents a conclusion about route independence. | Planning analysis |
| Both plans: evidence boundary | No source-power, configured failover, storage-target or service-test result is supplied. | Neither example establishes an accepted redundant storage service or a safe test procedure. | Planning analysis |
Original fictional comparison. The plans are alternatives, not four links in one installation; identities are invented and no actual site topology is disclosed.
Common planning questions
Does this guide specify an NVMe-oF or RDMA design?
No. It describes the physical records needed to hand a connection to the responsible network/storage team. Architecture and configuration require their own qualified review.
Can we accept storage readiness from a link indicator?
An indicator is a limited observation. The accepting owner must assess the defined physical and system evidence for the actual storage service.
Do separate management, customer-data and storage records require separate physical networks?
The approved project design determines the physical arrangement. Keep the role definition and physical endpoint identity in separate fields, including an explicit design reference where infrastructure is shared. Separate worksheet rows do not prove physical isolation or redundancy; a shared heading does not make the connections interchangeable.
Does a reconciled pre-install port matrix establish installed connectivity?
No. It records agreement between the identified planning inputs within the stated scope. Actual endpoint observations, agreed physical verification and storage/network service acceptance require their own evidence and responsible owners. In EX-DOM19, resolving the one destination discrepancy would not create any installed or tested result for the four planned rows.
Should a server-to-storage row name the storage array when the cable ends at a switch?
Record the real planned or observed remote physical endpoint as such: a switch chassis and port when that is where the connection terminates. Reference downstream storage/service relationships in a separate owner-supplied design or operational record. Do not turn the service destination into a fictional direct cable endpoint.
Does an NVMe-oF requirement identify the cable to install?
No. Identify the actual transport, host and remote physical ports, fabric implementation and supported assembly or complete channel. NVMe over TCP, RDMA and Fibre Channel are transport categories, not one interchangeable cable specification.
Do two recorded storage links prove physical redundancy?
No. State the failure boundary and compare the actual shared and separate dependencies. Different switch IDs do not prove independent routes or power, and physical records do not establish configured failover or storage-service acceptance.
Primary sources
- [N5] NVIDIA DGX GB rack-scale systems: networkingChecked 2026-10-14. Model-specific compute, storage and management network roles. Select the applicable product documentation; this is not a generic topology for all GPU systems.
- [F1] Fluke Networks: OLTS and OTDR testing strategyChecked 2026-10-14. Measurement-role distinction only. A project must separately define method, personnel, instruments and acceptance criteria.
- [L4] Leviathan physical-layer verification scopeChecked 2026-10-14. Formal testing is conditional on confirmed resources, equipment, method and agreed criteria; no blanket certification promise.
- [L1] Leviathan structured cabling scopeChecked 2026-10-14. Approved physical cabling service boundaries; exact design, materials, authorization and verification are project-specific.
- [S6] SNIA: What is NVMe over Fabrics?Checked 2026-10-19. Transport taxonomy and host-to-storage fabric concept only. No actual compatibility, media choice, performance or field method.
- [V6] NVM Express: NVMe over TCP Transport Specification overviewChecked 2026-10-19. General NVMe-over-TCP command/data mapping over TCP/IP only. No specification conformance, configuration, encryption, performance or product qualification claim.
- [N6] NVIDIA NVL72 AI Factory: Network Logical ArchitectureChecked 2026-10-19. Named reference CPU-converged role and its two separate destination switches only. No generic topology, rate, power/route independence, configured failover or customer-system acceptance.