One HDF5 Virtual Dataset Does Not Mean One Self-Contained Input

Follow the selected positions beyond the array interface

The HDF Group describes a virtual dataset, or VDS, as a way to access datasets across a collection of HDF5 files through one dataset interface. Its source-to-virtual mappings persist with the dataset. The documented feature was introduced in HDF5 1.10; this explanation uses Field Guide pages read on October 20, 2026, whose headers state Last Updated on October 4, 2026.

A mapping identifies a source file, a dataset within it and a selection of source elements, together with their selected positions in the virtual dataset. Think of it as an address relationship: these source positions supply those logical positions. The dataset-creation reference even permits naming a source file or dataset that does not exist yet. A recorded mapping therefore cannot itself establish delivery of the named contents.

The mapping can also identify a source dataset in the same file. Here the decision concerns the named virtual dataset whose mapped values live in separate files. Its container could hold other datasets as well; calling one dataset virtual says nothing about whether those unrelated datasets contain ordinary stored payloads.

References: [VD01][VD03]

VDS-01: map two sources into one fixed 6 × 4 view

Consider a fictional scientific-AI input requested as six rows and four columns, with all 24 source values required at their original positions. Source A contains a 3 × 4 dataset and source B contains another 3 × 4 dataset. Each has 12 applicable values. Stipulate a direct mapping, without numerical transformation: A's rows 0 through 2 supply virtual rows 0 through 2, and B's rows 0 through 2 supply virtual rows 3 through 5. Columns 0 through 3 are preserved in both mappings.

The example explicitly uses a custom VDS fill value of -999. It also stipulates a supported recipient read that returns the fixed 6 × 4 extent. Source paths have resolved under the recipient's stated reading environment. A is present and readable with its 12 values; B's source file is missing at its resolved location, and the documented missing-source-file fill behavior applies.

These premises describe one explanatory case. Dataset access properties are runtime properties rather than persisted dataset properties. The documented VDS view controls and, for unlimited printf-style mappings, source-discovery gap rules qualify extent interpretation. The fixed returned extent here is a premise, not a default promised for every VDS. The named missing-file behavior also supplies no universal outcome for permission denial, malformed sources or other access failures.

References: [VD02][VD03]

Account for 12 source-backed positions and 12 fill returns

Rows 0 through 2 contain 3 × 4 = 12 positions backed by A's stipulated source values. Rows 3 through 5 contain another 3 × 4 = 12 positions, each returning -999 because B's file is missing under the case's stated behavior. The returned array therefore has 12 + 12 = 24 positions, while only 12 positions received their mapped source values.

A successful 24-number return has preserved the stipulated shape. It has not supplied the requested 24 source values: the lower half was filled without reading B's values. Describing that result as 24 delivered source values would count the fill returns as values obtained from B.

The arbitrary -999 is chosen only to make this case readable. It is neither a recommended missing-data encoding nor a universal provenance detector. A source value could itself equal that number. The accounting above follows the stipulated source availability and mapping behavior; it cannot be recovered universally by counting numbers equal to -999.

No HDF5 operation or test was run for this example. Its counts describe logical positions, with no implied file-byte, compression, memory, latency or transfer ratio.

References: [VD01][VD02]

Change B's availability while keeping the mapping unchanged

Now change only source availability. B's file is available and readable with its 12 applicable source values at the recipient's resolved location. Keep A, both mappings, the fill setting, the reader assumptions and the fixed returned extent unchanged.

Rows 0 through 2 can still obtain A's 12 values. Rows 3 through 5 can now obtain B's 12 values. All 24 positions can be source-backed even though the mapping is identical to the one that previously produced 12 fill returns.

The unchanged mapping describes the same address relationships in both cases. It did not capture B's contents while the file was absent, and its identity alone does not substantiate which version of those external contents is now available. A virtual mapping is therefore insufficient evidence of a transactional snapshot, pinned source version, immutable copy or backup. Those would be additional properties requiring their own support.

References: [VD03]

Compare both offers against the same requested 24 values

A virtual-composite offer can be appropriate when the intended source set remains available to an authorized, supported reader. For VDS-01, that means the recipient can obtain A's and B's applicable values through the mappings. The unified interface can be the requested deliverable even though those values remain distributed across files. The mapping file alone leaves that dependency unresolved.

A self-contained value offer answers a different delivery requirement: the returned artifact contains the requested values without depending on the separately supplied A and B files. In this case, meeting the full requirement means carrying all 24 applicable values, with their row and column positions preserved. Merely calling an export materialized does not establish which values it contains.

Suppose an offer copies the entire missing-B return into an independent stored array. That artifact could be self-contained as a representation of the observed return, yet it would carry A's 12 values and 12 occurrences of -999. Copying the result cannot supply B's absent values. It answers a request to preserve that return, rather than the stipulated request for all 24 source values.

With B available, a self-contained offer could instead carry all 24 source values. Evidence must connect its delivered contents and position correspondence to the intended A and B contents. Conversely, if the buyer wants continued access to the distributed sources through one interface, a usable virtual composite may satisfy the requirement without that independent copy. Neither representation is universally the better purchase.

Source-backed values still need an agreed scientific interpretation

For an actual offer, have the data or software provider explain which requested positions obtain values from which identified source contents at the intended recipient boundary. The useful distinction is whether the consumer receives the full requested source population, a virtual dependency on it, or a stored result containing fill returns. A demonstration made where B is available does not establish what a recipient missing B will receive.

Source-backed does not by itself mean scientifically valid. Units, variable meaning, coverage and fitness for the AI task remain to be established, together with permissions, intended source versions and the consuming application's interpretation of fill returns. B's absence in this case proves neither corruption nor bad science; the example supplies no cause for that absence.

The related S3 guide addresses coherent multi-object releases at an identified location. That is a separate question from whether a logical array obtains its selected values from external HDF5 sources. Keeping those questions distinct helps the buyer identify the result still needed from the offer.

Discuss a separate installation of customer-selected equipment

If the wider scientific-AI project also requires physical equipment installation, Leviathan's Rack-and-Stack & Colocation Installation service and the related project inquiry provide a route to discuss agreed mounting, physical IT cabling, labels and installation records. Share a nonconfidential equipment summary, rack layout, connection requirements, site and desired work window to discuss scope and availability.

That inquiry concerns the separate physical installation. Dataset preparation, HDF5 configuration, scientific interpretation and software acceptance remain with the appointed providers.

References: [L1]