Trace AI Dataset Input Through an Iceberg Snapshot

Follow the table selection before counting stored files

This explanation uses the Apache Iceberg specification tagged apache-iceberg-1.10.0, selecting table format-version 2, and the versioned 1.10.0 documentation. The table format version and the software documentation version are separate references.

The selection chain is table metadata, selected snapshot, manifest list, manifests and live file entries. A manifest list identifies manifest files; each manifest records entries for data files or delete files. A data-prefix listing identifies stored objects; it does not select that metadata chain. Live membership excludes entries marked deleted from the selected snapshot's scan.

An immutable data file can already be stored before the table commit that adds it succeeds. The commit atomically changes the table-metadata reference; it does not atomically move every data object. Readers obtain a committed table state through that reference. Concurrent writers must validate their assumptions when retrying after a competing commit; an unsuccessful attempt does not establish a new selected state. The tracking or catalog implementation determines the specific commit mechanism.

References: [IX1][IX2]

IS-01: a rewrite changes files, then an append changes rows

Consider one identified Iceberg format-v2 table, T, with a fixed two-column schema, row_id and value. Here row_id is an ordinary invented data column. The table is unpartitioned. Each observation concerns an ordinary full-table read with no filters, row-level delete files or external joins. Every valid snapshot in this example contains exactly one row for each of its logical IDs.

A contains IDs 1 through 100 and B contains 101 through 200: 100 rows each. C contains IDs 1 through 50 and D contains 51 through 100: 50 rows each. C and D together preserve exactly A's rows and values. E contains IDs 201 through 240: 40 rows. These letters label fictional immutable Parquet files, rather than real storage addresses or metadata payloads.

The stipulated chronology is snapshot 701, then 702, then 703. The numbers are illustrative snapshot IDs within T, not sequence numbers from which time can be inferred.

Initially, committed 701 selects A and B: 200 rows in two files. A committed rewrite produces 702, replacing A's membership with C and D while retaining B. Its three live files still contain the same 200 logical rows. Retaining 701 leaves A stored; removing A from the newly selected state does not physically erase it.

A later committed append adds E and produces 703: 240 rows in four live files. At the final observation, 703 is current, all three snapshots remain valid and retained, and all required metadata and file contents are readable by the stipulated authorized, capable reader. No writes occur during the final observations.

Trace 702 even after 703 becomes current

For the chosen 702 input, start with T's retained table metadata and select its 702 snapshot entry. Follow that snapshot's manifest list to its manifests, then resolve their live data-file entries to B, C and D. Their stipulated contents supply 100 + 50 + 50 = 200 rows. A's continued physical existence does not put it back into that selection.

A newly current selection follows 703's entry and its manifest list and manifests. That tree identifies B, C, D and E, supplying 100 + 50 + 50 + 40 = 240 rows. E's later existence does not insert it into 702's tree.

These are conceptual traces, not serialized Iceberg metadata examples. Unchanged manifests can be shared between snapshots; each snapshot need not create a complete physical copy of the data. B can remain part of all three selected states without being copied three times.

References: [IX1][IX2]

A perfectly accurate prefix listing can yield 340 records

Now stipulate the complete data-file inventory beneath T's data prefix at that final observation: A, B, C, D and E, with no other data files. Metadata objects are outside this five-file count. The listing is accurate and complete.

A consumer that concatenates all five listed files reads 100 + 100 + 50 + 50 + 40 = 340 row records. IDs 1 through 100 occur twice because A overlaps C and D. The concatenation contains 240 distinct IDs in this case, but its 340 records are neither the selected 702 input nor a correct ordinary full-table read of 703.

This discrepancy supplies no evidence of corrupt files or an inconsistent object listing. It follows from substituting physical existence for table membership. A is present for retained history while C and D represent its replacement in later states.

The example supplies no universal reconstruction algorithm. Other datasets may legitimately repeat IDs, and this case provides no basis for treating deduplication or deletion as a repair. The committed selection already defines which files belong. The complete listing faithfully describes a different population.

Specify a fixed snapshot and the reader that honors it

The useful input reference is T in its defined catalog or table context, with snapshot 702 explicitly selected. The bare number 702 and a latest-directory timestamp leave that relationship unresolved. The current selection advances to 703 in this chronology. A branch-head reference can also advance; it does not permanently identify one historical snapshot. A fixed snapshot-ID selection still identifies 702 while that historical state remains usable.

Iceberg's 1.10.0 Spark query documentation distinguishes snapshot-ID selection from branch-head selection. Its documented snapshot-ID time travel uses the snapshot's schema. IS-01 holds the schema fixed; a buyer changing that premise must establish the actual integration's schema behavior and reader/version support. An offer merely naming Iceberg does not establish that the consumer honors its requested historical selection.

Format-v2 readers must also handle applicable delete files when they exist. IS-01 deliberately has none, so adding its live data-file row counts gives the logical total. That arithmetic is not a universal logical-row-count rule for tables with row-level deletes.

References: [IX1][IX3]

Compare what the buyer can reproduce after each change

Suppose a supplier returns only the final prefix inventory. It has accounted for all five stored files, but it has not established whether the AI consumer follows retained 702, current 703 or the stipulated concatenate-all behavior. More accurate inventory detail cannot choose among those inputs.

A second offer identifies T, selects retained 702 and explains the supported consumer behavior and availability scope. Ask which table and snapshot the consumer actually resolved and what observation connects that result to the requested 702 reference. A request alone establishes intention. Under IS-01's premises, selecting retained 702 identifies the 200-row post-rewrite input even after the append. If the requirement instead follows current table contents, a newly current 703 selection identifies 240 rows. Those are different requested results.

The rewrite also distinguishes logical contents from file identity. Both 701 and 702 contain the same 200 rows and values, yet only 701 selects A and B. A requirement to return that original file representation cannot be answered merely by showing that 702 has the same row population. Conversely, selecting 702 deliberately preserves its B, C and D membership without admitting the later E append.

Neither offer is automatically the better purchase. Raw files may be the requested deliverable. Reproducing a specified historical table state needs the second selection capability and its dependencies. Have the appointed data-platform provider connect its offered result to the buyer's required state.

The related S3 coherent-release guide addresses generic cross-object generation and location evidence. The checksum guide addresses exact byte-integrity comparisons. Once table membership is selected, the Parquet guide explains file and within-file exclusion and remaining scan work. Each resolves a different question.

Retain the selected state without overstating reproducibility

A recorded snapshot ID does not preserve its own dependencies. The selected state requires usable retained metadata, referenced contents and the consumer's authorized ability to read them. Iceberg's 1.10.0 maintenance documentation explains that expiring a snapshot removes its historical time-travel availability; data files remain needed while retained snapshots still reference them. Recording 702 therefore supplies neither an independent backup nor an indefinite-retention guarantee.

What IS-01 positively establishes is a selected input population: 200 rows and their values from B, C and D for T's 702 state. It does not specify row ordering, feature-transform or training-code versions, labels held elsewhere, representative sampling, event-time or known-time correctness, or identical future training. Those are additional application requirements, rather than properties supplied by the table-state identity.

Likewise, committing this single table does not create an atomic multi-table or whole-application release. If the proposed return also covers training-state continuation or a model-serving release, the related checkpoint and registry guides address those distinct downstream dependencies.

References: [IX4]

Connect a separate physical storage-network project

If the selected platform also requires physical connections between installed compute and storage equipment, Leviathan's Structured Cabling & Fiber Installation service and the related project inquiry provide a route to discuss that work. Bring a nonconfidential equipment and site summary, approved endpoints and routes, material-supply responsibilities and required connection records.

The inquiry concerns agreed physical cabling, labels and handoff. Table configuration, catalog administration, reader integration and dataset validation belong to the appointed software and storage providers.

References: [L1]