Strong S3 Reads Do Not Make a Multi-Object AI Dataset Release Atomic

Locate the consistency boundary at one object

AWS describes strong read-after-write consistency for S3 object operations: after a successful write, a subsequent read returns the written value. The examples here exclude competing or later writes while each stated observation is made.

AWS also describes updates to a single key as atomic. A reader overlapping an overwrite receives a whole earlier or later value, rather than a partially overwritten object. The documentation explicitly places updates at the key boundary and provides no atomic update across different keys.

These properties answer an object-visibility question. Dataset coherence asks whether the selected objects belong together under the application's release definition. A correct answer to the first question can coexist with an incomplete release. Read the supplier's use of available against the unit it actually describes.

References: [A1][A2]

Read four artifacts from two generations

Consider a fictional training dataset with exactly four required artifacts: samples-01.parquet, samples-02.parquet, labels.parquet and split-index.json. Each occupies a separate key. Generation A is initially complete. Generation B changes sample identifiers, so this application's B release requires both B sample shards, B labels and the B split index.

At the source bucket, the B replacements for samples-01.parquet, samples-02.parquet and labels.parquet have completed successfully. The B replacement for split-index.json has not yet been submitted. Its last completed contents remain A. There are no other writes during the following reads.

All four reads succeed. They return B samples-01.parquet, B samples-02.parquet, B labels.parquet and A split-index.json. Every object is whole, and every result agrees with that key's completed write history. The fourth read has no B write to reflect.

The set nevertheless fails the stipulated B release requirement. Its split index identifies the earlier sample population. Four readable artifacts establish neither four B artifacts nor the required relationship between index, samples and labels. Successful reads have faithfully exposed the intermediate state.

That distinction changes the buyer's conclusion. An offer evidenced only by four successful reads has shown object availability under the stated conditions. To support a B dataset claim, the evidence must also identify the generation of each returned artifact and its membership in the required set.

Connect a release declaration to actual contents

A manifest can make the intended set explicit. In the example, a B manifest would need to identify all four required B artifacts. Reading that manifest proves that its contents are available; the declaration still needs to be reconciled with the objects it describes.

Suppose it lists only the four familiar key names. During the intermediate state above, those names resolve to three B objects and one A object. A B label on the manifest cannot change that selection. The declaration needs references that identify the intended contents, including the relevant version identities where applicable, and evidence connecting the consumer's selected objects to those references.

A manifest can itself be one atomically updated object. That property concerns the manifest's bytes. It does not validate the relationships asserted inside it, make every referenced artifact available or confer permission to use the data. The appointed application and storage teams must explain what their release declaration establishes.

This is the gap to resolve when an offer says the manifest exists. Ask which actual contents it identifies and what establishes that the consumer receives that complete set.

References: [A1]

Assess the replicated destination separately

AWS describes S3 replication between buckets as asynchronous. Cross-Region Replication copies between buckets in different AWS Regions. A completed source write therefore supplies no immediate conclusion about whether its copy has reached the destination.

Advance the fictional example. The B split index has now been successfully written at the source, and all four source objects meet the example's B identity requirement. A configured replication arrangement includes all four B updates. At the stated observation, the first three B copies have arrived at the destination; the B split-index copy remains pending, with A still present there.

Source reads now select the complete B set. Destination reads still select B, B, B and A. A whole A split index at the destination can accurately reflect its local stored state while the B copy is pending. This is compatible with strong object reads: propagation between the buckets is a separate event.

Replication coverage also matters. AWS distinguishes live replication for new and updated objects from on-demand Batch Replication for existing objects. Enabling live replication alone does not copy objects that predate its setup. Establish which required contents the proposed arrangement actually covers before treating a destination gap as something that merely needs more time.

A destination readiness claim needs evidence for the intended release there. Neither the source's successful reads nor the existence of a replication arrangement establishes when that complete destination set will be available.

References: [A3]

Compare the availability claim with the required result

Three statements can carry different value in the same proposal. One named object is readable. The source holds the complete identified B release. The destination holds that same release for the proposed consumer. Evidence for an earlier statement leaves the later statements unresolved.

If training will read from the destination, ask the supplier to identify the release, its required artifacts, the exact contents selected there and the observation supporting completeness at that location. A source-only demonstration answers a narrower question. A destination object count can also leave the generation mismatch hidden, as the four-artifact example shows.

Choose the offer against the missing result. Buying additional replication coverage can address a missing-copy requirement; the application still needs a coherent release definition. Buying release coordination can address the grouping requirement; a complete source set still needs destination evidence when that is where the consumer will read.

The related Object Lock guide addresses retained version identity and current-key selection. The Training Checkpoint or Storage Snapshot guide addresses captured application state. Those questions remain useful alongside this article's distinct question: which complete dataset generation is present at the location being claimed?

Scope any physical storage connection work separately

If the wider deployment also needs physical storage-network connections, the related GPU Storage Network Handoff guide covers endpoint and connection records. Leviathan's Structured Cabling & Fiber Installation service and its project inquiry provide a route to discuss agreed cabling, labels and physical handoff records.

Share a nonconfidential equipment summary, approved endpoints, requested physical work and required records. Dataset release coordination, cloud replication administration and application validation remain with the appointed application and storage providers.

References: [L1]