An S3 ETag Is Not Always a File Checksum: Compare AI Dataset Integrity Evidence

Identify the delivered byte object before comparing values

A dataset delivery might be one archive, one shard or a collection of objects. Begin the comparison with the particular stored object under discussion: its bucket and key, version ID where applicable, and exact size in bytes. Also identify the sender's reference file. A release name shared by both records does not establish that they concern the same bytes.

AWS's GetObjectAttributes model separates ETag, Checksum, ObjectParts and ObjectSize, and can identify the object's version. These are different pieces of evidence. A byte count helps describe the object, but two files can have the same length and different contents.

For a compressed archive, the stored archive and an extracted member are separate byte objects. Their relationship may be useful to the application team, but a checksum of one is not a checksum of the other. Keep that boundary visible before treating any disagreement as a transfer defect.

References: [A4]

Read the ETag's creation and encryption scope

AWS documents MD5 ETags for single-part objects created through PUT, POST or Copy, including the corresponding console cases, when the object is plaintext or uses server-side encryption with S3 managed keys, SSE-S3. For those qualifying objects, interpreting the ETag as the MD5 digest of the object data is appropriate.

For the named creation operations with AWS KMS keys, SSE-KMS, or customer-provided keys, SSE-C, the ETag is not the MD5 digest of the object data. Objects created through multipart upload or part copy also lack a full-file MD5 ETag, regardless of encryption.

These are the general purpose S3 cases in AWS's documentation checked on October 20, 2026. Directory buckets do not support MD5. AWS also states that an ETag reflects content changes rather than metadata changes, so the tag cannot establish that accompanying metadata stayed unchanged.

References: [A2]

Separate full object coverage from a composite calculation

A full object checksum represents the ordered bytes of the complete object. A composite checksum combines individual part checksums. Both can concern a completed upload, but they use different calculations.

For a multipart ETag, AWS describes hashing the concatenated binary MD5 digests of the parts, then adding a part-count suffix. Removing that suffix does not turn the remaining value into a whole-file MD5.

Multipart upload can also have a full object checksum: AWS supports CRC64NVME, CRC32 and CRC32C for that type. Its documented multipart SHA-256 checksum is composite. A multipart ETag therefore does not mean that full object integrity evidence is unavailable.

References: [A1]

Distinguish a different calculation from a different encoding

The S3 Checksum model exposes the checksum type as FULL_OBJECT or COMPOSITE. It also describes ChecksumSHA256 as Base64 encoded and warns that a multipart value may be derived from part checksums. Seeing SHA-256 beside two values is insufficient to establish that they were calculated over the same inputs in the same way.

Encoding is a separate question. RFC 4648 defines Base64 and hexadecimal as ways to represent binary data in text. The same digest bytes can have different written forms under those encodings. A supplier's hexadecimal whole-file SHA-256 value and a Base64 whole-file SHA-256 value need a representation-aware comparison.

Aligning representation can resolve that textual difference. It cannot turn MD5 into SHA-256, convert a composite calculation into a full object calculation, or make a checksum for another file relevant to this object.

References: [A3][R1]

Follow one archive through three evidence gaps

Consider a fictional delivery consisting of one compressed archive, R, containing two dataset shards. One extracted shard is S. R and S are symbolic object labels. The example supplies no digest strings or performed transfer result.

The supplier's reference report explicitly gives a whole-file SHA-256 value for S. Its S3 record identifies R, created through multipart upload with SSE-KMS, and includes an ETag plus a COMPOSITE SHA-256 checksum.

The report for S leaves R's transfer integrity unassessed because it describes different bytes. Comparing R's ETag with that SHA-256 value adds a calculation mismatch: this ETag is not R's whole-file MD5, and MD5 would not be the same algorithm as SHA-256 anyway.

Now suppose the supplier replaces the report with a whole-file SHA-256 reference explicitly for R. The object gap is resolved, but the S3 composite SHA-256 value still uses a different construction. Merely aligning Base64 and hexadecimal would leave that distinction in place.

The useful next evidence connects the same R bytes across the stated transfer boundary using a common calculation and compatible representation. That could be a documented full object comparison, or a composite comparison with the corresponding part structure and expected value. The three gaps above justify clarification of the evidence; they establish neither that R was corrupted nor that its bytes matched the sender's reference.

References: [A2][A3][R1]

Read what the supplier actually compared

A useful supplier explanation names the source reference and delivered object, identifies the algorithm and checksum type, states the representation, and records which values were actually compared and at what transfer stage. With composite evidence, the corresponding part calculation must also be established.

AWS describes S3 comparing an uploaded object with a supplied MD5 value and returning an error if they differ. That is a specific comparison. A checksum first calculated from the received object, without a connection to an expected source value, leaves source-to-destination agreement unestablished. Conversely, a valid comparison need not be rejected solely because the evidence uses an appropriate checksum other than MD5.

Distinguish an invalid comparison from a demonstrated mismatch before requesting another transfer. The related Object Lock article explains version selection and retention, while the SBOM and build-provenance article explains software component and production claims. Use those guides when the supplier's claim extends to retained versions or software origin.

References: [A3][A5]

Connect the surrounding physical storage work

If the wider project also requires storage-facing cabling, the GPU Storage Network Handoff guide explains how to trace installed physical connections. Leviathan's Structured Cabling & Fiber Installation service covers agreed customer-supplied cabling, endpoint identification, labels and connection records.

Use the related physical-project inquiry with a nonconfidential equipment outline, approved endpoints and routes, requested installation work and required handoff records. Dataset transfer methods and checksum validation remain with the appointed storage and application providers; the inquiry concerns the surrounding physical installation.

References: [L1]