---
title: "Zarr Sharding: Keep Inner Read Chunks Separate from Storage Objects"
description: "Leviathan Systems — Technical content Published 2026-10-21; updated 2026-10-21."
canonical: "https://www.leviathansystems.co/articles/zarr-inner-chunks-shard-object-granularity"
last-updated: "2026-10-21"
---

# Zarr Sharding: Keep Inner Read Chunks Separate from Storage Objects

Leviathan Systems — Technical content
Published 2026-10-21; updated 2026-10-21.

Zarr sharding lets several independently encoded inner chunks share one indexed storage object. A supported reader can locate an inner chunk through that index while preserving the array's sample coordinates. Compare the inner shape, outer shard shape and reader's codec support together. Counting fewer stored objects establishes a different layout; actual requests and transferred bytes still depend on the access path.

## Two chunk shapes describe two storage layers

The useful mechanism is separation of the independently encoded read unit from the stored object. ZEP0002 describes sharding as a way to retain smaller chunks while combining nearby chunks into larger storage units. This gives a scientific-AI data provider another layout choice without requiring a change to the logical sample grid.

In the codec model, the core array's grid describes the outer chunks, called shards. The sharding_indexed codec subdivides each shard into inner chunks. Its nested codecs encode those inner chunks, and an index connects their positions to their encoded contents. Inner chunks can be individually addressed even though they share an object.

An offer saying only “chunk shape” leaves the layer ambiguous. An outer 4 × 4 shard can contain four independently encoded 2 × 2 inner chunks. A single ordinary 4 × 4 chunk has the same outer footprint but does not establish that inner organization.

References: [ZEP2](https://zarr.dev/zeps/accepted/ZEP0002.html), [ZS10](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html)

## ZR-40: reconcile 64 values, 16 inner chunks and four shards

Consider a fictional, fully populated 8 × 8 scalar array. Coordinates are written as (row, column), beginning at zero. Both layouts below retain the same data type, values and sample positions through supported lossless encoding. All 16 inner chunks are stored. There are no missing chunks, fill substitutions, additional transformations or changes of resolution in this case.

The sharded layout uses a regular outer grid with 4 × 4 shards and 2 × 2 inner chunks. There are 8 / 4 = 2 shards along each axis, hence 2 × 2 = four shards. Each contains (4 / 2) × (4 / 2) = four inner chunks. Across the array that is 4 × 4 = 16 inner chunks, each holding four scalar positions: 16 × 4 = 64 values.

In the individually stored alternative, the core regular grid instead uses 2 × 2 chunks. The same population of 16 small chunks occupies 16 data keys. Sharding places those small units within four data keys. These counts exclude metadata keys and describe an explanatory layout, with no file or reader execution implied.

References: [GRID](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html), [ZS10](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html)

## Window A crosses four inner chunks inside one shard

Use half-open coordinate intervals: [1, 4) includes 1, 2 and 3, stopping before 4. Window A selects rows [1, 4) and columns [1, 4), giving 3 × 3 = nine scalar values.

The inner row bands are [0, 2) and [2, 4). The request uses one row from the first and two from the second. Its column intersections have the same lengths. The four intersected inner chunks, addressed in the whole array's inner grid, are (0, 0), (0, 1), (1, 0) and (1, 1).

Their contributions to the requested window are respectively 1 × 1 = 1, 1 × 2 = 2, 2 × 1 = 2 and 2 × 2 = 4 values. The sum is nine. All four inner chunks belong to outer shard (0, 0), which spans rows [0, 4) and columns [0, 4).

The complete intersected inner chunks contain 4 × 4 = 16 positions. Nine positions are requested. The larger count describes the chunks' coordinate coverage, rather than a required returned population or measured decoding workload.

References: [GRID](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html)

## Window B crosses four inner chunks across four shards

Move the window to rows [3, 6) and columns [3, 6). Window B still selects nine values at the same sample resolution. Along each axis, it uses one position from inner band [2, 4) and two from [4, 6).

The four global inner-grid coordinates are now (1, 1), (1, 2), (2, 1) and (2, 2). The boundary at coordinate 4 is also an outer-shard boundary, so each selected inner chunk belongs to a different shard. The table connects each global inner address to its parent and local index position.

The four complete inner chunks again cover 16 positions. Their four parent shards together span the entire 64-position array, but the requested return remains nine values. Changing the window's placement has changed shard intersection while leaving its size and the input resolution unchanged.

References: [GRID](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html), [ZS10](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html)

ZR-40 Window B: global inner addresses, parent shards and local inner addresses
| Global inner (row, column) | Parent shard (row, column) | Inner address within shard | Requested values | Source |
| --- | --- | --- | --- | --- |
| (1, 1) | (0, 0) | (1, 1) | 1 | Planning analysis |
| (1, 2) | (0, 1) | (1, 0) | 2 | Planning analysis |
| (2, 1) | (1, 0) | (0, 1) | 2 | Planning analysis |
| (2, 2) | (1, 1) | (0, 0) | 4 | Planning analysis |
All addresses are grid indices, not byte offsets. Contributions sum to nine requested values; the four inner chunks occupy four different shards.

## The index connects an inner address to encoded bytes

For Window B's sample at array coordinate (3, 4), global inner chunk (1, 2) lies in shard (0, 1). Its local inner address (1, 0) selects the shard-index entry.

Each entry stores an encoded byte offset and length as unsigned 64-bit integers. The separately encoded index can sit at either end of the shard, defaulting to the end.

A supported implementation can use the entry to locate that inner chunk's representation. Physical chunk order is not fixed, and encoded lengths may differ. Sample coordinates alone therefore cannot establish its byte range.

References: [ZS10](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html)

## An addressable range leaves the actual transfer unresolved

ZEP0002 describes both whole-shard reading and reading an index followed by selected inner ranges. The core specification also makes partial codec decoding optional. A store can fulfill a partial-value request by obtaining the whole value and returning a subset. An inner-range interface therefore does not establish how many bytes crossed the underlying storage connection.

For ZR-40, the encoding chain, index access, store, reader and cache state remain unspecified. A reader might already hold the needed shard or index, retrieve selected ranges, combine accesses or obtain whole shards. Additional codecs around sharding can also affect whether selective decoding is available through the complete path.

Consequently, Window A's one intersected shard versus Window B's four does not establish a fourfold request, latency or cost difference. Neither does reducing 16 data keys to four establish a fourfold transfer saving. Compare actual byte and timing evidence only for the same requested values, named implementation, cache conditions and measurement boundary.

References: [ZEP2](https://zarr.dev/zeps/accepted/ZEP0002.html), [ZA31](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)

## Choose the deliverable and supported reader together

The documentation read on October 20, 2026 labels the core specification 3.1 and the sharding codec 1.0. ZEP0002 was accepted on November 1, 2023. The array's zarr_format marker is the major version, 3; it is not a list of every supported minor-version feature or codec.

For a real offer, have the data or software provider identify the array's dimensions, scalar type, coordinate meaning and requested windows alongside both chunk shapes and the complete encoding chain. Reader compatibility needs sharding_indexed, its inner codecs and index codecs, plus any surrounding encoding. A generic “Zarr v3 supported” label leaves that combination unresolved.

Then distinguish a correct returned window from selective access to it. A supported reader that decodes a whole shard may supply the correct nine values. That result alone does not demonstrate the finer access behavior a buyer may require. If selective access matters to the consuming application, make its supported path and the provider's evidence part of the deliverable decision.

The related HDF5 guide addresses external source mappings; the GeoTIFF guide addresses image resolution; the OME-TIFF guide addresses plane identity. Here the sample grid stays fixed. The decision is whether the offered encoding and storage granularity suit the intended reader while preserving the requested values.

References: [ZA31](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html), [ZS10](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html), [ZEP2](https://zarr.dev/zeps/accepted/ZEP0002.html)

## Scope physical connections for the surrounding equipment

If the selected storage and compute equipment also needs physical connections, Leviathan's Structured Cabling & Fiber Installation service provides a route to discuss agreed routing, dressing, endpoint labels and connection records. Use the related project inquiry with a nonconfidential equipment, site and approved-endpoint outline, including material-supply responsibilities and the required physical handoff.

That scope concerns the equipment's physical cabling. Dataset encoding, Zarr reader integration and application acceptance remain with the appointed data and software providers.

References: [L1](https://www.leviathansystems.co/services/structured-cabling)

## Primary sources

- [ZA31 — Zarr core specification 3.1: Stability Policy, Array metadata, Chunk encoding and Abstract store interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html). Checked 2026-10-20. Observed core edition 3.1. Defines the major-version marker, codec chain, optional partial decoding and whole-value fallback for partial store access; it does not establish a particular client's feature support.
- [ZS10 — Zarr Sharding codec 1.0: Configuration parameters, Definitions and Binary shard format](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/index.html). Checked 2026-10-20. Defines inner shapes, nested and index codecs, encoded offsets and lengths, and index placement. ZR-40's geometry is an original hypothetical.
- [GRID — Zarr Regular chunk grid 1.0: Description](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html). Checked 2026-10-20. Defines origin-aligned regular grids and the relation between array coordinates, chunk indices and local positions. Both dimensions in ZR-40 divide exactly, so no edge overhang enters its accounting.
- [ZEP2 — ZEP0002: Sharding codec — Motivation, Backward Compatibility and Reading data per chunk](https://zarr.dev/zeps/accepted/ZEP0002.html). Checked 2026-10-20. Accepted November 1, 2023. Explains the separation of reading and storage granularity, required codec support and alternative access strategies. The current codec specification supplies the binary-layout rules used here.
- [L1 — Leviathan Systems: Structured Cabling & Fiber Installation](https://www.leviathansystems.co/services/structured-cabling). Checked 2026-10-20. Supports a separately agreed physical equipment-cabling discussion covering routing, dressing, identifiers and connection records.

## Related Leviathan resources

- [HDF5 logical arrays and external source mappings](/articles/scientific-ai-hdf5-virtual-dataset-source-values)
- [GeoTIFF sample resolution and storage tiles](/articles/geospatial-ai-cog-overview-resolution-window-tiles)
- [OME-TIFF plane coordinates and stored image identity](/articles/microscopy-ai-ome-tiff-plane-axis-mapping)
- [Structured Cabling & Fiber Installation](/services/structured-cabling)
- [Discuss physical cabling for the selected equipment](/contact?service=structured-cabling&from=%2Farticles%2Fzarr-inner-chunks-shard-object-granularity)

## Does the selected equipment need physical cabling?

Share a nonconfidential equipment, site, endpoint and material-responsibility outline to discuss a separate physical cabling scope and handoff.

[Discuss the scope](/contact?service=structured-cabling&from=/articles/zarr-inner-chunks-shard-object-granularity)