--- title: "Arrow Dictionary Codes Are Not Stable AI Category IDs" description: "Leviathan Systems — Technical content Published 2026-10-21; updated 2026-10-21." canonical: "https://www.leviathansystems.co/articles/arrow-dictionary-indices-categorical-input-values" last-updated: "2026-10-21" --- # Arrow Dictionary Codes Are Not Stable AI Category IDs Leviathan Systems — Technical content Published 2026-10-21; updated 2026-10-21. An Arrow dictionary index identifies a position in its associated dictionary. The integer alone does not identify a category across independently encoded chunks or exchanges. Buy a data path that preserves decoded values or explicitly reconciles the dictionaries and indices before turning them into application category IDs. Equal schemas, row counts and integer buffers can otherwise conceal different input values. ## Identify the value behind the integer An AI dataset can arrive with a status or class column represented by small integers. Those integers may look ready for a model, an aggregate or a selection rule. First establish whether they are application-defined category IDs or positions used by the storage representation. The two roles can share the same integer type. Apache Arrow's Columnar Format 1.5, as documented with Arrow 19.0.1, describes dictionary encoding as indices plus a separate dictionary array. A value is recovered through that association. The representation can be useful for repeated values without assigning permanent application meaning to the positions. A buyer requesting the normal-category records needs the values interpreted correctly through the delivered path. A supplier returning the original integers has not answered that question merely by preserving their bytes. Conversely, different integers can be a correct return when the dictionaries change and the decoded values stay the same. This is a categorical-input question. The related Parquet guide owns file selection and scan work; the Iceberg guide owns the table state selected for input. Neither selection decides what a dictionary position means. References: [AD1](https://arrow.apache.org/docs/19.0/format/Columnar.html) ## AD-01: identical indices, different categories AD-01 is an original fictional exchange with eight records, A1 through A4 and B1 through B4. The IDs identify rows only. Its status column has no null values. Each four-row chunk uses the same integer sequence: 0, 1, 0, 2. The dictionaries contain the same three distinct strings in different orders. Chunk A's dictionary assigns 0 to normal, 1 to review and 2 to hold. Its decoded sequence is normal, review, normal, hold. Chunk B assigns 0 to review, 1 to normal and 2 to hold. Its sequence is review, normal, review, hold. Both chunks therefore have four rows, three dictionary entries and the same index sequence. Both can have the same dictionary index type and string value type. Those equalities leave the ordering of the associated dictionary values unresolved. The correct combined population is three normal records, three review records and two hold records. It contains no universal rule that index 0 means normal. This is stipulated data for reasoning, not a captured Arrow file, implementation test or model result. It describes two in-memory chunks; it does not claim that arbitrary dictionary replacements are permitted inside an Arrow IPC file. ## A code-only filter selects the wrong four records Suppose the recipient wants only status normal. Decoding against each chunk's own dictionary selects A1, A3 and B2: three records. A consumer that instead imports A's meaning of 0 and selects every 0 chooses A1, A3, B1 and B3. It returns four records, including two review records from B, and misses B2. There are two false inclusions and one missed intended row. The error comes from reusing a dictionary meaning, rather than from a changed row count or damaged integer buffer. The same mistake affects aggregation. Counting raw 0 positions produces four and counting raw 1 positions produces two. Relabeling these totals normal and review with A's dictionary gives4 and 2 instead of the correct 3 and 3. The total still sums to eight after including two hold positions. A conserved total therefore does not establish correct category assignment. For a buyer comparing input pipelines, ask the provider to explain the actual value sequence at the point the consumer receives it. An aggregate total or an index-buffer checksum is too narrow to settle AD-01's category question. The provider can demonstrate correct behavior without exposing confidential production records by using an agreed synthetic case. ## A shared dictionary requires corresponding index changes One valid conceptual return uses a shared dictionary ordered hold, normal, review. Its positions are0,1 and 2. To preserve the original values, A's indices become 1, 2, 1, 0; B's become 2, 1, 2, 0. Following those new indices through the shared dictionary reproduces both original decoded sequences. The return changes the integer buffers while retaining all eight status values. A value-level comparison passes this stipulated transformation; a requirement that every index byte remain unchanged would reject it for a different reason. Decide whether the purchase requires the original encoded representation, the logical values or both. Merely replacing B's dictionary with A's and leaving B's 0, 1, 0, 2 sequence intact is different. It would reinterpret B as normal, review, normal, hold, changing three of B's four values. Equal dictionary contents considered as an unordered set do not prevent that change. This is a consequence of which referenced positions changed meaning. Replacing a dictionary with an identical one would preserve the values, as would changing only entries that no valid row references. PyArrow 19.0.1 documents dictionary unification across a ChunkedArray as returning an equivalent array with shared dictionary values and transposed indices. That named behavior supports the distinction between reconciliation and dictionary replacement. It does not establish a specific unified ordering, memory cost or performance result for AD-01; the ordering above is our explicit example. References: [AD2](https://arrow.apache.org/docs/19.0/python/generated/pyarrow.ChunkedArray.html#pyarrow.ChunkedArray.unify_dictionaries) ## Serialization carries dictionary state as well as the schema The Arrow format also describes dictionary messages alongside record batches. Stream dictionaries can be extended by deltas or replaced. A later index must be interpreted with the applicable dictionary state. Arrow IPC files have a different constraint: replacement through multiple non-delta dictionaries for the same ID is not supported. A stream reader and a file reader therefore cannot be treated as the same dictionary-state case. The buyer needs the actual exchange form and supported consumer behavior. AD-01's independently supplied chunks do not waive either format's rules. If an offer supplies only a schema and integer arrays, have the supplying software party establish where the dictionary values come from and how they remain associated with the affected rows. A dictionary ID in one exchange is not, by itself, the organization's permanent category identity. The requested value interpretation must survive the actual transport and consumer boundary. This question concerns semantic preservation, not a demand to decode every column into strings in advance. A capable implementation can retain dictionary encoding and operate correctly with its associations. The required result is correct values for the recipient's task, rather than a prescribed intermediate layout. References: [AD1](https://arrow.apache.org/docs/19.0/format/Columnar.html) ## Model category IDs need their own mapping Now stipulate that the application consumes category IDs 10 for normal, 20 for review and 30 for hold. Those are fictional application assignments, not Arrow rules. Correct conversion produces 10, 20, 10, 30 for A and 20,10,20,30 for B. Passing the original 0, 1, 0, 2 vectors directly would not implement that application mapping. Passing the reconciled 1, 2, 1, 0 vector directly would not implement it either. Reconciliation makes the encoding consistent within the illustrated exchange; the application still needs its declared vocabulary. This matters when an offer describes data as model-ready. That phrase should identify the consumer's expected meanings, including unknown or missing categories and any supplied mapping version. A storage encoding that happens to produce integers has not established those application decisions. AD-01 deliberately excludes nulls. A real dictionary array also carries index validity. In the named Arrow format, a masked slot is null; its stored integer is not a usable category code. For a valid slot, the index is a non-negative, zero-based position in the associated dictionary, so it must address an existing entry. An out-of-range integer supplies no category value. PyArrow 19.0.1 documents the null mask and range checking in DictionaryArray.from_arrays; these are interpretation boundaries, not instructions to change reader options. The application owner still decides what missing or unknown input means for the consuming task. A masked slot must not silently become category 0, and an in-range position must not be treated as an application ID without the separate mapping. A valid reference to a null dictionary entry also supplies no named category; it is distinct from the index-validity metadata discussed next. References: [AD1](https://arrow.apache.org/docs/19.0/format/Columnar.html), [AD3](https://arrow.apache.org/docs/19.0/python/generated/pyarrow.DictionaryArray.html) ## Dictionary length is not a category-frequency result The Arrow format permits duplicate values and null entries in a dictionary. Its length therefore counts dictionary positions, not necessarily distinct logical categories. Even with unique entries, some positions may be unreferenced in the delivered rows. Observed frequencies depend on the applicable dictionary, valid indices and actual row population. Fictional AD-02 has dictionary positions 0 through 3 containing normal, normal, hold and null. Rows C1, C2 and C3 have valid indices 0, 1 and 2. C4 is masked null, so its stored index does not supply a status. The four delivered rows are normal, normal, hold and a masked null: normal occurs twice and hold once, with one missing row. Four dictionary entries have produced two distinct observed non-null category values, not four categories or four category observations. The duplicate normal entries merge at the logical-value level while remaining separate positions. The null entry at position 3 is unused in AD-02. Replacing that unused entry would leave these four row values unchanged, illustrating the qualification to dictionary replacement. If another valid row referenced the null entry, the reader would need to preserve that value state rather than invent a category label for it. Keep the array's reported null count separate too. The format defines a dictionary array's null count from the validity of its indices, irrespective of null entries inside the dictionary. Counting null dictionary entries is therefore not a way to obtain the index-null population. AD-02 has one masked index; its unused null dictionary entry does not add another masked row. For a model-input or reporting purchase, ask for the intended logical-value population and the declared treatment of missing values, not just a dictionary length or a count per integer. The supplier can preserve the encoding; it still needs to show that duplicates, unreferenced entries and validity are interpreted correctly at the consumer boundary. References: [AD1](https://arrow.apache.org/docs/19.0/format/Columnar.html), [AD3](https://arrow.apache.org/docs/19.0/python/generated/pyarrow.DictionaryArray.html) ## Compare offers at the consumer-facing value boundary An offer that preserves the original chunks with their dictionaries can satisfy a requirement for those exact encoded inputs if the consumer handles them. An offer that returns decoded strings can satisfy a logical-value requirement while changing the representation. An offer that reconciles dictionaries can retain encoding while changing index positions. None is automatically the fastest or smallest purchase. For AD-01, the essential value result is the eight-row sequence and its 3, 3, 2 category population. A normal-only request must select A1, A3, B2. An application-ID return must implement the separate 10, 20, 30 assignments. AD-02 adds the validity and logical-category population when dictionary length is insufficient. These are concrete outputs the parties can agree before comparing throughput or storage figures. Keep any performance comparison attached to the same request, input values, consumer and measurement boundary. Correctly preserving categories does not establish bytes transferred, memory consumption or elapsed time. Conversely, a rapid code-only filter returning the wrong four rows is not a faster implementation of the stated three-row request. The appointed data-platform and application providers own this software behavior. The related dataset-release and scan articles help establish the input state and the work performed once the category meanings are preserved. ## Bring a separate equipment connection need to the cabling scope If the wider AI project also needs physical connections between installed compute and storage equipment, Leviathan's Structured Cabling & Fiber Installation service provides a route to discuss that defined physical work. Bring the equipment and site summary, approved endpoints and routes, supplied assemblies and required installation records. That inquiry concerns agreed cabling, labels and handoff. It makes no offer of Arrow integration, category-vocabulary design, dataset validation or software operation. ## Primary sources - [AD1 — Apache Arrow 19.0.1: Columnar Format 1.5, dictionary layout and IPC dictionary messages](https://arrow.apache.org/docs/19.0/format/Columnar.html). Checked 2026-10-20. Historical Columnar Format 1.5 in Arrow 19.0.1 documentation: dictionary association, validity, duplicate/null dictionary values and index-based null counts, stream deltas/replacements and the file replacement constraint. This named edition supplies representation semantics rather than a current product-support claim. - [AD2 — PyArrow 19.0.1: ChunkedArray.unify_dictionaries](https://arrow.apache.org/docs/19.0/python/generated/pyarrow.ChunkedArray.html#pyarrow.ChunkedArray.unify_dictionaries). Checked 2026-10-20. Equivalent chunked array with shared dictionary values and transposed indices; no claimed ordering or measured cost. - [AD3 — PyArrow 19.0.1: DictionaryArray](https://arrow.apache.org/docs/19.0/python/generated/pyarrow.DictionaryArray.html). Checked 2026-10-20. PyArrow 19.0.1 meanings for zero-based dictionary indices, null masks, in-range references and decoded arrays. The distinct from_pandas null convention is not generalized to all dictionary arrays. ## Related Leviathan resources - [Parquet selection, predicate work and scanned populations](/articles/ai-parquet-row-groups-projection-and-filter-scan) - [Iceberg table snapshots and dataset file membership](/articles/iceberg-table-snapshot-versus-object-prefix-input) - [Coherent S3 dataset releases](/articles/ai-dataset-s3-consistency-versus-complete-release) - [Structured Cabling & Fiber Installation](/services/structured-cabling) - [Discuss defined physical cabling work](/contact?service=structured-cabling&from=%2Farticles%2Farrow-dictionary-indices-categorical-input-values) ## Discuss a separate physical storage connection project Share the selected equipment, approved endpoints, site constraints and required installation records for an engagement-defined physical cabling discussion. [Discuss the scope](/contact?service=structured-cabling&from=/articles/arrow-dictionary-indices-categorical-input-values)