--- title: "Packed NetCDF Values: Keep Missing Samples Out of the Transform" description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20." canonical: "https://www.leviathansystems.co/articles/scientific-ai-netcdf-packed-values-missing-mask" last-updated: "2026-10-20" --- # Packed NetCDF Values: Keep Missing Samples Out of the Transform Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20. Before choosing a scientific-AI array offer, establish whether the recipient receives packed codes or already unpacked quantities, and how missing positions remain identifiable there. For CF-packed data, assess validity in the stored-value domain before applying scale_factor and then add_offset to valid values. A numeric array slot can represent missingness rather than a measurement; applying the transform does not turn it into one. ## Identify the values the recipient actually receives A scientific-AI input may arrive as a NetCDF variable, an extracted array or another supplied representation. The buyer's question is what the values mean at the point where the intended consumer receives them. The source file's representation and a later extracted array need not be the same stage. For a packed variable, a stored code is interpreted through the applicable scale and offset. An unpacked deliverable has already made that conversion. A units label alone does not distinguish them: the same physical units can describe the intended quantity while the file still stores codes. Nor does an integer dtype by itself establish that the supplied numbers await conversion. Ask the supplier to identify the delivered variable or array, its representation stage, applicable metadata and the intended consuming component and version. A statement that the original file contains the attributes leaves open whether those attributes still apply to the offered payload. This is a representation question before it is a question about the scientific use of the values. References: [CF81](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data) ## Resolve missingness before scale and offset This explanation uses the historical CF 1.12 conventions, dated December 4, 2024. Section 2.5.1 places missing values for transformed variables in the external, stored-value domain. It says applications should check validity first and should not transform values identified as missing. Section 8.1 specifies multiplication by scale_factor before adding add_offset. Relevant declarations can include _FillValue, missing_value and valid_min, valid_max or valid_range. A missingness mask means a record of which positions are missing, regardless of how the deliverable represents that record. A missing position is not automatically zero or an unusually low measurement. The distinction matters even when arithmetic on the marker produces an ordinary finite number. The result's numerical appearance cannot override the declaration that the original code represents missingness. The buyer needs that meaning to survive the conversion and delivery, rather than relying on a later plausibility check to rediscover it. References: [CF251](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#missing-data), [CF81](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data) ## CF-PACK-01: three codes, two measurements Consider fictional data variable CF-PACK-01, with signed-short stored codes 100, 200 and -32768 in that order. It explicitly declares a signed-short _FillValue of -32768, a double scale_factor of 0.1 and a double add_offset of 250.0. Its units K describe the unpacked quantities. The first two codes are stipulated valid; the third is the declared missing marker. This is an invented explanatory case: no file, library, climate observation, model or empirical test was used. For the first valid code, 100 × 0.1 + 250.0 = 260.0 K. For the second, 200 × 0.1 + 250.0 = 270.0 K. The third position remains missing. The representation therefore contains two valid measurements and one missing position, even though all three stored slots contain numeric codes. Incorrectly applying the arithmetic to the marker gives -32768 × 0.1 + 250.0 = -3026.8. There is no corresponding observation in this case. Reporting that result as a third temperature would invent a measurement from the missing marker. For the stipulated arithmetic mean of the two valid values, (260.0 + 270.0) / 2 = 265.0 K. The denominator is two because the case supplies two valid measurements. This calculation illustrates the population being counted; it recommends no spatial aggregation or weighting method. References: [CF251](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#missing-data), [CF81](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data) ## Keep the marker type separate from the quantity type The example's short _FillValue and double transform attributes serve different roles. CF 1.12 requires _FillValue to have the variable's type; for packed data, the listed validity attributes valid_min, valid_max and valid_range must also have the packed type. The conformance summary likewise requires missing_value, when present, to match its variable's type. Section 8.1's written packed-data rules require scale_factor and add_offset to have the unpacked float or double type, with matching types when both are present. Signed short is an allowed packed type for double unpacked data. Thus the example's matching double transform attributes and short marker are consistent; making every attribute short would not preserve those roles. Keep requirements distinct from recommendations. Section 8.1 says read data should be unpacked to the transform attributes' type, and units should describe the unpacked values. Section 2.5.1 recommends _FillValue when one missing value is needed and says it should lie outside a specified valid range. These statements do not establish the behavior of an unspecified reader. References: [CF251](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#missing-data), [CF81](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data), [CFR](https://cfconventions.org/Data/cf-documents/requirements-recommendations/conformance-1.12.html) ## Already unpacked values need a different handoff Now suppose an alternative deliverable already supplies CF-PACK-01's 260.0 K and 270.0 K, with the third position still identified as missing. Those first two values are quantities at the delivered stage. They are not the original 100 and 200 codes. If the recipient mistakes them for packed codes and applies the same transform again, 260 × 0.1 + 250.0 = 276 and 270 × 0.1 + 250.0 = 277. The numerical values changed because the delivered stage was misunderstood, not because the climate changed. This is hypothetical arithmetic, not a diagnosis of an actual evaluation failure or a claim that all readers decode twice. A supplied scale and offset therefore need an explicit relationship to the payload: do they still define a required conversion, or do they describe the source representation from which this output was obtained? Provenance can remain useful after conversion without making the same conversion applicable again. The supplier's explanation and intended consumer's interpretation must agree at that boundary. ## Compare complete raw-code and unpacked offers For CF-PACK-01, a raw-code offer can be useful if it supplies the three codes, their types and applicable transform and missing-value declarations to a consumer that interprets them. The recipient must be able to account for 100 becoming 260.0 K, 200 becoming 270.0 K and the third position remaining missing. Merely preserving the codes while losing their interpretation leaves this requirement unresolved. An unpacked offer can also be useful. It can supply the two quantities and an explicit missing third position, with K identified as their unit and the completed conversion made clear. This article prescribes no particular missing-data encoding. What matters for the comparison is that the intended consumer can distinguish the two measurements from the missing position without repeating the source transform. An output containing only the two numbers, without a supplied mapping back to the original positions, loses part of this case. It no longer identifies where the missing position belonged. Counting three slots as three supplied measurements fails for a different reason: it confuses array occupancy with measurement availability. Compare offers using an explanation for the same requested variable and subset at the actual delivery boundary. Have the data or software provider identify the received values, missingness representation, units and any conversion still to occur. Evidence about a different export stage does not settle what the offered payload supplies. Neither raw nor unpacked delivery is inherently the winner; each needs a complete interpretation for its intended consumer. ## A preserved mask leaves scientific questions open Preserving CF-PACK-01's missing position does not explain why it is missing. The case supplies no cause, geographic or temporal coverage, or scientific eligibility criterion. Missingness alone proves neither bad science nor suitability for the proposed AI task. Once the representation is understood, the appointed scientific and software owners still need to assess the actual source, coverage and intended use. The two-value mean does not recover an absent observation, and correct unpacking supplies no model-quality or forecasting guarantee. This article establishes no imputation, repair, filtering or scientific acceptance method. Other data decisions have their own evidence. The related Parquet guide examines which data a query must scan; the model-quantization guide examines complete inference memory. Those resource questions remain separate from whether a delivered scientific value is a packed code, an unpacked quantity or a missing position. ## Discuss installation of the customer-selected equipment If the wider scientific-AI project also requires a physical equipment installation, Leviathan's rack-and-stack service and the related project inquiry provide a route to discuss agreed mounting, physical IT cabling, labels and installation records. Share a nonconfidential equipment list, rack layout, connection requirements, site authorization and desired work window to discuss scope and availability. That inquiry concerns the separate physical installation of customer-selected equipment. Data preparation, reader behavior, scientific interpretation and AI software remain with the appointed providers. References: [L1](https://www.leviathansystems.co/services/rack-and-stack) ## Primary sources - [CF251 — CF 1.12, December 4, 2024: Section 2.5.1, Missing data, valid and actual range of data](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#missing-data). Checked 2026-10-20. Historical edition: stored-domain missingness and validity before transformation. Recommendations retain their stated force; no reader implementation is established. - [CF81 — CF 1.12, December 4, 2024: Section 8.1, Packed Data](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#packed-data). Checked 2026-10-20. Scale-before-offset interpretation, unpacked units and packed/unpacked attribute types. Supports the fictional arithmetic, without establishing a library default or scientific result. - [CFR — CF Conformance Requirements and Recommendations 1.12: Sections 2.5.1 and 8.1](https://cfconventions.org/Data/cf-documents/requirements-recommendations/conformance-1.12.html). Checked 2026-10-20. Concise type requirements and recommendations. The conventions document remains authoritative if the summary differs. - [L1 — Leviathan Systems: Rack-and-Stack & Colocation Installation](https://www.leviathansystems.co/services/rack-and-stack). Checked 2026-10-20. Agreed physical mounting, IT cabling, labels and installation records for a separate equipment installation. ## Related Leviathan resources - [Parquet projection and filtering: identify the scan work](/articles/ai-parquet-row-groups-projection-and-filter-scan) - [Quantized model weights and complete inference memory](/articles/quantized-model-weights-versus-inference-memory) - [Rack-and-stack installation scope](/services/rack-and-stack) - [Discuss physical installation of customer-selected equipment](/contact?service=rack-and-stack&from=%2Farticles%2Fscientific-ai-netcdf-packed-values-missing-mask) - [Industrial AI Data Feeds: Sampling, Queues and Missing Changes](/articles/industrial-ai-opcua-sampling-queue-publishing) ## Does the selected equipment need physical installation? Share a nonconfidential equipment, rack, site and IT connection outline to discuss a separate physical installation scope, handoff and availability. [Discuss the scope](/contact?service=rack-and-stack&from=/articles/scientific-ai-netcdf-packed-values-missing-mask)