A Small Dataset Result Does Not Specify a Small Parquet Scan

Follow the dataset, file and column boundaries

A dataset can span several Parquet files. Within a file, row groups divide the rows logically, and each group contains a column chunk for each column. An individual chunk is contiguous in the file; the row group has no guaranteed physical structure. Chunks contain pages, the units of encoding and compression described by Apache Parquet's Concepts page, dated March 8, 2024.

The File Format documentation explains that metadata locates column chunks. This makes selected column data addressable within a file without making an entire row group one indivisible physical read.

Use five stages to understand an offer: dataset file selection, row-group exclusion, remaining predicate evaluation, output construction and measured byte/time evidence. These are explanatory stages, not a promised execution order. The examples below use a conceptual reader that exploits correct partition information when supplied and valid ID minimum/maximum bounds, then evaluates the remaining candidate IDs. No page index or other elimination mechanism is supplied; the examples are not observed executions of Arrow or another product.

References: [PQ1][PQ2]

PS-223D: partition information first excludes one whole file

Apache Arrow 20.0.0's Dataset guide shows partition fields supplied by a recognized directory scheme or explicit partition expressions. Such fields can be absent from the Parquet files and still participate in dataset filtering. A nonmatching partition can exclude a whole file. The dataset's file fragments and the row-group fragments within a file are distinct.

Consider fictional extension PS-223D, separate from the original one-file PS-223 below. Its dataset contains two files. Each holds exactly one row for each integer ID from 1 through 200 and six stored scalar columns: ID, feature, label, aux_a, aux_b and aux_c. Each has four 50-row groups with consecutive ID bounds. Correct, known partition expressions assign every row of one file day 19 and every row of the other day 20. The virtual day field is supplied by the dataset; it is not a seventh stored column.

The request returns feature and label where day equals 20 AND ID is from 110 through 119, inclusive. The reader can exclude the day 19 file because none of its rows satisfies the day condition. Of 400 dataset rows, only the 200 in the day 20 file remain for the within-file reasoning that follows.

This exclusion depends on the stipulated reader and reliable partition information. A directory label alone does not establish that premise. Missing or untrusted partition information removes this particular justification for exclusion; it does not prove that a physical reader fetched all 400 rows.

References: [AR1]

PS-223: row-group bounds leave fifty candidate IDs

Original fictional example PS-223 contains one 200-row file with the same six stored columns and exactly one row for each integer ID from 1 through 200. Its four groups each contain 50 consecutive IDs: 1 through 50, 51 through 100, 101 through 150 and 151 through 200. Valid ID minimum and maximum statistics describe each group. The request returns feature and label for ID 110 through 119, inclusive.

A group can be excluded when its maximum is below 110 or its minimum is above 119. The first two groups have maxima of 50 and 100; the fourth has a minimum of 151. All three can be excluded by this bounds test.

The group spanning 101 through 150 remains. Its interval overlaps the requested interval, leaving 50 candidate IDs for value evaluation. Ten match: 119 - 110 + 1 = 10. The result has 10 of 200 rows, or 5%; the candidate population has 50 of 200 rows, or 25%. Neither percentage describes bytes.

Apply this same reasoning to PS-223D's selected day 20 file. Partition information reduced the dataset population from 400 rows to 200; group bounds then reduce it to 50 candidate IDs. The two exclusions use different information at different boundaries.

Bounds establish that excluded groups cannot match. Overlap leaves a group eligible; it does not identify the exact matching row positions inside it. Neither example supplies a rule mapping an ID directly to a physical position.

Evaluate ID before deciding which output rows belong

Both requests return only feature and label, but the remaining ID values establish which rows satisfy the ID condition. The logical stored inputs therefore include ID, feature and label. The three auxiliary columns have no role in these requests.

Projection selects output fields or expressions. Arrow 20.0.0's Scanner reference describes passing column requirements to fragments to avoid unnecessary column work. Its filter description distinguishes metadata-based exclusion, when possible, from filtering loaded record batches. A correct ten-row result alone cannot show how much nonmatching data was skipped.

For the stipulated reader, evaluating the 50 candidate IDs selects ten rows. In PS-223D the day condition is already satisfied by the selected file's partition expression, so no stored day values are needed. The complete population trace is 400 dataset rows, 200 rows in one selected file, 50 candidate IDs and ten returned feature-and-label rows.

Those dependencies do not prescribe when feature and label are fetched or how many of their values are decoded. An implementation may handle output values differently after locating matches; these examples establish no fixed read or decode order.

References: [AR2]

Interleaved IDs leave the same result and more candidates

Now change only how original PS-223's 200 rows are assigned to its four groups. Each still has 50 rows. Group one contains IDs 1, 5, 9 and so on through 197; group two contains 2, 6, 10 through 198; group three contains 3, 7, 11 through 199; and group four contains 4, 8, 12 through 200.

Their valid bounds are 1 through 197, 2 through 198, 3 through 199 and 4 through 200. Every interval overlaps 110 through 119. The same bounds-only test excludes no group, leaving all 200 IDs as candidates for the reader's value evaluation. Exactly ten still match.

If PS-223D's day 20 file used this interleaved arrangement, correct partition information would still exclude the day 19 file. Within the selected file, however, the wider bounds would leave 200 candidate IDs rather than 50. File exclusion and group exclusion do not replace each other.

This comparison explains a layout effect without prescribing a sort order or row-group setting. Identical result counts and stored-column counts can coexist with different candidate populations.

Finer page exclusion needs its own support

Parquet defines optional page-index metadata for column chunks. ColumnIndex supports locating candidate data pages from value bounds; OffsetIndex supports navigation to corresponding row ranges in other columns. Page selection can therefore be finer than the group-level test.

That possibility requires relevant metadata in the file and support in the selected reader. Candidate-page bounds need not identify every matching value. Neither worked example supplies a page index, so its potential benefits do not change their candidate counts. The cited Arrow 20.0.0 passages describe conditional metadata pushdown, not universal use of every optional Parquet feature.

References: [PQ3][AR2]

Keep population counts separate from bytes and time

Neither example supplies encoded column sizes, compressed page sizes, metadata sizes or actual I/O behavior. Parquet supports different value encodings, including plain and dictionary representations, and compression of data and dictionary pages. Equal row counts or scalar-column counts do not establish equal stored sizes.

In PS-223, ten returned rows do not establish a one-twentieth-file scan. Retaining one of four equally populated groups does not establish one quarter of the bytes. Needing three of six stored columns does not establish a 3/6 byte share. Multiplying those fractions invents a uniform byte layout.

In PS-223D, excluding one of two equally populated files likewise does not establish half the transferred bytes. Arrow's Dataset guide also identifies filesystem and metadata overhead per file and discovery costs associated with partitioned directories. Successful exclusion does not make that surrounding work free.

A byte claim needs a stated counting boundary and evidence there. Reader-facing file bytes and backing-device transfers can differ: Linux's page cache can retain file data to avoid subsequent disk reads. The related page-cache boundary guide explains that specific distinction. It does not change which IDs satisfy the predicate.

An elapsed-time claim additionally needs a defined interval and matching execution conditions, including whether discovery and metadata work are included. None of the population counts supplies that duration. The related IOPS, throughput and latency guide explains how to interpret measured rates once their byte and clock boundaries are known.

References: [PQ4][PQ5][AR1][K1]

Ask two questions when comparing access offers

First, what trusted information excluded which populations? For the same dataset release, filter and output fields, have the software provider identify the partition expressions, usable group bounds or indexes, and the reader and version that exploit them. Follow each claimed exclusion to the file, group or page population it removes, then identify the values still needed for the predicate and output.

Suppose offer A explains PS-223D's correct day assignments, consecutive group bounds and the stipulated reader. It accounts for 400 rows becoming 200, then 50 candidates and ten results. Offer B supplies only the same ten-row result. A explains the exclusions; B leaves them unresolved. That is a difference in supporting evidence, without yet establishing a performance winner.

Second, what byte and time evidence supports this same request? Require the actual byte boundary, covered interval, cache conditions and implementation behind a stated result. An explanation of possible pruning cannot establish a measured transfer total, and an observed total for another request cannot fill the gap.

Establish dataset identity before making that comparison. The related S3 consistency and complete-release article explains why readable objects can still belong to different generations. Once the intended release is identified, this article's question is how its partition information, within-file layout and reader determine exclusion and remaining predicate work.

Connect a separate storage-network cabling requirement

If the selected system also needs physical storage-network connections, Leviathan's Structured Cabling & Fiber Installation service is the route to discuss agreed routing, dressing, labeling and connection records. Use the related project inquiry with a nonconfidential equipment and site summary, approved endpoints, material-supply responsibilities and required physical handoff.

Parquet layout, reader selection, dataset changes and performance evaluation remain with the appointed software and storage providers. The cabling inquiry concerns the physical connections required by the selected equipment.

References: [L1]