Skip to content

fix: prune statistics only through casts that keep order - #26982

Open
jayzhan211 wants to merge 2 commits into
apache:mainfrom
jayzhan211:fix-pruning-cast-order-rule
Open

jayzhan211 wants to merge 2 commits into
apache:mainfrom
jayzhan211:fix-pruning-cast-order-rule

Conversation

@jayzhan211

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Parquet statistics pruning can drop row groups that contain matching rows when the predicate casts the column. With i in (-1, 0, 1, 255, 256) and timestamps at 23:00, 00:00 and 01:00 across midnight, each in one row group:

Query In-memory table Parquet
SELECT count(*) FROM t WHERE CAST(i AS BOOLEAN) <= false 1 0
SELECT count(*) FROM t WHERE CAST(i AS BYTEA) < X'05' 3 0
SELECT count(*) FROM t WHERE CAST(ts AS TIME) < TIME '02:00:00' 2 0

Pruning casts the min/max statistics and compares them with the literal, which is only correct when the cast keeps the order of the values. Pruning decides that with its own check: any cast is allowed unless exactly one side is a string. So int → boolean, int → binary (little-endian bytes), timestamp → time of day and named-timezone timestamp → date all pass, although none of them keep order.

Sort elimination already asks the same question through is_order_preserving_cast, which rejects these casts. The two checks disagree in both directions: pruning accepts the casts above, and is_order_preserving_cast rejects casts that do keep order and that pruning relies on today (integer ↔ date/timestamp/duration, string and binary re-encodings, naive → named-timezone timestamps).

What changes are included in this PR?

  1. is_order_preserving_cast (now public in datafusion_physical_expr::expressions) also accepts the casts that keep order and that only pruning knew about:
    • string ↔ string and binary ↔ binary re-encodings (including FixedSizeBinary)
    • numbers ↔ temporal types stored as a count (date, timestamp, time, duration)
    • date → timestamp with a timezone, and naive timestamp → named-timezone timestamp. Arrow reads these as local times and rejects local times that a transition skips or repeats; the local times that remain keep their order in UTC.
  2. Pruning (verify_support_type_for_prune) asks is_order_preserving_cast instead of its own string check.
  3. A test that checks is_order_preserving_cast against the Arrow cast kernel: every pair it accepts must keep the order of sample values that convert (including values around midnight, a DST gap and overlap, and a timezone that moved the date backwards). A positive control asserts the samples expose the casts the rule rejects.

Pruning keeps every other cast it used before (compared over all castable pairs of the numeric, string, binary and temporal types) and stops using:

Cast Why
integer/float → boolean -1 and 1 are true, 0 is false
integer → binary little-endian bytes do not sort like the integers
timestamp → time wraps at midnight
named-timezone timestamp → date a transition can move the local date backwards
Time64 → Time32, Time64(µs) → Time64(ns) Arrow does not check these for overflow, so values can wrap (existing rule)
Null → any, any → list/struct, interval casts the rule has no entry for them

What is the testing strategy for this PR?

  • test_order_preserving_cast_matches_cast_kernel (physical-expr): kernel check of the rule, with a positive control. Making Timestamp(tz) → Date32 return true fails it with 562129259 -> 1987-10-25, but 562129260 -> 1987-10-24.
  • row_group_predicate_cast_must_preserve_order (pruning): casts that must not prune and casts that must keep pruning, for both CAST and TRY_CAST.
  • push_down_filter_parquet.slt: the three queries above against an in-memory table and Parquet, plus an EXPLAIN showing a naive timestamp compared with a TIMESTAMPTZ under a named session timezone still prunes.
  • cast.slt: an EXPLAIN showing the sort on arrow_cast(i, 'Timestamp(Second, None)') is removed when the input is ordered by i.
  • The new pruning unit test and the three Parquet queries fail with the old pruning check; the cast.slt case fails without the widened rule.

Are there any user-facing changes?

  • Queries that filter Parquet through the casts in the table above return the correct rows; those predicates no longer prune row groups.
  • Sort elimination now treats the newly accepted casts as order-preserving under CAST (not TRY_CAST), so a sort on, for example, CAST(int_col AS TIMESTAMP) over input ordered by int_col can be removed.
  • New public function datafusion_physical_expr::expressions::is_order_preserving_cast (addition only).

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Nov 1, 2026
// then we can compare them.
// PruningPredicate does not support casting of strings to numbers and such.
if from_type.is_string() == to_type.is_string() {
if phys_expr::is_order_preserving_cast(from_type, to_type) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Call is_order_preserving_cast

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet pruning drops rows for CAST(int AS BOOLEAN) and CAST(int AS BYTEA) predicates

1 participant