Repository navigation
fix: prune statistics only through casts that keep order - #26982
Open
jayzhan211 wants to merge 2 commits into
Open
jayzhan211 wants to merge 2 commits into
jayzhan211 wants to merge 2 commits into
Conversation
jayzhan211
commented
Nov 1, 2026
| // then we can compare them. | ||
| // PruningPredicate does not support casting of strings to numbers and such. | ||
| if from_type.is_string() == to_type.is_string() { | ||
| if phys_expr::is_order_preserving_cast(from_type, to_type) { |
Contributor
Author
There was a problem hiding this comment.
Call is_order_preserving_cast
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Parquet statistics pruning can drop row groups that contain matching rows when the predicate casts the column. With
iin (-1, 0, 1, 255, 256) and timestamps at 23:00, 00:00 and 01:00 across midnight, each in one row group:SELECT count(*) FROM t WHERE CAST(i AS BOOLEAN) <= falseSELECT count(*) FROM t WHERE CAST(i AS BYTEA) < X'05'SELECT count(*) FROM t WHERE CAST(ts AS TIME) < TIME '02:00:00'Pruning casts the min/max statistics and compares them with the literal, which is only correct when the cast keeps the order of the values. Pruning decides that with its own check: any cast is allowed unless exactly one side is a string. So int → boolean, int → binary (little-endian bytes), timestamp → time of day and named-timezone timestamp → date all pass, although none of them keep order.
Sort elimination already asks the same question through
is_order_preserving_cast, which rejects these casts. The two checks disagree in both directions: pruning accepts the casts above, andis_order_preserving_castrejects casts that do keep order and that pruning relies on today (integer ↔ date/timestamp/duration, string and binary re-encodings, naive → named-timezone timestamps).What changes are included in this PR?
is_order_preserving_cast(now public indatafusion_physical_expr::expressions) also accepts the casts that keep order and that only pruning knew about:FixedSizeBinary)verify_support_type_for_prune) asksis_order_preserving_castinstead of its own string check.is_order_preserving_castagainst the Arrow cast kernel: every pair it accepts must keep the order of sample values that convert (including values around midnight, a DST gap and overlap, and a timezone that moved the date backwards). A positive control asserts the samples expose the casts the rule rejects.Pruning keeps every other cast it used before (compared over all castable pairs of the numeric, string, binary and temporal types) and stops using:
Time64→Time32,Time64(µs)→Time64(ns)Null→ any, any → list/struct, interval castsWhat is the testing strategy for this PR?
test_order_preserving_cast_matches_cast_kernel(physical-expr): kernel check of the rule, with a positive control. MakingTimestamp(tz) → Date32returntruefails it with562129259 -> 1987-10-25, but 562129260 -> 1987-10-24.row_group_predicate_cast_must_preserve_order(pruning): casts that must not prune and casts that must keep pruning, for bothCASTandTRY_CAST.push_down_filter_parquet.slt: the three queries above against an in-memory table and Parquet, plus anEXPLAINshowing a naive timestamp compared with aTIMESTAMPTZunder a named session timezone still prunes.cast.slt: anEXPLAINshowing the sort onarrow_cast(i, 'Timestamp(Second, None)')is removed when the input is ordered byi.cast.sltcase fails without the widened rule.Are there any user-facing changes?
CAST(notTRY_CAST), so a sort on, for example,CAST(int_col AS TIMESTAMP)over input ordered byint_colcan be removed.datafusion_physical_expr::expressions::is_order_preserving_cast(addition only).