Repository navigation
Push decimal aggregates through Widen - #11004
Draft
connortsui20 wants to merge 4 commits into
Draft
connortsui20 wants to merge 4 commits into
connortsui20 wants to merge 4 commits into
Conversation
This was referenced Oct 29, 2026
This was referenced Oct 29, 2026
Merging this PR will regress 3 benchmarks1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
11 of 32 tasks
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
October 30, 2026 17:50
384deaa to
5d992ff
Compare
Decimal widening keeps order, equality and validity, so MinMax, Min, Max, IsConstant and IsSorted have the same partial on the narrower child. Serve them from the child instead of widening every value. Sum still declines, because a decimal sum checks overflow against the precision of its input. Signed-off-by: Connor Tsui <[email protected]>
The MinMax kernel ran a fresh accumulator on the values, which always scans them. Call `min_max` instead, which returns exact cached Min and Max first, so a child that records its extrema needs no scan. The partial is a struct of the extrema cast to the non-nullable logical dtype, or null without a valid value, which is what the canonical accumulator reports. Signed-off-by: Connor Tsui <[email protected]>
The Widen aggregate kernel declined every decimal Sum, so the default path executed the Widen and widened every value before the sum. Serve `Sum` with `skip_nans` from the values instead, with the sum dtype of the logical dtype. A child accumulator would sum in the lower sum precision of the child and could overflow where the canonical batch does not. The partial is exact only with the legacy `Stat::Sum` slot, which sums each batch from zero and then folds it. The kernel then makes the same checked adds as the canonical batch. Without the slot, a batch total can fail the precision where the running sum does not, so `SumV2` and `Sum` without `skip_nans` keep the default path. The kernel neither reads nor caches `Stat::Sum` on the values, whose sum has another dtype. Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
October 30, 2026 19:28
5d992ff to
1af871e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking Issue: #10701
Stacked on #11003. #11005 builds on it. No release gate.
Summary
Decimal widening preserves order, equality and validity, so ordered aggregates can use the child instead of copying it first. Sum also needs the logical result precision: using the child's sum dtype could overflow a result that should fit.
Changes
Ordered aggregates widen their child partials, and MinMax reuses exact cached child extrema before scanning. Decimal
Sumruns in the logical sum dtype only when the legacyStat::Sumpath combines independent batch totals.SumV2andSumwithoutskip_nanskeep the fallback because their running sum can cancel a batch total that would overflow on its own. Tests compare running partials against fresh canonical batches so cached statistics cannot hide kernel work.Historical local benchmark results
These retained measurements predate this style pass and rebase. They do not measure the current head.
cargo bench -p vortex-array --bench widen_decimal, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as aWiden(child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself.min_maxsummedians are within 8% of the canonical input at every precision.API Changes
None.