Skip to content

Push decimal aggregates through Widen - #11004

Draft
connortsui20 wants to merge 4 commits into
ct/decimal-widen-comparefrom
ct/decimal-widen-aggregates
Draft

connortsui20 wants to merge 4 commits into
ct/decimal-widen-comparefrom
ct/decimal-widen-aggregates

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Tracking Issue: #10701

Stacked on #11003. #11005 builds on it. No release gate.

Summary

Decimal widening preserves order, equality and validity, so ordered aggregates can use the child instead of copying it first. Sum also needs the logical result precision: using the child's sum dtype could overflow a result that should fit.

Changes

Ordered aggregates widen their child partials, and MinMax reuses exact cached child extrema before scanning. Decimal Sum runs in the logical sum dtype only when the legacy Stat::Sum path combines independent batch totals. SumV2 and Sum without skip_nans keep the fallback because their running sum can cancel a batch total that would overflow on its own. Tests compare running partials against fresh canonical batches so cached statistics cannot hide kernel work.

Historical local benchmark results

These retained measurements predate this style pass and rebase. They do not measure the current head.

cargo bench -p vortex-array --bench widen_decimal, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a Widen (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself.

bench p10 canonical p10 Widen p18 canonical p18 Widen p38 canonical p38 Widen
min_max 14.93 µs 15.12 µs 14.62 µs 15.12 µs 21.58 µs 14.95 µs

sum medians are within 8% of the canonical input at every precision.

API Changes

None.

@connortsui20
connortsui20 added this pull request to stack #11007 October 29, 2026 21:33
@connortsui20 connortsui20 changed the title ct/decimal widen aggregates Push decimal aggregates through Widen Oct 29, 2026
@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 3 benchmarks

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 9 improved benchmarks
❌ 3 regressed benchmarks
✅ 2348 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime threshold_neon[80] 4.5 µs 5.7 µs -21.94%
❌ CPU Simulation encode_primitives[u8, (2000, 4)] 134.3 µs 163 µs -17.6%
❌ CPU Simulation take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10] 20.5 µs 23.8 µs -14.09%
⚡ CPU Simulation min_max[p38_childp18_widen] 1,393.8 µs 436.9 µs ×3.2
⚡ CPU Simulation filter_powerlaw_by_random[10000] 1,013.5 ns 348.4 ns ×2.9
⚡ CPU Simulation sum[p38_childp18_widen] 1,221.6 µs 426.4 µs ×2.9
⚡ CPU Simulation sum[p10_childp5_widen] 563.5 µs 236.3 µs ×2.4
⚡ CPU Simulation sum[p18_childp9_widen] 563.5 µs 237.8 µs ×2.4
⚡ Walltime mul_u64_nonnull_neon 39 µs 20.5 µs +90.42%
⚡ CPU Simulation min_max[p18_childp9_widen] 614.5 µs 324 µs +89.65%
⚡ CPU Simulation min_max[p10_childp5_widen] 614.6 µs 326 µs +88.5%
⚡ Walltime lt_i64_nullable_avx512 8.2 µs 7.4 µs +11.96%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/decimal-widen-aggregates (5d992ff) with ct/decimal-widen-compare (e4a453e)

Open in CodSpeed

@connortsui20 connortsui20 mentioned this pull request Oct 30, 2026
11 of 32 tasks
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from 384deaa to 5d992ff Compare October 30, 2026 17:50
@connortsui20
connortsui20 removed this pull request from stack #11007 October 30, 2026 18:02
@connortsui20
connortsui20 added this pull request to stack #11058 October 30, 2026 18:04
Decimal widening keeps order, equality and validity, so MinMax, Min,
Max, IsConstant and IsSorted have the same partial on the narrower
child. Serve them from the child instead of widening every value. Sum
still declines, because a decimal sum checks overflow against the
precision of its input.

Signed-off-by: Connor Tsui <[email protected]>
The MinMax kernel ran a fresh accumulator on the values, which always
scans them. Call `min_max` instead, which returns exact cached Min and
Max first, so a child that records its extrema needs no scan. The
partial is a struct of the extrema cast to the non-nullable logical
dtype, or null without a valid value, which is what the canonical
accumulator reports.

Signed-off-by: Connor Tsui <[email protected]>
The Widen aggregate kernel declined every decimal Sum, so the default
path executed the Widen and widened every value before the sum. Serve
`Sum` with `skip_nans` from the values instead, with the sum dtype of
the logical dtype. A child accumulator would sum in the lower sum
precision of the child and could overflow where the canonical batch
does not.

The partial is exact only with the legacy `Stat::Sum` slot, which sums
each batch from zero and then folds it. The kernel then makes the same
checked adds as the canonical batch. Without the slot, a batch total
can fail the precision where the running sum does not, so `SumV2` and
`Sum` without `skip_nans` keep the default path. The kernel neither
reads nor caches `Stat::Sum` on the values, whose sum has another dtype.

Signed-off-by: Connor Tsui <[email protected]>
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from 5d992ff to 1af871e Compare October 30, 2026 19:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant