Skip to content

Export decimal Widen arrays to Arrow with one copy - #10994

Draft
connortsui20 wants to merge 2 commits into
ct/decimal-widenfrom
ct/widen-arrow-export
Draft

connortsui20 wants to merge 2 commits into
ct/decimal-widenfrom
ct/widen-arrow-export

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Tracking Issue: #10701

Stacked on #10946. It needs only decimal children in Widen, so it can merge before the rest of the decimal Widen stack, and it has no release gate.

Summary

Arrow exports every decimal up to precision 38 as Decimal128, but the exporter first executed its input to the canonical DecimalArray. A decimal Widen at precision 18 or less therefore copied its values twice, once into the canonical i64 width and again into i128. After #10905, every DecimalByteParts column with a narrow MSP decodes to a Widen, so this is the common export path for decimals that DataFusion reads from a file.

With this change, exporting widen_i32_p18 costs the same as exporting the canonical array (27.12 µs, down from 42.06 µs).

Changes

The exporter stops at a Widen and converts its child straight into the Arrow width, with the precision and scale of the logical dtype. The converters take the logical dtype separately from the values, so the export never builds a DecimalArray whose dtype is wider than its values, which CD8a will forbid. Struct, list, dictionary and run-end children reach the same exporter. Above precision 18 the canonical width is already i128, so those exports don't change.

When a child is stored wider than the Arrow width, the export narrows it. A null row whose value didn't fit used to fail the whole export. Narrowing now skips null rows and writes zero into them, as copy_to_canonical_width does, which also fixes the same failure for a plain DecimalArray.

The tests check that a Widen export equals the canonical export at the largest precision of each child width, for every Arrow decimal type the parent precision fits, with garbage in the null rows.

Benchmark results (Apple M4, medians)

cargo bench -p vortex-arrow --bench to_arrow -- decimal_export, 100,000 rows exported to Decimal128. Before is the benchmark commit of this PR and after is its tip, so the comparison isolates this layer. Other jobs shared the machine during the runs.

case before after
widen_i32_p18 42.06 µs 27.12 µs
widen_i64_p38 31.16 µs 27.16 µs
canonical_i64_p18 28.24 µs 27.04 µs

widen_i32_p18 is the case this PR changes. widen_i64_p38 and canonical_i64_p18 copy once both before and after, so they are controls.

API Changes

No public API changes. Exporting a decimal stored wider than the Arrow width no longer fails on a null row whose value does not fit.

@connortsui20 connortsui20 added the changelog/performance A performance improvement label Oct 29, 2026
@connortsui20 connortsui20 changed the title ct/widen arrow export Export decimal Widen arrays to Arrow with one copy Oct 29, 2026
@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 2360 untouched benchmarks


Comparing ct/widen-arrow-export (dacf961) with ct/decimal-widen (fdf9ac2)

Open in CodSpeed

A decimal Widen at precision 18 or less exports as Decimal128, and
today the exporter copies its child to the canonical width before it
copies again to the Arrow width. Add a Widen case under precision 18,
where that double copy happens, and one under precision 38, where the
canonical width already equals the Arrow width, so the next change can
be measured against both.

Signed-off-by: Connor Tsui <[email protected]>
Arrow exports every precision up to 38 as Decimal128, and the decimal
exporter executed its input to the canonical DecimalArray first. A
decimal Widen at precision 18 or less therefore copied its child twice:
once into the canonical width, and again into the Arrow width.

Stop at a Widen and export its child with the precision and scale of
the logical dtype, so each value is copied once. The converters take
that dtype apart from the values, so no DecimalArray ever pairs it with
the narrower child width. Struct, list, dictionary and run-end children
reach this exporter too, so they get the single copy as well.

The converters narrowed every row, so a null row whose value does not
fit in the Arrow width failed the export. Executing the Widen hid this,
because the canonical copy skips null rows. Narrow only the valid rows
and write zero into the null rows, like the canonical copy does. This
also fixes the same failure for a plain DecimalArray stored wider than
its canonical width.

Signed-off-by: Connor Tsui <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant