[connortsui20]

Tracking Issue: #10701

Stacked on #11005, the layer of the decimal Widen stack that adds from_stored_decimal. It does not need #11006.

Summary

Older writers stored a decimal dtype at any width that held its values, so old files contain vortex.decimal nodes at a width other than the canonical width of their dtype. Once every DecimalArray must have the canonical width, these nodes can't read back as stored, so they have to convert on read. They convert through WidenArray::from_stored_decimal, the same path that DecimalByteParts decoding uses (#10905), so every kind of old decimal data reads as the canonical array or a decimal Widen.

Do not merge until 0.91.0 publishes the CD2 compat fixtures. Then run uv run vortex-test/compat-gen/scripts/compat.py check --mode last on this branch. The aim is to ship this in the same release as #10905 and #10992.

Changes

DecimalPlugin takes over the vortex.decimal ID in the default session. A node at the canonical width reads as before, with no copy and no scan, and its buffer stays on the device. Any other node reads into the canonical array or a Widen over the smallest precision that holds its valid values. The plugin can't use stored statistics (the deserializer attaches them after the plugin returns), so an old node costs one scan of its values.

Old nodes are rare, since the compressor has put every decimal that fits in 64 bits into DecimalByteParts since 2025-05. So instead of adding a device path (CUDA has no Widen kernel), the plugin copies the buffers and validity of an old node to the host. The new crate-private ArrayRef::try_into_host_sync copies the validity tree, and a failed copy returns an error. Open question for a CUDA reviewer: does a GPU scan over such a column still work?

Decimal::deserialize now rejects a non-canonical width. Both paths decode through a shared StoredDecimal, which checks the buffer length, alignment and validity against the dtype, so a malformed node returns an error instead of panicking.

Follow-up: in Python, an old narrow node now reads as a decimal Widen, which has no Python subclass, so it isn't a DecimalArray there.

API Changes

Reading an old vortex.decimal node returns the canonical array or a decimal Widen instead of a DecimalArray at the stored width. Adds vortex_array::arrays::decimal::DecimalPlugin, which the default ArraySession registers in place of Decimal. A session that registers only Decimal rejects old nodes.

[codspeed]

Merging this PR will regress 2 benchmarks

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 12 improved benchmarks
❌ 2 regressed benchmarks
✅ 2346 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	missing_stats[16]	67.9 µs	85.9 µs	-21.02%
❌	CPU Simulation	random_i8[0.01]	98.8 µs	112.2 µs	-11.99%
⚡	Walltime	mul_u64_nonnull_neon	39.7 µs	20.4 µs	+94.58%
⚡	Walltime	filter_neon[64]	2.9 µs	1.9 µs	+53.95%
⚡	Walltime	threshold_neon[64]	3.7 µs	2.6 µs	+39.69%
⚡	Walltime	filter_neon[80]	3.7 µs	2.7 µs	+36.81%
⚡	CPU Simulation	filter_powerlaw_by_random[10000]	700.5 ns	513.4 ns	+36.44%
⚡	CPU Simulation	bench_compare_varbin[(50000, 64)]	163.6 µs	133.9 µs	+22.19%
⚡	CPU Simulation	bench_compare_varbin[(50000, 2)]	161.4 µs	132.2 µs	+22.09%
⚡	Walltime	threshold_neon[80]	4.3 µs	3.5 µs	+21.93%
⚡	CPU Simulation	bench_compare_sliced_dict_varbinview[(7500, 10000)]	227.1 µs	199 µs	+14.11%
⚡	CPU Simulation	bench_compare_sliced_dict_varbinview[(10000, 10000)]	233.3 µs	205.2 µs	+13.71%
⚡	CPU Simulation	bench_compare_sliced_dict_varbinview[(1000, 10000)]	89.2 µs	78.5 µs	+13.56%
⚡	CPU Simulation	take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10]	24.3 µs	21.4 µs	+13.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/decimal-read-plugin (3cfb191) with ct/decimal-widen-stored (eb6bfc5)}