[joseph-isaacs]

Summary

Use V2 with one shared io_uring for the normal local DataFusion TPC-H, TPC-DS, ClickBench, and FineWeb benchmarks. Results appear in the existing PR-versus-develop benchmark tables and verdict.

Add a completion-driven positional read API and io_uring backends for V2 scans. Existing future-based readers remain usable through an adapter. The library backend is opt-in; one or two shared rings and worker-owned rings remain available for comparison.

This branch includes develop at 96fc43dfc6, including async-channel and the orchestrator's explicit per-benchmark scan selection. It also includes prerequisite sparse/suffix-read work from #10668.

Selected configuration

• One shared reactor, OS-page-cache-backed mappings for resident reads, and buffered io_uring READ for misses.
• In-flight read reuse, a resident-page recheck before submission, and FIFO coalescing with a 256-request lookahead, zero gap, and a 1 MiB merged-read cap.
• Queue depth 64, asynchronous kernel dispatch, 128K-row work chunks, shared plans, and consumer-side Arrow conversion.
• Standard worker affinity, with no reserved or pinned I/O core, direct I/O, SQPOLL, or additional copied byte cache.

SQL CI builds with datafusion-bench/io-uring. Other benchmarks retain develop's scan selection; Parquet, DuckDB, and remote object-store readers retain their respective backends.

Implementation

• Add VortexReadAt2, completion scopes, polling/blocking completion APIs, cancellation, a future-reader adapter, and an IoService adapter.
• Drain ready resident completions without dispatching queued cache misses, preserving batching opportunities.
• Integrate completion services with prepared V2 scans. Each admitted root owns an independent I/O scope; worker-owned scopes are created and driven on their owning worker.
• Retain encoded segment arrays until overlapping row consumers retire. Concurrent readers share initialization; partial reads and filtered scans preserve reuse.
• Skip whole-segment retention for unfiltered reads that cannot cross chunk or projection-morsel boundaries. Store the policy in the shared allocation so cache handles remain pointer-sized.
• Provide preloaded-memory baselines and diagnostics for runtime, CPU, logical reads, kernel operations, SSD bytes, cold-page eviction, and output equality.
• Add explicit DataFusion memory and spill budgets for large local studies, and a data-gen conversion-only mode that reuses prepared Parquet.

Current validation

Head 7bb41d86c8 adds optional fair memory reservations, spill directory/size limits, and conversion-only data preparation. Optimized builds of both benchmark binaries, CLI rejection cases, conversion of only an existing input, pinned formatting, and whitespace checks pass. Earlier verification of the unchanged I/O/layout implementation includes 621 serial I/O tests, 834 layout tests, strict Clippy across the affected crates, workflow YAML lint, and the V1 reader-selection regression test. All branch commits have matching human DCO sign-offs.

CPU bench-all for this head is running. The previous CPU bench-all succeeded at ddcaab2ee4. Normal automatic CI jobs are skipped on draft PRs by develop's workflow policy; those skipped jobs are not reported as passed.

The ongoing SF10/SF100 study uses the same frozen ddcaab2ee4 binary for V1, V2 futures, and one/two shared rings, with 32 workers, 128K-row chunks, verified cold eviction, output validation, and a 48 GiB process cap. Initial normal-format cold results cover 21 of 22 TPC-H SF100 queries and 27 of 99 TPC-DS SF100 queries in all four configurations. One-ring/V1 geometric-mean ratios are 0.910 and 0.900 respectively; two-ring ratios are 0.917 and 0.900. These are partial-suite results. The default TPC-H Q18 plan reaches the memory guard in every mode and is excluded from those ratios. Descriptor-limit failures are retried with the same 65,535-descriptor limit for all modes. Memory-bounded alternative query plans are recorded separately.

The bounded TPC-H SF100 Q18 comparison completed with equal outputs in all four modes: 23.08–23.15 seconds and 14.2–15.8 GiB peak RSS, using eight workers, sort-merge joins, a 16 GiB fair memory pool, and LZ4-compressed spill files. This is a different plan from the default hash-join case above. A separate TPC-DS SF100 Q19 repeat used 12 cold iterations in each of two run orders: one ring measured 239.3–239.8 ms, two rings 242.9–244.6 ms, and V1 304.3–313.6 ms, with equal outputs. The initial apparent two-ring advantage did not repeat. Both Vortex formats are now prepared for both SF100 suites, and the full matrix is continuing.

Matched local builds show the single-use segment optimization reducing the 10,000-column diagnostic scan from 117.31 to 111.37 ms (5.1%). All four reversed-order pairs improved by 4.4–5.2%. The same frozen builds completed 32 focused SQL processes with 28 exact output comparisons and verified input hashes. ClickBench Q5 hot was within 0.2% of baseline with one ring in both orders; two rings was slightly faster. Cold FineWeb Q6 showed no repeatable slowdown. The wider ten-case follow-up also completed, bringing the combined SQL study to 192 process runs and 168 exact output comparisons. No case slowed by more than 3% in both run orders.

Those performance measurements use frozen study builds from before the develop merge and shape-matched synthetic wide-table data, not the exact CI RNG stream. They do not establish merged-head or all-query performance. Current CI comparisons follow develop's per-benchmark scan selection, which now uses V2 for the four suites above; the historical local comparisons below explicitly use V1.

Historical local measurements

These results are from frozen binaries before the integration with current develop. They do not establish performance of this updated head; the normal CI report compares this PR's candidate against the recorded develop baseline.

All 294 queries completed hot and cold: TPC-H SF1/SF10, TPC-DS SF1/SF10, ClickBench, and FineWeb. The configuration uses 32 workers/partitions, mapped OS-cache reads, and 32K-row work chunks except the labeled 128K variant. Ratios are geometric means of per-query median runtime ratios; values below 1 favor V2.

Mode	Hot V2/V1	Cold V2/V1	Regressions over 5%, hot/cold
Existing V2 control	1.029×	0.968×	88 / 42
Per-core rings, 32K	0.947×	1.046×	52 / 107
Per-core rings, 128K	0.940×	0.969×	40 / 57
One shared ring	0.871×	0.932×	7 / 17
Two shared rings	0.869×	0.928×	7 / 19
Preloaded memory, runtime workers	0.875×	—	6 / —

One shared ring beats V1 on 275 hot and 243 cold queries; two rings on 277 and 239. Summed hot medians are 94.33s for V1, 92.87s for one ring, and 92.35s for two. Summed cold medians are 106.51s, 105.76s, and 106.43s respectively. TPC-DS SF10 Q72 dominates these totals. The objective of beating V1 on every query is not achieved.

The subsequent pinned-reactor pilot completed 340 equal-result comparisons. Its hot results were close to shared-ring results; its cold geometric means were about 2–5% slower than one shared ring across the pilot groups. It does not displace the selected candidate. Observed medians are not confidence intervals.