Repository navigation
Keep dictionary columns encoded on write, selected by field path - #10814
Open
joseph-isaacs wants to merge 12 commits into
Open
joseph-isaacs wants to merge 12 commits into
joseph-isaacs wants to merge 12 commits into
Conversation
Merging this PR will improve performance by 18.61%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing 
|
joseph-isaacs
added
changelog/performance
Claude
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (59)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.854x ✅, 5↑ 0↓)
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 1↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.976x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.125x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.138x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.033x ➖, 0↑ 1↓)
|
joseph-isaacs
force-pushed
the
ji/hopeful-bohr-l12an2
branch
from
October 29, 2026 16:51
c5ab0bf to
a40a081
Compare
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
joseph-isaacs
force-pushed
the
ji/hopeful-bohr-l12an2
branch
from
October 29, 2026 17:08
a40a081 to
412f069
Compare
Add a way to hand the compressor and writer a set of encodings to keep. The compressor compresses only a preserved array's child slots, and repartitioning merges dictionary blocks, including dictionaries nested in structs and lists, into one compacted dictionary instead of canonicalizing them. `DictStrategy` passes input that is already dictionary-encoded to its fallback rather than re-encoding it. The benchmark conversion keeps Arrow dictionary columns encoded when the Parquet schema has them (PolarSignals), so scans export the stored dictionaries to Arrow instead of hashing every value into a new one. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS Signed-off-by: Joe Isaacs <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS Signed-off-by: Joe Isaacs <[email protected]>
Replace the compressor-wide `preserved_encodings` with a per-column writer selected by field path. `WriteStrategyBuilder::with_dict_field(path)` writes that column without dictionary encoding: repartitioning merges its dictionary chunks into one compacted dictionary and `DictChildrenCompressor` compresses only the codes and values. The compressor and `DictStrategy` are unchanged from develop. `TableStrategy` now routes overrides into list elements through `Field::ElementType`, so a path like `locations.[].mapping_file` reaches its column. The repartition above a list keeps nested dictionaries so they survive until their field writer. The benchmark conversion collects the paths of the Arrow dictionary fields and passes them as dict fields. Signed-off-by: Joe Isaacs <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS
joseph-isaacs
force-pushed
the
ji/hopeful-bohr-l12an2
branch
from
October 29, 2026 17:51
412f069 to
f311cea
Compare
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
changed the title
Replace `with_dict_field(path)` with `with_allow_encodings(path, encodings)`: the field keeps arrays of the given encodings it arrives with, and only their child slots are compressed. `AllowedEncodingsCompressor` walks struct fields and list elements to the allowed paths, and repartitioning keeps allowed encodings, merging dictionary blocks. Without the list layout, which the default editions leave out, list elements are not written as columns of their own, so a path into a list is now kept by the writer of the list column that holds it. This is what the PolarSignals `locations` columns need in the benchmark session. The benchmark conversion allows `Dict` at each Arrow dictionary field path. Signed-off-by: Joe Isaacs <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
added
the
action/bench-all
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.022x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.979x ➖, 1↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.831x ➖, 6↑ 1↓)
datafusion / vortex-compact / ns (0.860x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.940x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.942x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.835x ➖, 4↑ 0↓)
|
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.988x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.980x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.977x ➖, 2↑ 1↓)
random-access / parquet / ns (1.042x ➖, 1↑ 2↓)
random-access / lance / ns (1.019x ➖, 0↑ 1↓)
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.007x ➖, 0↑ 0↓)
random-access / vortex-file-compressed / ns (0.960x ➖, 0↑ 0↓)
random-access / parquet / ns (0.995x ➖, 0↑ 0↓)
random-access / lance / ns (0.978x ➖, 0↑ 0↓)
random-access / litdata / ns (1.002x ➖, 0↑ 0↓)
|
…ding it" This reverts commit 263a0b1. Keeping dictionaries nested in list elements that way needs the list layout, which only the unstable edition permits, so the PolarSignals conversion failed with "Layout encoding vortex.list not permitted by ctx". The previous version keeps them in the compressor of the list column instead, and needs no layout outside the default editions. Signed-off-by: Joe Isaacs <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
joseph-isaacs
marked this pull request as ready for review
October 30, 2026 15:25
mhk197
added
the
action/bench-sql-extended
github-actions
Bot
removed
the
action/bench-sql-extended
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
Signed-off-by: Joe Isaacs <[email protected]>
Under the list layout, list element columns are read back as chunked arrays of per-chunk dictionaries. to_arrow_dictionary only kept a single Dict or Constant, so these were canonicalized and re-encoded row by row. Export each chunk as an Arrow dictionary and concatenate them instead, which merges only the dictionary values. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS Signed-off-by: Joe Isaacs <[email protected]>
…hopeful-bohr-l12an2 Signed-off-by: Joe Isaacs <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS Signed-off-by: Joe Isaacs <[email protected]>
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the PolarSignals data is written, its Arrow dictionary columns get decoded, and every scan then has to hash them back into Arrow dictionaries. Most of the scan time goes on the nested columns under
locations.This PR lets the writer keep chosen encodings at chosen field paths. The dictionaries stay dictionary-encoded on disk, and the scan hands them back as
DictArrays. Arrow export then keeps those dictionaries instead of re-encoding rows.API
WriteStrategyBuilder::with_allow_encodings(path, encodings): the field atpathkeeps the given encodings it arrives with, and only their children (codes and values) are compressed.Field::ElementTypelets paths reach into list elements.Changes
AllowedEncodingsCompressorwalks struct fields and list elements to the allowed paths, keeps the allowed arrays there, and compresses only their child slots.RepartitionStrategykeeps allowed encodings instead of canonicalizing them, and merges dictionary blocks by rebasing their codes.vortex-arrow): under the list layout, an element column reads back as aChunkedArrayof per-chunk dictionaries.to_arrow_dictionaryused to canonicalize it and re-hash every row. It now exports each chunk as an Arrow dictionary and concatenates them, which only merges the dictionary values. It falls back to the row-wise path when the merged values overflow the codes type.vortex-bench's parquet conversion allowsDictat each Arrow dictionary field path.Testing
allowed_dicts_keep_their_dictionaries(vortex-file): the scan returnsDictfor a top-level column and for a column nested in a list only when the field is allowed, and the round trip is exact in both cases.chunked_dicts_keep_their_values(vortex-arrow): a chunked array of dictionaries exports with the chunks' dictionary values, not re-encoded ones.fa42c42vs baseefc7889; both sides write PolarSignals with the list layout):to_arrow_dictionaryre-hashing rows.🤖 Generated with Claude Code
https://claude.ai/code/session_01AUHM7GB9hiFe56JK863CKS