GH-47752: [C++][Python][Parquet] Add minimum compression savings for V2 pages by MashrafeeAryan · Pull Request #52580 · apache/arrow

MashrafeeAryan · 2026-10-09T17:39:40Z

Rationale for this change

Parquet's V2 pages currently keep compression whenever the compressed values are smaller than the original values. Even if it saves very little space, it still does it. For instance, it can compress 100 KB to 99 KB and save it. Even though that's 1% storage efficient, it might not be worth the decompression overhead while reading the file.

What changes are included in this PR?

In the PR, I added an optional min_compression_savings setting between 0.0 and 1.0. It gives users an option to select the minimum threshold to compress V2 page data values. For instance, if the original file is 100 KB and the user has min_compression_savings= 0.1, it means the compressed file needs to save at least 10% space. As a result, unless the file is <= 90 KB, it won't get saved.

I keep the current behavior when the user does not set min_space_savings.

Added C++ and Python implementation.

Are these changes tested?

C++ implementation has been tested with unit tests.
PyArrow implementation has been tested with unti tests.

Are there any user-facing changes?

C++ and Python users can set min_compression_savings when writing V2 pages. For example, 0.1 requires at least 10% space savings before keeping compression.

Was AI used for this PR?

ChatGPT helped to set up the build for PyArrow.

Codex helped find a bug in dataset_writing which ignored min_space_savings.

PR code and description written by:

Human (I wrote it. Used ChatGPT to fix my grammar.)
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Parquet] Make compression adaptive with V2 data pages #47752

github-actions · 2026-10-09T17:40:24Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-10-09T17:43:30Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

Add min_space_savings to control how much space compression must save. V2 pages below the threshold are written uncompressed. Keep the default behavior and V1 pages unchanged.

Add an option to choose how much space compression must save when writing Parquet files from Python. Add tests as well.

Dataset writing ignored min_space_savings. Move the argument to the end of the call so it reaches the writer. Add a test to check that the setting changes the file size.

github-actions · 2026-10-10T09:56:31Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

rok

This looks great @MashrafeeAryan!
I left some inline comments, noting major.
Besides addressing the linter errors, we should update the Python docs. A new option is added to write_table, ParquetWriter, write_to_dataset and ParquetFileWriteOptions. We should update _parquet_writer_arg_docs in python/pyarrow/parquet/core.py and the data-page-V2 paragraph in docs/source/cpp/parquet.rst.

Throw an error when users set min_space_savings for V1 pages. Update C++ and Python tests.

Require compressed data to be smaller, even when min_space_savings is 0.0. Simplify the compression check and update the test.

Check min_space_savings when users set it. Update the test and Python error handling.

Check compressed and original sizes in the existing test. Keep checking that the data reads back correctly.

Explain min_space_savings in the Python and C++ docs.

Fix spacing and line wrapping in the C++ changes.

rok · 2026-10-23T12:02:00Z

Doc issue appears to be from wrong ordering in _parquet_writer_arg_docs. min_space_savings must com last before kwargs.

PR03: Wrong parameters order. Actual: ('table', 'where', 'row_group_size', 'version', 'use_dictionary', 'compression', 'write_statistics', 'use_deprecated_int96_timestamps', 'coerce_timestamps', 'allow_truncated_timestamps', 'data_page_size', 'flavor', 'filesystem', 'compression_level', 'use_byte_stream_split', 'column_encoding', 'data_page_version', 'use_compliant_nested_type', 'encryption_properties', 'write_batch_size', 'dictionary_pagesize_limit', 'store_schema', 'write_page_index', 'write_page_checksum', 'sorting_columns', 'store_decimal_as_integer', 'write_time_adjusted_to_utc', 'max_rows_per_page', 'bloom_filter_options', 'use_content_defined_chunking', 'min_space_savings', '**kwargs'). Documented: ('table', 'where', 'row_group_size', 'version', 'use_dictionary', 'compression', 'write_statistics', 'use_deprecated_int96_timestamps', 'coerce_timestamps', 'allow_truncated_timestamps', 'data_page_size', 'flavor', 'filesystem', 'compression_level', 'use_byte_stream_split', 'column_encoding', 'data_page_version', 'min_space_savings', 'use_compliant_nested_type', 'encryption_properties', 'write_batch_size', 'dictionary_pagesize_limit', 'store_schema', 'write_page_index', 'write_page_checksum', 'sorting_columns', 'store_decimal_as_integer', 'write_time_adjusted_to_utc', 'max_rows_per_page', 'bloom_filter_options', 'use_content_defined_chunking', '**kwargs')

rok · 2026-10-23T12:12:32Z

@MashrafeeAryan this is looking good, some minor suggestions from me, but I think it's ready for another reviewer. Feel free to take it out of draft.

@pitrou could we ask you for a review here?

rok · 2026-10-26T11:12:25Z

Ping @pitrou :)

pitrou · 2026-10-26T16:40:05Z

I've submitted #55272 to write V2 data pages by default.

rok · 2026-10-26T16:48:59Z

I've submitted #55272 to write V2 data pages by default.

@pitrou that's good. I don't think it changes anthing for this PR?

Also, please note @MashrafeeAryan is my intern and will work on fast read/write path for fixed size lists. (As proposed by hardwood).

pitrou · 2026-10-26T16:50:36Z

@pitrou that's good. I don't think it changes anthing for this PR?

No.

Allow zero with V1 and reject nonzero values. Combine dataset tests and keep threshold checks in C++. Update docs to match.

MashrafeeAryan · 2026-10-26T20:12:25Z

Hi @rok and @pitrou, I have addressed the comments. Please let me know if there is anything else needed! Thank you so much for taking the time to review the PR!

pitrou

Looks mostly good to me now, just two small suggestions

pitrou · 2026-10-27T09:23:06Z

+    page_is_compressed =
+        compressed < values->size() &&
+        1.0 - static_cast<double>(compressed) / values->size() >= minimum;


I think the compressed < values->size() has become superfluous now. If compressed is >= values->size() then the compression savings check will fail.

Suggested change

page_is_compressed =

compressed < values->size() &&

1.0 - static_cast<double>(compressed) / values->size() >= minimum;

page_is_compressed =

(1.0 - static_cast<double>(compressed) / values->size()) >= minimum;

Hi @pitrou, thank you so much for the suggestion! I was thinking that since we set the default minimum at 0.0, if the compressed size equals the original size, the savings would be 0.0, so the condition 0.0 >= 0.0 would pass.

Would it make sense to keep compressed < values->size() so we avoid decompression with no space savings? Please let me know if I am overlooking something.

Could we change the default of min_compression_savings to 0.01? or some epsilon-like value? @pitrou

Could we change the default of min_compression_savings to 0.01? or some epsilon-like value? @pitrou

I think we can.

pitrou · 2026-10-27T09:23:53Z

@MashrafeeAryan Could you please also update the PR description for the name change? Thank you!

Co-authored-by: Antoine Pitrou <[email protected]>

github-actions · 2026-10-27T14:53:47Z

⚠️ GitHub issue #47752 has been automatically assigned in GitHub to PR creator.

rok · 2026-10-28T11:03:43Z

Is this ready for another round of review @MashrafeeAryan ?

MashrafeeAryan · 2026-10-28T15:33:35Z

Is this ready for another round of review @MashrafeeAryan ?

Hi @rok and @pitrou , this is ready for another round of review. For minimum threshold, I believe the check maybe a good option but if any of you have any suggestions, I would love to implement it.

github-actions Bot added Component: Parquet Component: C++ Component: Python awaiting review Awaiting review labels Oct 9, 2026

MashrafeeAryan added 3 commits October 9, 2026 20:43

Add adaptive compression for Parquet V2 pages

688cfcd

Add min_space_savings to control how much space compression must save. V2 pages below the threshold are written uncompressed. Keep the default behavior and V1 pages unchanged.

Let Python users set minimum compression savings

01512cc

Add an option to choose how much space compression must save when writing Parquet files from Python. Add tests as well.

Fix ignored compression savings in dataset writing

89bf1b2

Dataset writing ignored min_space_savings. Move the argument to the end of the call so it reaches the writer. Add a test to check that the setting changes the file size.

MashrafeeAryan force-pushed the adaptive-compression branch from 65825cc to 89bf1b2 Compare October 10, 2026 09:57

rok reviewed Oct 16, 2026

View reviewed changes

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread cpp/src/parquet/column_writer.cc Outdated

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread python/pyarrow/tests/parquet/test_parquet_writer.py Outdated

MashrafeeAryan added 6 commits October 18, 2026 09:12

Reject min_space_savings for V1 pages

7d705a9

Throw an error when users set min_space_savings for V1 pages. Update C++ and Python tests.

Skip compression when it saves no space

d8d4642

Require compressed data to be smaller, even when min_space_savings is 0.0. Simplify the compression check and update the test.

Reject bad savings values right away

f4cc427

Check min_space_savings when users set it. Update the test and Python error handling.

Check that Python uses compression savings

f20f66c

Check compressed and original sizes in the existing test. Keep checking that the data reads back correctly.

Explain minimum compression savings

adef6d4

Explain min_space_savings in the Python and C++ docs.

Fix C++ formatting

e4953da

Fix spacing and line wrapping in the C++ changes.

github-actions Bot added the Component: Documentation label Oct 18, 2026

rok reviewed Oct 23, 2026

View reviewed changes

Comment thread cpp/src/parquet/properties.h Outdated

rok reviewed Oct 23, 2026

View reviewed changes

Comment thread python/pyarrow/tests/parquet/test_parquet_writer.py

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Oct 23, 2026

MashrafeeAryan marked this pull request as ready for review October 24, 2026 03:03

MashrafeeAryan requested review from AlenkaF, pitrou, raulcd and wgtmac as code owners October 24, 2026 03:03

MashrafeeAryan requested a review from HuaHuaY as a code owner October 24, 2026 03:03

MashrafeeAryan added 2 commits October 24, 2026 12:12

Fix documentation for rok's doc comment

98a8f31

Add compression savings threshold test

4ebfd62

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 25, 2026

pitrou reviewed Oct 26, 2026

View reviewed changes

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread cpp/src/parquet/properties.h Outdated

Comment thread python/pyarrow/tests/test_dataset.py Outdated

Comment thread python/pyarrow/tests/parquet/test_parquet_writer.py Outdated

MashrafeeAryan added 2 commits October 26, 2026 14:02

Test both sides of the compression savings threshold

6ea5a99

Rename compression savings and use zero as the default

68ead0d

Allow zero with V1 and reject nonzero values. Combine dataset tests and keep threshold checks in C++. Update docs to match.

pitrou requested changes Oct 27, 2026

View reviewed changes

Update docs/source/cpp/parquet.rst

714a097

Co-authored-by: Antoine Pitrou <[email protected]>

Remove extra space in Parquet docs

5dae58c

Conversation

MashrafeeAryan commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

rok left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok commented Oct 23, 2026

Uh oh!

Uh oh!

rok commented Oct 23, 2026

Uh oh!

rok commented Oct 26, 2026

Uh oh!

pitrou commented Oct 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok commented Oct 26, 2026

Uh oh!

pitrou commented Oct 26, 2026

Uh oh!

MashrafeeAryan commented Oct 26, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

MashrafeeAryan Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

rok Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

rok commented Oct 28, 2026

Uh oh!

MashrafeeAryan commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MashrafeeAryan commented Oct 9, 2026 •

edited

Loading

rok left a comment •

edited

Loading