chore: Update to arrow/parquet `60.1.0` by alamb · Pull Request #26867 · apache/datafusion

alamb · 2026-10-28T21:49:19Z

Which issue does this PR close?

Related to Release arrow-rs / parquet Minor version 60.1.0 (October 2026) arrow-rs#10525
replaces / closes [DO NOT MERGE] Test DataFusion against arrow/parquet 60.1.0 release candidate (apache/arrow-rs#11750) #26543

Rationale for this change

Update to the latest arrow/parquet release, 60.1.0. See the upstream CHANGELOG for details.

What changes are included in this PR?

Update to arrow/parquet 60.1.0
Adjust code for upstream API differences (see inline comments)
Update test expectations for upstream behavior changes

What is the testing strategy for this PR?

Existing CI. This release candidate was also tested in #26543.

Are there any user-facing changes?

New dependency versions.

alamb · 2026-10-29T16:34:14Z


    fn value_offsets(&self, row: usize) -> (i64, i64) {
-        let start = self.value_offset(row) as i64;
+        let start = self.value_offset_at(row) as i64;


Due to

fix(arrow-array): deprecate FixedSizeListArray::value_offset, which wraps past i32::MAX arrow-rs#11070

alamb · 2026-10-29T16:34:14Z

        // while spill_file_count remains 1 (since we're writing to the same file)
        in_progress_file.append_batch(&batch1)?;
-        verify_metrics(&in_progress_file, 1, 440, 3)?;
+        verify_metrics(&in_progress_file, 1, 424, 3)?;


Due to

Write empty validity buffers for arrays without nulls in IPC writer arrow-rs#11694

size is smaller 🎉

alamb · 2026-10-29T16:34:14Z

                        .map(|row| (input * rows_per_input + row) % dictionary_len),
                );
-                take(&dictionary, &indices, None)
+                let taken = take(&dictionary, &indices, None)?;


Due to

Reuse buffers when interleaving byte views arrow-rs#11281

Co-Authored-By: Claude Fable 5.1 <[email protected]>

alamb · 2026-10-29T17:04:38Z

-## all. Every route into a named zone -- a column cast, `AT TIME ZONE`, a
-## string literal, or `::timestamptz` under a session zone -- fails with an
-## error, where PostgreSQL resolves the value.
+## Casting a naive timestamp into a named zone resolves an ambiguous or


Due to

Resolve DST-ambiguous and nonexistent local times when casting timestamps to a named timezone arrow-rs#11038

Co-Authored-By: Claude Fable 5.1 <[email protected]>

alamb · 2026-10-29T17:28:36Z

 01)ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[<slt:ignore>]
 02)--AggregateExec: mode=Single, gby=[], aggr=[count(Int64(1))], metrics=[<slt:ignore>]
-03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=8, input_rows=516, <slt:ignore>]
+03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=6, input_rows=387, <slt:ignore>]


Due to

Write empty validity buffers for arrays without nulls in IPC writer arrow-rs#11694

The right side is replayed once per left chunk. Spilled batches are now read back without validity buffers for null-free columns, so they take less memory and the build side fits in three chunks instead of four.

alamb · 2026-10-29T17:28:38Z

 SELECT approx_distinct(c14) AS a, approx_distinct(c15) AS b, approx_distinct(arrow_cast(c15, 'Date64')), approx_distinct(arrow_cast(c15, 'Time32(Second)')) as c, approx_distinct(arrow_cast(c15, 'Time64(Nanosecond)')) AS d FROM aggregate_test_100
 ----
-18 60 60 60 60
+18 60 7 60 60


Due to

fix(arrow-cast): truncate Timestamp to Date64 casts to whole days arrow-rs#11632

Casting a Timestamp to Date64 now truncates to whole days (previously it kept the time of day), so only 7 distinct dates remain.

alamb · 2026-10-29T17:28:39Z

 FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)')), (NULL)) AS t(ts)
 ----
-NULL NULL NULL NULL
+7 NULL NULL NULL


Due to

perf: Optimize date_part for timestamp values without timezones arrow-rs#11187

second is now computed directly from the stored value for timestamps without a timezone,

i64::MAX seconds yields 7 (i64::MAX % 60) instead of overflowing to NULL.

The sub-second parts still go through chrono and remain NULL. The nullability reproducer further down now uses millisecond so it still produces a NULL from a non-null input.

alamb · 2026-10-29T17:32:28Z

    /// Arrays that each hold views into one shared dictionary buffer, as the
    /// Parquet reader produces for a dictionary-encoded column, combined the
-    /// way a sort combines its input batches. Each output row cycles through
+    /// way a sort combines its input batches: the combined array lists the


note this is test code

alamb · 2026-10-30T19:44:29Z

-    /// Parquet reader produces for a dictionary-encoded column, combined the
-    /// way a sort combines its input batches. Each output row cycles through
+    /// Parquet reader produces for a dictionary-encoded column, combined with
+    /// `concat` as a sort combines its buffered batches. The rows cycle through


interleave no longer duplicates the buffers

kumarUjjawal · 2026-10-31T03:54:24Z

Should this also be updated https://github.com/apache/datafusion/blob/601a21700337d62410f315634d48d5cd4ce1715e/docs/source/library-user-guide/upgrading/56.0.0.md#upgrade-arrowparquet-to-6000-and-object_store-to-0142

alamb · 2026-11-01T12:47:41Z

Should this also be updated https://github.com/apache/datafusion/blob/601a21700337d62410f315634d48d5cd4ce1715e/docs/source/library-user-guide/upgrading/56.0.0.md#upgrade-arrowparquet-to-6000-and-object_store-to-0142

Good call, done in 4e77671

alamb · 2026-11-01T12:51:22Z

Ok,let's get this in

alamb mentioned this pull request Oct 29, 2026

[DO NOT MERGE] Test DataFusion against arrow/parquet 60.1.0 release candidate (apache/arrow-rs#11750) #26543

Closed

alamb commented Oct 29, 2026

View reviewed changes

chore: Update to arrow 60.1.0

5a12062

alamb force-pushed the alamb/arrow_60.1.0 branch from ce310c5 to 5a12062 Compare October 29, 2026 16:42

github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Oct 29, 2026

alamb and others added 3 commits October 29, 2026 12:56

trim comments

19e5457

cleaning up

e17a2c4

Update sqllogictest expectations for arrow-rs 60.1.0 behavior changes

2d6fe15

Co-Authored-By: Claude Fable 5.1 <[email protected]>

alamb commented Oct 29, 2026

View reviewed changes

Move behavior-change explanations out of slt comments

5051604

Co-Authored-By: Claude Fable 5.1 <[email protected]>

alamb commented Oct 29, 2026

View reviewed changes

alamb marked this pull request as ready for review October 29, 2026 17:33

alamb added 3 commits October 29, 2026 15:23

Merge branch 'main' into alamb/arrow_60.1.0

26d9ba0

Merge remote-tracking branch 'apache/main' into alamb/arrow_60.1.0

bee542f

simplify test

601a217

alamb commented Oct 30, 2026

View reviewed changes

kumarUjjawal reviewed Oct 31, 2026

View reviewed changes

Comment thread datafusion-testing Resolved

Jefffrey added 2 commits November 1, 2026 14:33

Merge branch 'main' into alamb/arrow_60.1.0

dedf179

fix datafusion-testing submodule

aaa4740

Jefffrey approved these changes Nov 1, 2026

View reviewed changes

geoHeil mentioned this pull request Nov 1, 2026

deps: zstd 0.13, zstd-safe 7 and bincode 1 advisories wait on upstream releases telekom/sutura#1686

Open

4 tasks

alamb added 2 commits November 1, 2026 07:46

clarify versions

4e77671

Merge remote-tracking branch 'apache/main' into alamb/arrow_60.1.0

e7ddd45

github-actions Bot added the documentation Improvements or additions to documentation label Nov 1, 2026

alamb added this pull request to the merge queue Nov 1, 2026

Merged via the queue into apache:main with commit 6a9a129 Nov 1, 2026
45 of 46 checks passed

alamb deleted the alamb/arrow_60.1.0 branch November 1, 2026 13:07

github-actions Bot added the v56.0.0 label Nov 1, 2026

Conversation

alamb commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kumarUjjawal commented Oct 31, 2026

Uh oh!

alamb commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Nov 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alamb commented Oct 28, 2026 •

edited

Loading

alamb Oct 29, 2026 •

edited

Loading

alamb commented Nov 1, 2026 •

edited

Loading