Skip to content

fix: probe both signed zeros in Parquet bloom filter pruning - #26978

Open
akshat-lakhera wants to merge 1 commit into
apache:mainfrom
akshat-lakhera:fix/26491-bloom-signed-zero
Open

akshat-lakhera wants to merge 1 commit into
apache:mainfrom
akshat-lakhera:fix/26491-bloom-signed-zero

Conversation

@akshat-lakhera

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

A Parquet bloom filter hashes the plain-encoded value, so 0.0 and -0.0 are different members. DataFusion probed only the literal's own bit pattern, so a row group holding only -0.0 was skipped for WHERE c = 0.0, and a row group holding only 0.0 was skipped for WHERE c = -0.0. SQL treats the two as equal, and the query returns those rows once bloom_filter_on_read is turned off, so rows were silently missing. Since bloom_filter_on_read is on by default, this affects any file with a bloom filter on a float column.

What changes are included in this PR?

In BloomFilterStatistics::check_scalar, a Float64 or Float32 zero literal now probes both +0.0 and -0.0, and the row group is kept if either may be present. Every path that reaches the probe (=, IN lists, OR of equalities) goes through this function, so one change covers them. Float16 isn't probed at all and decimals have no signed zero, so neither needed a change.

Are these changes tested?

Yes. I added unit tests in bloom_filter.rs that build files whose bloom filter holds only one sign of zero, for Float64 and Float32, and check that =, IN and OR keep the row group. I also added a sqllogictest in predicates.slt that writes with bloom_filter_on_write and max_row_group_size = 2 and checks that WHERE c = 0.0 and WHERE c = -0.0 both return all four signed-zero rows. The new tests fail without the fix.

While testing I noticed that c IN (0.0) also misses -0.0 rows when it is evaluated in memory, even with bloom filters off. That looks like a separate problem in the IN list evaluation, so the sqllogictest only covers equality and I left it out of this PR.

Are there any user-facing changes?

Queries with = or IN on float columns over Parquet files with bloom filters now return rows that were wrongly pruned before. No API changes.

@akshat-lakhera

Copy link
Copy Markdown
Author

PTAL @alamb

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Oct 31, 2026
@alamb

alamb commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

@hanke580 can you please help review this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet bloom filter pruning drops rows holding the other signed zero: WHERE c = 0.0 misses -0.0 rows (and vice versa)

2 participants