Repository navigation
fix: probe both signed zeros in Parquet bloom filter pruning - #26978
Open
akshat-lakhera wants to merge 1 commit into
Open
akshat-lakhera wants to merge 1 commit into
akshat-lakhera wants to merge 1 commit into
Conversation
Signed-off-by: Akshat Lakhera <[email protected]>
Author
|
PTAL @alamb |
Contributor
|
@hanke580 can you please help review this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
WHERE c = 0.0misses-0.0rows (and vice versa) #26491.Rationale for this change
A Parquet bloom filter hashes the plain-encoded value, so 0.0 and -0.0 are different members. DataFusion probed only the literal's own bit pattern, so a row group holding only -0.0 was skipped for WHERE c = 0.0, and a row group holding only 0.0 was skipped for WHERE c = -0.0. SQL treats the two as equal, and the query returns those rows once bloom_filter_on_read is turned off, so rows were silently missing. Since bloom_filter_on_read is on by default, this affects any file with a bloom filter on a float column.
What changes are included in this PR?
In BloomFilterStatistics::check_scalar, a Float64 or Float32 zero literal now probes both +0.0 and -0.0, and the row group is kept if either may be present. Every path that reaches the probe (=, IN lists, OR of equalities) goes through this function, so one change covers them. Float16 isn't probed at all and decimals have no signed zero, so neither needed a change.
Are these changes tested?
Yes. I added unit tests in bloom_filter.rs that build files whose bloom filter holds only one sign of zero, for Float64 and Float32, and check that =, IN and OR keep the row group. I also added a sqllogictest in predicates.slt that writes with bloom_filter_on_write and max_row_group_size = 2 and checks that WHERE c = 0.0 and WHERE c = -0.0 both return all four signed-zero rows. The new tests fail without the fix.
While testing I noticed that c IN (0.0) also misses -0.0 rows when it is evaluated in memory, even with bloom filters off. That looks like a separate problem in the IN list evaluation, so the sqllogictest only covers equality and I left it out of this PR.
Are there any user-facing changes?
Queries with = or IN on float columns over Parquet files with bloom filters now return rows that were wrongly pruned before. No API changes.