Repository navigation
issue: Materialize the newest rotated partition - #301
Merged
Merged
Conversation
A partition is only cached as a full scan when the materialization range covers all of its flows. The range padded the start by a day but the end by only a second. A flow is written to the partition of the time it was captured, yet can end after that partition's window -- a flow still open as the partition rotates is captured in it but ends in the next one, and an exporter whose clock runs ahead reports an end time past the window. The newest partition that rotated is the one most likely to hold such a flow, and is also the one still on InnoDB before the poller converts it. With the one second end pad its MAX(end_time) fell outside the range, so it was never a full scan and never cached, and its Filter stayed Required even immediately after a materialization. Pad the end by a day, as the start already is. flowview_get_materialize_range is only used by the materializer, so report ranges are unchanged. Co-authored-by: Copilot <[email protected]>
|
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The localized fix addresses the materialization boundary condition with targeted regression coverage and no unresolved findings.
No open issues 🎉
What changed in this PR
Fixes FlowView materialization skipping the newest rotated partition when a flow’s end time extends beyond its nominal window, leaving the Filter marked Required.
Changes:
- Extends the materialization end padding from one second to one day, matching the start padding.
- Updates range expectations and adds regression coverage for boundary-spanning flows and live-partition exclusion.
- Documents the fix in the function comment and changelog.
| File | Description |
|---|---|
| tests/Unit/CoreFunctionsTest.php | Tests the wider range and full-scan eligibility while preserving live-partition exclusion. |
| includes/functions.php | Adds one-day end padding and explains its purpose. |
| CHANGELOG.md | Records the materialization fix and its user-visible effect. |
Review effort: Balanced
xmacan
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Fixes a materialization bug where the newest cold partition is never cached, so a Filter keeps showing Required even immediately after a successful materialization. On an install with 1,523 partitions this showed as NAT/Non-NAT shards capping at 1,521 while there were 1,522 cold partitions — exactly one cold partition (the most recently rotated) perpetually missing.
Root cause
A partition is cached only when its shard is a full scan, which requires the materialization range to lie strictly outside the partition's data (
flowview_is_full_scan).flowview_get_materialize_rangepadded the start by a full day but the end by only one second.A flow is written to the partition of the time it was captured, but its
end_timecan fall after that partition's nominal window:The newest rotated partition is the one most likely to hold such a boundary-spanning flow (and is also the one still on InnoDB before the poller converts it to Aria). Its
MAX(end_time)fell a few minutes past the one-second end pad, so it was never a full scan and never cached — and re-materializing never helped, because it is still the newest cold partition with the same too-tight range end.Fix
Pad the end by a day too, symmetric with the start (
[$start - 86400, $end + 86400]).flowview_get_materialize_rangeis only used by the materializer, so report ranges are unaffected. The live partition is still excluded from full scans regardless of range.How I verified it
I reproduced it on a scratch Cacti 1.2.32 + FlowView install:
full_scan = NO, cached 4/5, perpetual Required (matches the report).full_scan = YESagain, cached 5/5.Testing
flowview_get_materialize_rangetest for the new end pad and added a regression test that a partition whose newest flow ends after its window is a full scan (and that the live partition never is).developunder Windows (shell quoting).