Skip to content

issue: Materialize the newest rotated partition - #301

Merged
TheWitness merged 1 commit into
developfrom
fix/materialize-newest-partition
Oct 23, 2026
Merged

TheWitness merged 1 commit into
developfrom
fix/materialize-newest-partition

Conversation

@TheWitness

Copy link
Copy Markdown
Member

Summary

Fixes a materialization bug where the newest cold partition is never cached, so a Filter keeps showing Required even immediately after a successful materialization. On an install with 1,523 partitions this showed as NAT/Non-NAT shards capping at 1,521 while there were 1,522 cold partitions — exactly one cold partition (the most recently rotated) perpetually missing.

Root cause

A partition is cached only when its shard is a full scan, which requires the materialization range to lie strictly outside the partition's data (flowview_is_full_scan). flowview_get_materialize_range padded the start by a full day but the end by only one second.

A flow is written to the partition of the time it was captured, but its end_time can fall after that partition's nominal window:

  • a flow still open when the partition rotates is captured in it but ends in the next one, and
  • an exporter whose clock runs ahead of the collector reports an end time past the window.

The newest rotated partition is the one most likely to hold such a boundary-spanning flow (and is also the one still on InnoDB before the poller converts it to Aria). Its MAX(end_time) fell a few minutes past the one-second end pad, so it was never a full scan and never cached — and re-materializing never helped, because it is still the newest cold partition with the same too-tight range end.

Fix

Pad the end by a day too, symmetric with the start ([$start - 86400, $end + 86400]). flowview_get_materialize_range is only used by the materializer, so report ranges are unaffected. The live partition is still excluded from full scans regardless of range.

How I verified it

I reproduced it on a scratch Cacti 1.2.32 + FlowView install:

  1. Created 6 daily partitions (5 cold + 1 live), the newest cold one left on InnoDB, all data inside its window → all 5 cold partitions cache, status would be Up to date.
  2. Added one flow to the newest cold partition ending 5 minutes past its window → that partition drops to full_scan = NO, cached 4/5, perpetual Required (matches the report).
  3. Applied the fix → the newest cold partition is full_scan = YES again, cached 5/5.

Testing

  • Updated the flowview_get_materialize_range test for the new end pad and added a regression test that a partition whose newest flow ends after its window is a full scan (and that the live partition never is).
  • Unit suite: 240 passed locally; the only failures are 4 pre-existing background-exec tests that fail the same way on unmodified develop under Windows (shell quoting).

A partition is only cached as a full scan when the materialization range
covers all of its flows.  The range padded the start by a day but the end by
only a second.  A flow is written to the partition of the time it was
captured, yet can end after that partition's window -- a flow still open as
the partition rotates is captured in it but ends in the next one, and an
exporter whose clock runs ahead reports an end time past the window.

The newest partition that rotated is the one most likely to hold such a flow,
and is also the one still on InnoDB before the poller converts it.  With the
one second end pad its MAX(end_time) fell outside the range, so it was never a
full scan and never cached, and its Filter stayed Required even immediately
after a materialization.

Pad the end by a day, as the start already is.  flowview_get_materialize_range
is only used by the materializer, so report ranges are unchanged.

Co-authored-by: Copilot <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:25
@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The localized fix addresses the materialization boundary condition with targeted regression coverage and no unresolved findings.

No open issues 🎉

What changed in this PR

Fixes FlowView materialization skipping the newest rotated partition when a flow’s end time extends beyond its nominal window, leaving the Filter marked Required.

Changes:

  • Extends the materialization end padding from one second to one day, matching the start padding.
  • Updates range expectations and adds regression coverage for boundary-spanning flows and live-partition exclusion.
  • Documents the fix in the function comment and changelog.
File Description
tests/​Unit/​CoreFunctionsTest.php Tests the wider range and full-scan eligibility while preserving live-partition exclusion.
includes/​functions.php Adds one-day end padding and explains its purpose.
CHANGELOG.md Records the materialization fix and its user-visible effect.

Review effort: Balanced

@TheWitness
TheWitness merged commit 2153537 into develop Oct 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants