Skip to content

feat(orchestrator): pool-pressure-adaptive free-page hinting interval - #3768

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/adaptive-hinting-pressure
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/adaptive-hinting-pressure

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Next step toward #3763 (raising sandboxes-per-node), building on the observability in #3766. Flag-gated, strict no-op by default.

The gap (what's already done, and what isn't)

Free-page hinting already returns a guest's freed pages to the hugepage pool: the guest reports free pages, Firecracker calls discard_range, and for the shared/hugetlbfs memfd backing that is madvise(MADV_REMOVE) — which punches the backing file and frees the physical hugepages. (Verified by disassembling the deployed v1.14-0.2.0 binary: discard_range uses MADV_DONTNEED only for anonymous mappings and MADV_REMOVE for the shared memfd; a quick kernel experiment confirms MADV_REMOVE/PUNCH_HOLE return hugepages to the pool while MADV_DONTNEED does not.)

What is not adaptive is when hinting runs. The periodic loop ticks on a fixed interval, blind to how full the node's hugepage pool is. So between a guest freeing memory and the next tick (tens of seconds at the default interval), those pages stay charged to the pool. On a densely packed node that reclaim-lag window is exactly what makes placement refuse a node that is about to have room — the pool reads full of pages the guests have already freed but not yet hinted back.

Change

  • PeriodicHintingConfig gains PressureInterval and PressureThresholdPct. When the node hugepage pool is at/above the threshold, the loop waits the shorter PressureInterval instead of Interval, so freed pages return to the pool sooner and the reclaim-lag window shrinks. Below threshold, or either knob unset → the fixed Interval (unchanged default).
  • effectiveHintInterval — pure, fully tested: base interval unless pressure ≥ threshold AND a shorter PressureInterval is set; never slows hinting down; unknown pressure (-1) fails safe to the base interval.
  • Node pool pressure read from /proc/meminfo (HugePages_Total/Free) with a 1-second process-level cache, so the per-sandbox loops don't each read meminfo every tick — one read/second serves every sandbox on the node. hugePageUsagePct is also pure/tested.

Safety / scope

  • The only behavioural change is the inter-tick wait under pool pressure. It does not touch the UFFD serve path, the discard/punch path, or any of the latch / silent-guest / unresponsive-retry invariants in the hinting loop.
  • PressureInterval = 0 (default) → strict no-op.
  • go build/go vet clean; pure-function tests (interval selection, usage %, TTL caching) pass; existing hinting/periodic suites pass.

Relationship to #3766

#3766's reclaimable_estimate gauge quantifies the exact reclaim-lag window this PR shrinks, and is how an operator picks PressureThresholdPct / PressureInterval. Measure first (#3766), then act (this).

Refs #3763.

Free-page hinting already returns a guest's freed pages to the hugepage pool:
the guest reports them, Firecracker calls discard_range, and for the
shared/hugetlbfs memfd backing that is madvise(MADV_REMOVE), which punches the
backing file and frees the physical hugepages (verified on the deployed
v1.14-0.2.0 binary — DONTNEED is used only for anonymous mappings, REMOVE for
the shared memfd). What is NOT adaptive is WHEN hinting runs: the periodic
loop ticks on a fixed interval, blind to how full the node's hugepage pool is.

So between a guest freeing memory and the next tick (tens of seconds at the
default interval), those pages stay charged to the pool. On a densely packed
node that reclaim-lag window is exactly what makes placement refuse a node
that is really about to have room — the pool reads full of pages the guests
have already freed but not yet hinted back.

This makes the interval pressure-aware:

- PeriodicHintingConfig gains PressureInterval and PressureThresholdPct. When
  the node hugepage pool is at or above the threshold, the loop waits the
  (shorter) PressureInterval instead of Interval, so just-freed pages return
  to the pool sooner and the reclaim-lag window shrinks. Below the threshold,
  or when either knob is unset, the fixed Interval is used — unchanged default
  behaviour.
- effectiveHintInterval is a pure function (fully tested): base interval
  unless pressure >= threshold AND a shorter PressureInterval is configured;
  it never slows hinting down, and unknown pool pressure (-1) fails safe to
  the base interval.
- Node pool pressure is read from /proc/meminfo (HugePages_Total/Free) with a
  one-second process-level cache, so the per-sandbox loops do not each read
  meminfo every tick — one read per second serves every sandbox on the node.

Scope: the only behavioural change is the inter-tick wait under pool pressure.
It does not touch the UFFD serve path, the discard/punch path, or any of the
latch / silent-guest / unresponsive-retry invariants in the hinting loop. With
PressureInterval=0 (the default) it is a strict no-op.

Builds on the per-sandbox working-set/reclaimable observability (e2b-dev#3766): that
PR's reclaimable_estimate gauge quantifies the very reclaim-lag window this one
shrinks, and is how an operator picks PressureThresholdPct / PressureInterval.

Refs e2b-dev#3763.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants