Repository navigation
Conversation
…aim estimate Density observability, off by default. Guest RAM is faulted into the hugepage pool lazily (UFFD + hugetlbfs), so a node's real ceiling is the working set actually faulted in, not the RAM sandboxes nominally declare — and an idle sandbox keeps holding pool pages it no longer uses, because nothing reclaims them at runtime. Today there is no per-sandbox visibility into either, so any reclaim/sharing/tiering work has no data to justify or tune against. This adds two per-sandbox gauges, gated behind the sandbox-density-metrics flag (default false → no new series until enabled): - orchestrator.sandbox.mem.working_set: host bytes faulted into the pool for the sandbox since resume, read from the UFFD handler's existing ServeStats (no new instrumentation on the fault path) via a new Sandbox accessor MemoryWorkingSetBytes. An upper bound on current pool residency. - orchestrator.sandbox.mem.reclaimable_estimate: pool memory the node holds that the guest is not actively using — max(0, workingSet - guestInUse), where guestInUse is the envd-reported used RAM. Combined with free-page reporting: when the sandbox's balloon returns freed pages itself (BalloonModeReporting), the pool is self-reclaiming and the estimate is 0, so it does not double-count pages the device already discards. The estimate feeds a later reclaim decision, so it is floored at zero and never over-reports; reclaimableBytes is a pure function with table tests (gap, floor-at-zero, no-guest-report upper bound, free-page-reporting zero). Scope: observability only — nothing is reclaimed, paused or shared here. It is the data floor for later density work (live balloon reclaim, same-template page sharing, pool-pressure auto-pause); see e2b-dev#3763. Off by default means zero behaviour and zero added metric cardinality until an operator turns it on. Refs e2b-dev#3763.
The data floor for same-template page sharing. Sandboxes of one template (one buildID) already share a single source memfile on the host — the pinned template cache (GetTemplatePinned(buildID)) hands every one the same ReadonlyDevice, so the base image exists once as a source. But the UFFD handler serves each sandbox's guest RAM via UFFDIO_COPY (MISSING mode), so every sandbox faults its OWN private physical copy of those identical, read-only base pages into the pool. N same-template sandboxes = N physical copies of a source that is already one. Nothing today measures how much pool that duplication costs, so there is no basis to justify (or decline) the cross-repo page-sharing work it would take to collapse N to 1. This adds, behind the existing sandbox-density-metrics flag (default off): - ServeSnapshot.SourceBytes = SourcePages x pageSize: the base-image content a sandbox faulted from the shared source (private written pages excluded — they cannot be shared). Derived from an existing counter, no new fault-path work. - Sandbox.MemorySourceBytes() exposes it. - orchestrator.sandbox.mem.shareable_base_estimate, a per-build_id gauge: the resident sandboxes are grouped by build in an in-memory pre-pass (no RPC) and each build with >= 2 sandboxes reports (N-1) x the smallest base footprint among them — a lower bound on the pool a shared base-page copy would free on this node. shareableBaseBytes is a pure function with table tests (floor at <2, (N-1)*min shape, a zero member dragging the bound to 0). Scope: observability only. Nothing is shared, reclaimed, or changed in the serve path; off by default means zero behaviour and zero added cardinality until enabled. It is the go/no-go evidence for same-template page sharing (the FC-side shared_memfd_path mechanism exists as prior art in the fork but is not in the deployed FC, and the runtime has no MINOR-fault support yet): turn it on, and if a node's templates are dense enough for N->1 to free meaningful pool, the gauge says so before any risky cross-repo work. Builds on the per-sandbox working-set observability (same flag). Refs e2b-dev#3763.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 29, 2026 12:36
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The go/no-go data floor for same-template page sharing (the biggest remaining density lever in #3763). Observability only, off by default.
Why
Sandboxes of one template (one
buildID) already share a single source memfile on the host: the pinned template cache (GetTemplatePinned(buildID)) hands every one the sameReadonlyDevice. But the UFFD handler serves each sandbox's guest RAM viaUFFDIO_COPY(MISSING mode), so each faults its own private physical copy of those identical, read-only base pages into the pool. N same-template sandboxes = N physical copies of a source that is already one. Nothing measures what that duplication costs, so there's no basis to justify — or decline — the cross-repo work (FC-forkshared_memfd_path+ runtime MINOR/UFFDIO_CONTINUE) that collapsing N→1 would take.What
Behind the existing
sandbox-density-metricsflag (default off):ServeSnapshot.SourceBytes=SourcePages × pageSize— base-image content faulted from the shared source (private written pages excluded; they can't be shared). Derived from an existing counter, no new fault-path work.Sandbox.MemorySourceBytes()exposes it.orchestrator.sandbox.mem.shareable_base_estimate— per-build_idgauge. Resident sandboxes are grouped by build in an in-memory pre-pass (no RPC); each build with ≥2 sandboxes reports(N-1) × min(base footprint)— a lower bound on the pool a single shared base-page copy would free on this node.shareableBaseBytesis pure, table-tested (floor at <2,(N-1)×minshape, a zero member dragging the bound to 0).Safety / scope
SandboxObserverregression pass;go build/go vetclean; uffduserfaultfdsuite green.How it gates the risky work
Same-template page sharing is a cross-repo, SIGBUS-class, hot-path change (FC fork forward-port of
shared_memfd_pathfrom the v1.12 fork — it is not in the deployedv1.14-0.2.0— plus runtime MINOR-fault support, which today is actively rejected). This PR is the measurement that says whether a node's templates are dense enough for N→1 to free meaningful pool before any of that is attempted. Measure first, then decide.Refs #3763.