Repository navigation
Conversation
ludfjig
requested review from
andreiltd,
danbugs,
dblnz,
devigned,
jprendes,
jsturtevant,
simongdavies,
squillace and
syntactically
as code owners
October 12, 2026 20:01
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 13, 2026 18:56
96fab64 to
ab775e0
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Lazy no-reserve allocations need aggregate admission or commitment accounting to prevent deferred host-wide OOM.
1 open finding
What changed in this PR
Optimizes sandbox creation by constructing initial snapshot memory directly in shared memory and relying on lazy page population.
Changes:
- Writes guest initialization data in place.
- Freezes shared memory without copying.
- Documents the performance improvement.
| File | Description |
|---|---|
src/hyperlight_host/src/sandbox/snapshot/mod.rs |
Builds snapshots directly in shared memory. |
src/hyperlight_host/src/mem/shared_mem.rs |
Restricts the copying helper to tests and HVF. |
CHANGELOG.md |
Documents lazy memory population. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 13, 2026 21:01
ab775e0 to
107a858
Compare
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 13, 2026 23:00
107a858 to
aa55340
Compare
This comment has been minimized.
This comment has been minimized.
andreiltd
previously approved these changes
Oct 14, 2026
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 15, 2026 22:00
aa55340 to
421a453
Compare
This comment has been minimized.
This comment has been minimized.
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 21, 2026 19:27
421a453 to
5631afe
Compare
Benchmark ResultsMeasured commit: kvm / amd (Linux) (🌟 6.07x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (🌟 5.92x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (🌟 6.72x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (🌟 7.17x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |
call_and_create times a guest call plus snapshot creation, so the call pays for memory remapping done by the previous snapshot. restore_alternate restores two snapshots in turn, so each restore targets memory that is not current. Signed-off-by: Ludvig Liljenberg <[email protected]>
Share immutable data blobs between related snapshots. Each blob holds guest data followed by the page tables of the snapshot that created it. Persist layered snapshots as OCI image layouts. Capture copies only pages that changed since the snapshot the sandbox was created from or last restored to. Signed-off-by: Ludvig Liljenberg <[email protected]>
Snapshot capture switches the running sandbox to the captured snapshot. The next capture reuses its layers and copies only pages changed after it. Scratch pages used before the capture become free again. They are zeroed so guest allocations return zeroed pages, as after restore. Capture removes map_region and map_file_cow regions. Pages of these regions that the guest has not mapped are lost. Signed-off-by: Ludvig Liljenberg <[email protected]>
walk_va_spaces merges mappings that have contiguous physical and virtual addresses and the same kind. Snapshot capture classifies and rebuilds each merged mapping once. Criterion snapshots/create: * MSHV: medium 1.18 ms to 950 us (-20%), large 3.56 ms to 2.85 ms (-20%) * KVM: medium 1.34 ms to 1.04 ms (-22%), large 3.59 ms to 2.91 ms (-19%) Signed-off-by: Ludvig Liljenberg <[email protected]>
A page-table walk reads every entry through GuestPhysicalMemoryView, which resolves the backing and copies 8 bytes through a locked volatile copy. The reader keeps a copy of the last table page it read, so each table costs one copy. Snapshot capture and crash dumps use the reader. Criterion snapshots/create: * MSHV: default 344 us to 248 us (-28%), medium 950 us to 594 us (-37%) * KVM: default 499 us to 379 us (-24%), medium 1.04 ms to 746 us (-28%) Signed-off-by: Ludvig Liljenberg <[email protected]>
By default, deleting a memslot flushes the guest mappings of every memslot. KVM keeps that flush as a workaround for VMs with assigned GPUs. Hyperlight assigns no devices and disables the quirk when the kernel supports it (Linux 6.12 and later). Deleting a memslot then flushes only that memslot's mappings. Snapshot capture deletes a memslot when a layer drops out of the snapshot. Restore to another snapshot and region unmapping also delete memslots. Criterion on KVM (Linux 6.17): * snapshots/call_and_create: 24% to 58% faster * snapshots/restore_alternate: 32% to 52% faster Signed-off-by: Ludvig Liljenberg <[email protected]>
* Iterate snapshot mappings in reverse and test membership with contains. * Rename GuestVirtualMemoryReader::read and the public SnapshotMemoryReader::read to read_exact. * Move the snapshot memory test helpers into the tests module. Signed-off-by: Ludvig Liljenberg <[email protected]>
Snapshot::from_env writes the guest code, PEB and init data straight into shared memory and freezes it. The host populates the remaining pages on first guest touch, so sandbox creation cost stays independent of heap size. Signed-off-by: Ludvig Liljenberg <[email protected]>
ludfjig
force-pushed
the
ll/2-lazy-sandbox-memory
branch
from
October 26, 2026 18:44
5631afe to
5c04ce3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

New sandboxes skip copying their initial memory.
Snapshot::from_envwrites the guest code, PEB, init data and page tables in place, and the host populates the rest, mostly the heap, on first guest touch. Sandbox creation gets 33 to 88% faster for small to large heaps.Benchmarks
Compared to the PR below.
sandboxes/create_initialized/smallsandboxes/create_initialized/mediumsandboxes/create_initialized/largesnapshot_files/cold_start_via_evolve/largeNo benchmark regresses.
Full results
All benchmarks on both hosts (click to expand)
Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.
snapshots
call_and_createcall_and_createcall_and_createcall_and_createcreatecreatecreatecreaterestorerestorerestorerestorerestore_alternaterestore_alternaterestore_alternaterestore_alternateguest_calls
callcallcallcallcall_with_host_functioncall_with_host_functioncall_with_host_functioncall_with_host_functioncall_with_restorecall_with_restorecall_with_restorecall_with_restoredifferent_threadinterrupt_latencyguest_functions_with_large_parameters
guest_call_with_large_parameterssample_workloads
24K_in_8K_out_c24K_in_8K_out_rustsandboxes
create_initializedcreate_initializedcreate_initializedcreate_initializedcreate_initialized_and_dropcreate_initialized_and_dropcreate_initialized_and_dropcreate_initialized_and_dropsandbox_from_snapshotsandbox_from_snapshotsandbox_from_snapshotsandbox_from_snapshotsnapshot_files
cold_start_via_evolvecold_start_via_evolvecold_start_via_evolvecold_start_via_evolvecold_start_via_snapshotcold_start_via_snapshotcold_start_via_snapshotcold_start_via_snapshotcold_start_via_snapshot_unverifiedcold_start_via_snapshot_unverifiedcold_start_via_snapshot_unverifiedcold_start_via_snapshot_unverifiedload_snapshotload_snapshotload_snapshotload_snapshotload_snapshot_unverifiedload_snapshot_unverifiedload_snapshot_unverifiedload_snapshot_unverifiedsave_snapshotsave_snapshotsave_snapshotsave_snapshot