Skip to content

feat(orchestrator): opt-in default-deny egress posture for sandboxes - #3762

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/egress-default-deny
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/egress-default-deny

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Closes #3696.

An opt-in default-deny egress posture for sandboxes, shipping complete across every layer that currently defaults to allow. default_deny defaults to false → zero behaviour change; a tenant that sets it flips the sandbox to deny-by-default and adds workload rules on top via allowed_cidrs / allowed_domains.

The control-plane allowlist the issue asked for already exists as structure

The issue worried a naive inversion would break resume/volume mounts (as DenyEgress does). It doesn't here, because control-plane traffic is already isolated from the egress path:

  • Control-plane TCP (hyperloop :80, portmapper :111, NFS proxy :2049 to the orchestrator in-sandbox IP) is REDIRECTed in nat/PREROUTING by network.go rules installed before the egress-proxy REDIRECT — first-match-wins routes it to the local control servers, never the egress proxy.
  • Control-plane UDP + DNS are already exempt in the non-DNS-UDP rule (predefinedAllowSet = control IP; :53 carve-out).

So resume/volume mounts keep working under default-deny with no new allowlist code.

Three layers flip together

  • proto: SandboxNetworkEgressConfig gains default_deny and denied_domains (the latter was referenced in handlers.go but had no proto field, as the issue noted).
  • TCP userspace firewall (handlers.go): isEgressAllowed now honours denied_domains, and under default_deny returns deny for anything not explicitly allowed — which also closes the no-SNI / no-Host blind pass the issue flagged (cidrOnlyHandler reaches the default with no hostname, so only an allowed CIDR lets it out).
  • slot nft firewall (firewall.go): under default_deny the non-DNS-UDP rule is forced to drop mode and a terminal non-TCP drop is installed after the established / allow-set accepts — so UDP (the DNS-tunnel vector) and other non-TCP not explicitly allowed are dropped instead of hitting the ACCEPT policy. TCP is never matched there (it leaves via the REDIRECT to the userspace proxy, which enforces deny there).

Addresses each gap the issue listed

Issue gap Fixed by
per-slot nft ends in ACCEPT for non-TCP → UDP unmediated / DNS-tunnelable terminal non-TCP drop + forced UDP drop under default_deny
userspace TCP "default: allow all" isEgressAllowed default→deny under default_deny
non-80/443 + TLS-without-SNI → CIDR-only, no domain inspection, blind pass same default→deny closes the no-hostname path
denied_domains mentioned in code but missing from proto added

Tests

  • isEgressAllowed: default_deny drops-by-default; allowed domain/CIDR still pass; no-SNI dropped; denied_domains blocks under default-allow; allow beats deny.
  • nft netns test: default_deny records posture=deny, forces the UDP mode to Drop, and installs the terminal non-TCP drop as the chain's last rule.
  • go build / go vet clean; existing network and tcpfirewall suites pass.

Note

The issue explicitly asked whether maintainers would accept a default-deny opt-in before investing (CONTRIBUTING's "unlikely to be merged" list re: core networking). This PR is scoped exactly as "opt-in, zero change when off, reuse existing mechanisms" rather than a core refactor, and is offered as a concrete basis for that decision.

Adds an opt-in default-deny egress posture to the sandbox network config.
When default_deny is false (the default) nothing changes: egress is
default-allow and only denied_cidrs/denied_domains subtract from it. When a
tenant sets default_deny, the sandbox flips to deny-by-default across every
layer that currently defaults to allow, and the operator adds workload rules
on top via allowed_cidrs/allowed_domains.

The runtime's own control-plane path needs no new allowlist: it is already
isolated from the egress path. Control-plane TCP (hyperloop :80, portmapper
:111, NFS proxy :2049 to the orchestrator in-sandbox IP) is REDIRECTed in
nat/PREROUTING by rules installed before the egress-proxy REDIRECT, so it
reaches the local control servers and never the egress proxy; DNS and the
control IP are already exempt in the non-DNS UDP rule. So resume/volume
mounts keep working under default-deny — the gap the existing DenyEgress
mode could not fill.

Three layers flip together so the posture ships complete:

- proto: SandboxNetworkEgressConfig gains default_deny and denied_domains
  (the latter was referenced in handlers.go but had no proto field).
- TCP userspace firewall (handlers.go): isEgressAllowed honors denied_domains
  and, under default_deny, returns deny for anything not explicitly allowed —
  which also closes the no-SNI / no-Host blind pass, since cidrOnlyHandler
  reaches the default with no hostname and only an allowed CIDR lets it out.
- slot firewall (firewall.go): under default_deny the non-DNS UDP rule is
  forced to drop mode and a terminal non-TCP drop is installed after the
  established/allow-set accepts, so UDP (DNS-tunnel vector) and other non-TCP
  not explicitly allowed are dropped instead of hitting the ACCEPT policy.
  TCP is never matched there (it leaves via the REDIRECT to the proxy).

Tests: isEgressAllowed cases for default_deny (drop-by-default, allow-domain
/allow-CIDR still pass, no-SNI dropped) and denied_domains (block under
default-allow, allow beats deny); an nft netns test asserting default_deny
records posture=deny, forces the UDP mode to Drop, and installs the terminal
non-TCP drop as the chain's last rule. go build/vet clean; existing network
and tcpfirewall suites pass.

Closes e2b-dev#3696.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: opt-in default-deny egress (with control-plane allowlist) for sandboxes

2 participants