Repository navigation
Conversation
…lient-closed The reverse proxy (shared by client-proxy and the orchestrator sandbox proxy) had no request-outcome metric — only connection gauges. There was no way to tell, on a dashboard, whether a proxied request failed because the destination could not be resolved (500), the node was unreachable (502), the client went away (499), or the sandbox was not found. During a node scale-down this is the difference between stale routing record and real routing bug, and it could only be recovered by scraping logs. Add a request-outcome counter, recorded at the points where a request's fate is decided: - handler() pre-routing terminal branches: client_closed (ctx ended before routing), sandbox_not_found, too_many_connections, upstream_unreachable (owner acquire failed), routing_failed (unexpected resolve error). - pool ErrorHandler: client_closed (context.Canceled / ECONNRESET / EOF) vs upstream_unreachable (everything else), and routing_failed when the destination is missing from context. - ModifyResponse: ok (the upstream answered). The shared proxy stays telemetry-agnostic: it classifies and calls an OnRequestOutcome(ctx, outcome) callback (mirroring the existing ConnectionLimitConfig callbacks). Each service owns its counter and name: client_proxy.request.outcome and orchestrator.proxy.request.outcome, labeled outcome = ok | routing_failed | upstream_unreachable | client_closed | sandbox_not_found | too_many_connections. Fixes e2b-dev#3757
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 28, 2026 09:18
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #3757.
The reverse proxy in
packages/shared/pkg/proxy(shared by client-proxy and the orchestrator sandbox proxy) has no request-outcome metric — only connection gauges (*.proxy.*.connections.open,*.proxy.pool.size). There is no way to tell on a dashboard whether a proxied request failed because the destination could not be resolved (500), the node was unreachable (502), the client went away (499), or the sandbox was not found.This is the blind spot behind node-churn incidents: during a scale-down, sandboxes on removed nodes surface as
Failed to route request to sandbox, and because routing failures (500) and upstream-unreachable failures (502) are indistinguishable at the metric layer, the only way to separate them is scraping logs. #3691 / #3692 split the message; this adds the metric so the split is visible and alertable (a spike inupstream_unreachableby sandbox = stale routing pointing at a drained node).Changes
Record a request outcome at the points where a request's fate is decided:
handler()pre-routing terminal branches →client_closed(ctx ended before routing),sandbox_not_found,too_many_connections,upstream_unreachable(owner acquire failed),routing_failed(unexpected resolve error).ErrorHandler→client_closed(context.Canceled/ECONNRESET/io.EOF) vsupstream_unreachable(dial/transport failure, closed sandbox port);routing_failedwhen the destination is missing from context.ModifyResponse→ok(the upstream answered).The shared proxy stays telemetry-agnostic: it classifies and calls an
OnRequestOutcome(ctx, outcome)callback threaded throughreverseproxy.New→pool.New(mirroring the existingConnectionLimitConfigcallbacks). Each service owns its counter + name:client_proxy.request.outcomeorchestrator.proxy.request.outcomelabeled
outcome = ok | routing_failed | upstream_unreachable | client_closed | sandbox_not_found | too_many_connections.Scope note: validation/capacity/resume/token error pages in
handler()are deliberately not folded into this counter — they are a separate concern with their own signals (resume-flight, capacity logs). This counter is scoped to the routing/transport outcome, which is the incident-relevant blind spot.Tests
outcome_test.go: end-to-end —ok(live backend),upstream_unreachable(dial to a closed port),sandbox_not_found(resolver returns the typed error), asserted via a recording callback.reverseproxy.New/pool.Newcall sites (incl. tests) for the new parameter.go test ./pkg/proxy/...and the telemetry desc/unit completeness test pass;go vet+golangci-lintclean on all changed packages; client-proxy and orchestrator build.