Skip to content

feat(proxy): request-outcome metric distinguishing routing (500) / upstream (502) / client-closed (499) - #3758

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/proxy-request-outcome-metric
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/proxy-request-outcome-metric

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Problem

Fixes #3757.

The reverse proxy in packages/shared/pkg/proxy (shared by client-proxy and the orchestrator sandbox proxy) has no request-outcome metric — only connection gauges (*.proxy.*.connections.open, *.proxy.pool.size). There is no way to tell on a dashboard whether a proxied request failed because the destination could not be resolved (500), the node was unreachable (502), the client went away (499), or the sandbox was not found.

This is the blind spot behind node-churn incidents: during a scale-down, sandboxes on removed nodes surface as Failed to route request to sandbox, and because routing failures (500) and upstream-unreachable failures (502) are indistinguishable at the metric layer, the only way to separate them is scraping logs. #3691 / #3692 split the message; this adds the metric so the split is visible and alertable (a spike in upstream_unreachable by sandbox = stale routing pointing at a drained node).

Changes

Record a request outcome at the points where a request's fate is decided:

  • handler() pre-routing terminal branches → client_closed (ctx ended before routing), sandbox_not_found, too_many_connections, upstream_unreachable (owner acquire failed), routing_failed (unexpected resolve error).
  • pool ErrorHandler → client_closed (context.Canceled / ECONNRESET / io.EOF) vs upstream_unreachable (dial/transport failure, closed sandbox port); routing_failed when the destination is missing from context.
  • ModifyResponse → ok (the upstream answered).

The shared proxy stays telemetry-agnostic: it classifies and calls an OnRequestOutcome(ctx, outcome) callback threaded through reverseproxy.New → pool.New (mirroring the existing ConnectionLimitConfig callbacks). Each service owns its counter + name:

  • client_proxy.request.outcome
  • orchestrator.proxy.request.outcome

labeled outcome = ok | routing_failed | upstream_unreachable | client_closed | sandbox_not_found | too_many_connections.

Scope note: validation/capacity/resume/token error pages in handler() are deliberately not folded into this counter — they are a separate concern with their own signals (resume-flight, capacity logs). This counter is scoped to the routing/transport outcome, which is the incident-relevant blind spot.

Tests

  • outcome_test.go: end-to-end — ok (live backend), upstream_unreachable (dial to a closed port), sandbox_not_found (resolver returns the typed error), asserted via a recording callback.
  • Updated all existing reverseproxy.New / pool.New call sites (incl. tests) for the new parameter.
  • go test ./pkg/proxy/... and the telemetry desc/unit completeness test pass; go vet + golangci-lint clean on all changed packages; client-proxy and orchestrator build.

…lient-closed

The reverse proxy (shared by client-proxy and the orchestrator sandbox proxy)
had no request-outcome metric — only connection gauges. There was no way to
tell, on a dashboard, whether a proxied request failed because the destination
could not be resolved (500), the node was unreachable (502), the client went
away (499), or the sandbox was not found. During a node scale-down this is the
difference between stale routing record and real routing bug, and it could
only be recovered by scraping logs.

Add a request-outcome counter, recorded at the points where a request's fate is
decided:
- handler() pre-routing terminal branches: client_closed (ctx ended before
  routing), sandbox_not_found, too_many_connections, upstream_unreachable
  (owner acquire failed), routing_failed (unexpected resolve error).
- pool ErrorHandler: client_closed (context.Canceled / ECONNRESET / EOF) vs
  upstream_unreachable (everything else), and routing_failed when the
  destination is missing from context.
- ModifyResponse: ok (the upstream answered).

The shared proxy stays telemetry-agnostic: it classifies and calls an
OnRequestOutcome(ctx, outcome) callback (mirroring the existing
ConnectionLimitConfig callbacks). Each service owns its counter and name:
client_proxy.request.outcome and orchestrator.proxy.request.outcome, labeled
outcome = ok | routing_failed | upstream_unreachable | client_closed |
sandbox_not_found | too_many_connections.

Fixes e2b-dev#3757

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proxy: no request-outcome metric to distinguish routing (500) / upstream (502) / client-closed (499) failures

2 participants