[TakGN]

Part of ipni/infra#163.

What

route-origin's Envoy (k8s/route-origin/envoy.yaml.j2) sends OpenTelemetry traces to ipni/infra's trace collector on the box's loopback (127.0.0.1:4317, kubernetes/platform/alloy-traces). The collector turns every span into latency metrics, keeps failed traces, traces over 1s and a tenth of the rest, and sends those to Tempo on lith-1. In Grafana they show on the Latency dashboard ("From traces") and in Explore.

• tracing on the connection manager: 5% of requests (random_sampling), about 7 traces/s per box at the fleet's ~424 req/s. Each sampled request gets Envoy's span plus, with spawn_upstream_span, one for the call to needle, so a trace shows which instance served it and how long it took. service_name: route-origin.
• Span names come from a decorator on each route: lookup, ipns-put, version, not-found, misdirected. Never the path: a path carries a CID, and every span name becomes a metric series.
• Clients can't force or continue a trace. client_sampling: 0 ignores x-client-trace-id, and an early header mutation drops traceparent and tracestate before Envoy reads them. (x-envoy-force-trace is already ignored from outside, as Envoy strips x-envoy-* from external requests.)
• Access log gets trace_id. Grafana's Loki data source (ipni/infra#246) turns it into a link to the trace. It is set on every request; only the sampled 5% (that survive tail sampling) are stored.
• otel_collector cluster: 127.0.0.1:4317, HTTP/2. If the collector is down, Envoy drops the spans; requests are unaffected.

Not in this PR: needle

The plan in ipni/infra#163 was to turn on needle's own traces with OTEL_* and NEEDLE_SAMPLING_FRACTION. That doesn't work with needle v1.0.0, so it's left out rather than paying for a needle rollout:

• NEEDLE_SAMPLING_FRACTION never samples. needle's root sampler only samples spans whose name starts with needle, but with otelhttp v0.70 (which v1.0.0 builds against) the request span is named after the method and route (GET /), not needle.request. Tested with the pinned image at fraction 1: no spans.
• With NEEDLE_TRACING_AUTH and an authorised traceparent, needle does export, but only one GET / span per request, with no breakdown by router.

That needs a fix in ipni/needle before needle is worth tracing. Until then needle ignores the traceparent Envoy sends it (no NEEDLE_TRACING_AUTH), so needle's side of a request is only Envoy's upstream span.

Merge order

1. Reserve the trace collector's ports #17 (reserves the collector's ports), applied.
2. ipni/infra#246 (Tempo and the collector), synced by Argo CD.
3. This PR, then roll it out.

If this is rolled out before the collector exists, Envoy just fails to send spans; nothing else changes.

Rollout

routing.yml isn't applied from CI. The ConfigMap hash changes, so the role rolls Envoy on each box, one box at a time with production_rollout:

ansible-playbook routing.yml --tags origin --check --diff
ansible-playbook routing.yml --tags origin

Afterwards, on any box, the access log should carry trace_id, and Envoy should report spans sent with no failures:

ssh [email protected] "sudo k3s kubectl -n route-origin logs deploy/route-origin --tail=3 | grep -c trace_id; curl -s 127.0.0.1:9901/stats | grep -E 'tracing.opentelemetry.(spans_sent|timer_flushed)|cluster.otel_collector.upstream_rq_(2xx|5xx)'"

Checked

• Rendered the template with the four instances and ran envoy --mode validate on the pinned image (distroless-v1.39.1): OK.
• Ran that Envoy, with sampling raised to 100% in a test copy, next to the collector, Tempo and Prometheus from ipni/infra#246 in Docker:
  • spans arrive named lookup, version and not-found, plus router needle egress for the upstream call, with service.name=route-origin;
  • the access log carries trace_id;
  • requests carrying an outside traceparent (and x-client-trace-id) got fresh trace IDs, so the strip works.
• ansible-playbook --syntax-check routing.yml.
• Not run on a box. The --check --diff above is the first look at the real render.

🤖 Generated with Claude Code