Skip to content

fix: correct terminal telemetry status and timing - #865

Open
zhongkechen wants to merge 4 commits into
mainfrom
codex/fix-853-otel-terminal-status
Open

zhongkechen wants to merge 4 commits into
mainfrom
codex/fix-853-otel-terminal-status

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available

Fixes #853.

Description of changes

Operations ending FAILED, TIMED_OUT, or STOPPED without diagnostic details now
produce ERROR spans in both OTel views. Supplied operation and attempt exceptions
are recorded at the same normalized timestamp used to end their span, so delayed
completion delivery cannot put the event after the span. A failed attempt without
diagnostics does not fabricate an exception event.

CANCELLED without an error remains UNSET, RETRY remains RETRYING/UNSET, and a caught
operation failure does not make a successful Invocation or Workflow span fail.
Existing exception type/message/stack preservation is retained. The README
documents the refined status mapping and timing behavior.

Design

  • Design note (rule 3): not needed; the existing lifecycle hooks and factory API
    are unchanged. The issue specifies the refined telemetry outcome contract.
  • Simpler options considered (rule 2): reuse each view's existing end-time
    normalization and the existing exception recorder. Replacing the timestamp
    normalization or adding another terminal-delivery path would duplicate logic.
    Merely changing status would leave exception events outside delayed spans.
  • Root cause (rule 5): status was inferred from diagnostic presence; exception
    recording independently chose the current clock while span completion used
    durable or invocation-normalized time. The exception helper also synthesized
    diagnostics for errorless failed attempts.
  • Net change: +31 source lines across three existing modules; no new public API
    members or configuration options.

Validation

  • The first commit adds regressions. Against unmodified main: 48 failed,
    120 passed
    in the focused status/timing and public-handler suite. Failures
    demonstrate missing ERROR status, fabricated events for errorless attempts,
    and exception events differing from or falling after their span end.
  • After the fix: the same focused suite 168 passed.
  • hatch run dev-otel:test -q --tb=short: 865 passed.
  • hatch run dev-otel:typecheck: no issues in 45 source files.
  • Package hatch fmt --check: passed.
  • Core and OTel hatch build: passed.
  • hatch run test-wheel-otel:test: 865 OTel tests and 12 lifecycle tests
    passed
    , after verifying built core/OTel source bytes and artifact hashes.
  • Python 3.13.17 and CI-pinned Hatch 1.16.5.

The public-handler tests cover caught checkpointed child, virtual child, and
chained-invoke failures, with and without a durable wait/replay. They assert
successful Invocation/Workflow status, bounded events, saved step execution
exactly once, and reuse of stored nonvirtual failure outcomes.

Existing expectation change: test_operation_status_and_exception now expects
ERROR, rather than UNSET, for FAILED/TIMED_OUT/STOPPED without an error. It still
requires no exception event and preserves the original durable status attribute.
The matrix is expanded to include STARTED/PENDING/READY controls. Other added
tests do not replace existing behavioral expectations.

CI follow-up

All four ADOT/X-Ray and Community/S3 jobs
each reported 19 passing cases, case 11 failing, and seven uncovered requirements.
The exact same case-11 history mismatch and counts occur on
main at d7ae955.
The handler/coverage repair remains #864/#796; this PR does not claim case-25
cloud qualification.

The cloud-example workflow
failed in Python 3.11/3.13 because GetDurableExecutionHistory was throttled after
four retries. The affected handlers have no OTel plugin and their polling code is
unchanged. Python 3.12/3.14 passed. No #853-caused regression was found in these
failures.

Combined local validation of all six scoped fixes passed 5,058 core/OTel/testing
tests. Merge coordination for the shared terminal renderers is recorded in #869.

Assumptions

  • The shared case-25 requirement requires an errorless failed callback to be
    ERROR while the caught workflow succeeds; see the
    shared execution requirement.
  • Explicit OTel exception timestamps are verified against a real TracerProvider,
    SimpleSpanProcessor, and InMemorySpanExporter in both views.
  • Deployed Lambda/backend conformance qualification is unverified here and
    remains in test(otel): cover replay and chained execution #796. No conformance requirement or handler is changed.

Decisions not covered by the issue or the design note

Checklist

  • Every new branch has a test that fails without it (rule 4).
  • The first commit adds failing regressions (rule 5).
  • Every edited existing test expectation is listed above.

By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

@zhongkechen
zhongkechen marked this pull request as ready for review October 27, 2026 23:18
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:19 — with GitHub Actions Active
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:19 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:41 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings after reviewing all six changed files against CONTRIBUTING.md rules 1–9. Tests were inspected but not executed. Deployed Lambda/exporter behavior remains unverified.

Reviewed commit 3d77e65eec89753af67d56b2b3765c1d60d992bd. Workflow run

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 3d77e65e Deployed Oct 27, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #2544
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OTel]: Report terminal failures and keep exception events within span bounds

1 participant