Repository navigation
[OTel]: Report terminal failures and keep exception events within span bounds #853
Description
Split from PR #758 — preserve replay completion telemetry, using reference head 0d407e1. The reference contains multiple concerns and outstanding review feedback; use its relevant implementation and tests as a starting point, not as a patch to merge wholesale.
Baseline inspected: main at 2e5135fc6f6c05024458e0be333a8325612abd6c. This issue is based on source and PR-review inspection; no runtime reproduction was executed when filing it. The scenarios below are the implementation and validation work to perform.
Problem and reproduction
Both OTel views currently set ERROR when an error object is present, but leave FAILED, TIMED_OUT, and STOPPED operations UNSET when diagnostic details are absent. Separately, record_exception() uses the current clock unless a timestamp is supplied, so a delayed/replayed completion can produce an exception event later than its exported span's end.
Use an in-memory exporter with each OTel view separately. Deliver terminal operation information with and without error details; also deliver an error with a backend end time earlier than the current clock, including a checkpointless virtual child with no durable start time. Inspect span status, durable.operation.status, and exception-event timestamps.
Required behavior
| Delivered operation outcome | Error details | OTel status | Exception event |
|---|---|---|---|
| SUCCEEDED | absent | OK | none |
| FAILED / TIMED_OUT / STOPPED | absent | ERROR | none |
| CANCELLED and other non-success statuses | absent | UNSET | none |
| Any outcome carrying an error | present | ERROR | preserve available details |
Use the same normalized end timestamp for the exception event and its operation/attempt span. Preserve each view's existing ordering and start/end normalization. Do not invent an exception message, type, or stack solely because the status indicates failure.
Acceptance criteria
- Cover the full status/error matrix in both Execution and Invocation views.
- Keep the original durable status attribute unchanged, including CANCELLED.
- Operation and attempt exception events satisfy
span.start_time <= event.timestamp <= span.end_time; for these terminal events, use the normalized span-end timestamp. - Cover absent start times, delayed completion delivery, equal start/end times, and caught virtual-child failures.
- A handler that catches an invoke/child failure and returns successfully exports a failed operation but successful Invocation/Workflow spans. Repeat across a durable wait and replay, asserting saved work is not repeated.
- Preserve
InvocationStatus.RETRY->durable.invocation.status=RETRYINGwith UNSET. This issue does not change invocation outcome semantics. - Update the OTel README's status table and timestamp explanation, and explicitly identify the changed expectations in
test_status_mapping.py.
Code and reference tests
Change terminal operation and user-function/attempt handling in execution_plugin.py and invocation_plugin.py, plus the OTel README.
Independence and compatibility
Implement against existing natural hooks on main; no pending-completion queue, exporter retry machinery, flush worker, or core snapshot change is required. When the export-recovery issue lands, retries must preserve the timestamps chosen here.
This deliberately refines the no-error status rule from the closed #750 / PR #752 (UNSET -> ERROR for FAILED, TIMED_OUT, and STOPPED); it does not reopen their already-fixed RETRYING mapping. Confirm the shared telemetry contract and document the compatibility impact. Original exception-type preservation is separately tracked in #804; avoid regressing any fix that lands for it.
Conformance qualification remains in #796 and shared conformance PR #131. Coordinate any shared requirement changes there; do not add requirement IDs or copy conformance handlers into this issue.
Validation
From the repository root:
hatch run dev-otel:test
hatch run dev-otel:typecheckRun hatch fmt --check from packages/aws-durable-execution-sdk-python-otel. Add focused unit tests and public-handler integration coverage for the behavior described above. For OTel changes, also build the core and OTel wheels with hatch build in their package directories and run hatch run test-wheel-otel:test from the root as documented in CONTRIBUTING.md. Use existing test environments; this split does not require testing-library or CI-filter changes. Document any changed existing test expectations and their rationale in the implementing PR.
Activity