Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/project-status-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,13 @@ When project IDs are supplied, the response includes only results for those IDs,

With no project IDs supplied, the response lists recorded project results by descending breakage degree, then ascending project ID, excluding the repository fact. If the result for the cursor project is missing, the API returns an error. Results are a live view; restart pagination to discover facts recorded before the cursor.

Each project result is an immutable validation fact keyed by queue, commit URI, and project ID. A result must identify the selected request. `project_results_complete` is false until the implementation has durably finished producing results for the supplied project IDs; a completed empty set is valid.
Each project result is an immutable validation fact keyed by queue, commit URI, and project ID. A result must identify the selected request. `project_results_complete` is false until the implementation has durably finished producing results for the supplied project IDs; a completed empty set is valid. Abandoned recording also completes result production; missing project results remain unknown, and `record_abandoned` history explains the abandonment.

Callers use `project_results_complete`, rather than the presence or absence of an individual project result, to determine whether the implementation has finished producing results for the requested projects. `next_page_token` is empty only on the final page. A terminal lifecycle notification is emitted only after the repository result and any applicable project completion record are durable. If notification delivery fails, consumers can recover the same result through either endpoint.

## Implementation

Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and recorded facts for requested projects in paginated request order, or recorded project results by descending breakage degree then ascending project ID when none are requested. `project_results_complete` is true when the durable request summary has a positive `project_results_completed_at_ms`, projected from the retained `project_facts_recorded` event. The record stage writes that event after all project facts and ordering mappings, including a successfully resolved empty set, and before publishing its terminal hook. The materializer persists the completion timestamp before the record stage continues, without changing lifecycle state or request version. Duplicate delivery repairs a failed projection using the original event timestamp, and subsequent state projections preserve completion. The API reads the summary before loading facts and includes its completion timestamp in `updated_at_ms`, without an additional event lookup. Missing completion remains false. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page.
Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and recorded facts for requested projects in paginated request order, or recorded project results by descending breakage degree then ascending project ID when none are requested. `project_results_complete` is true when the durable request summary has a positive `project_results_completed_at_ms`, projected from a retained `project_facts_recorded` or `record_abandoned` event. The first projected completion timestamp is preserved when both events exist. Abandonment leaves recorded facts and project counts unchanged. The record stage writes `project_facts_recorded` after all project facts and ordering mappings, including a successfully resolved empty set. The record DLQ retains `record_abandoned` when production stops. Both paths persist completion before publishing a terminal hook. The materializer persists the completion timestamp before the record stage continues, without changing lifecycle state or request version. Duplicate delivery repairs a failed projection using the original event timestamp, and subsequent state projections preserve completion. The API reads the summary before loading facts and includes its completion timestamp in `updated_at_ms`, without an additional event lookup. Missing completion remains false. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page.

A dedicated ordering mapping makes listing a bounded primary-key range read, not a sort over all facts. Listing reads at most `page_size + 1` ordering mappings and hydrates only the returned facts, plus the cursor fact on continuations. The continuation token identifies the last project; its immutable fact supplies the degree needed to resume this ordering.

Expand Down
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/request-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ For a Request transition, the controller:

Request creation, Build changes, and fact creation use the same source-write, log-write, dependent-publish ordering. A request-log outage can leave a source update visible, but it cannot allow dependent processing to move past an unrecorded transition.

The `project_facts_recorded` event also projects its retained timestamp into `RequestSummary.ProjectResultsCompletedAtMs`. This immutable fact is independent of lifecycle state and request version; state updates preserve it, and optimistic locking retries concurrent projection updates. Completion requires an existing lifecycle summary rather than fabricating state from the event. A failed projection fails persistence, and redelivery reconciles the retained occurrence before retrying the summary update. Other non-state events do not update the summary.
The `project_facts_recorded` and `record_abandoned` events project completion of project-result production into `RequestSummary.ProjectResultsCompletedAtMs`. Abandonment ends production without claiming that all project facts were established or changing recorded project counts. The first projected completion timestamp is immutable, including when both events are retained; successful recording still projects its project counts. Completion is independent of lifecycle state and request version; state updates preserve it, and optimistic locking retries concurrent projection updates. Completion requires an existing lifecycle summary rather than fabricating state from the event. A failed projection fails persistence, and redelivery reconciles the retained occurrence before retrying the summary update. Other non-state events do not update the summary.

### Controller ownership

Expand All @@ -219,7 +219,7 @@ The `project_facts_recorded` event also projects its retained timestamp into `Re
| Build | Create Build after runner acceptance, then retain `build_triggered`. | An identical existing Build ensures the event before buildsignal publication. |
| Buildsignal | Persist terminal Build and retain `build_finished`; CAS the Request outcome and retain its terminal state. | Existing terminal Build and Request outcome each ensure their own entry before record publication. |
| Record | Create or verify the whole-repository fact, then retain `validation_fact_recorded`; after recording project facts, retain one `project_facts_recorded` event. | An identical whole-repository fact and project-fact batch ensure their events before bookmark or promotion work. |
| Record DLQ | Retain `record_abandoned`; for failed Requests, create or verify the repository fact and retain `validation_fact_recorded` before publishing its hook. | Existing facts remain authoritative; stable log and hook IDs make replay idempotent without resolving projects or replaying promotion. |
| Record DLQ | Retain `record_abandoned` and project the end of project-result production; for failed Requests, create or verify the repository fact and retain `validation_fact_recorded` before publishing its hook. | Existing facts remain authoritative; stable log and hook IDs make replay idempotent without resolving projects or replaying promotion. |
| Build analysis DLQ | Retain `build_analysis_abandoned` without rerunning detection or changing validation state. | One stable event per build deduplicates recovery deliveries; only state reads and history retention retry. |
| Issue tracking DLQ | Protect the current cursor from delayed writes, retain its issue history and `issue_tracking_abandoned`, then mark the pass abandoned. | A stable hash of the notification ID deduplicates the event. Recovery retains accepted issue changes and never processes the remaining range. |
| Reconciler | CAS an unrecoverable non-terminal Request to failed, then retain failed. | An existing terminal Request is repaired from its persisted outcome without relabeling it. |
Expand Down
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/steps/record.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,9 @@ Every primary subscription comes from `DefaultSubscriptionConfig`, which enables

Two different things put a message there, and only one is a poison payload. A delivery that fails with its retry budget spent is dead-lettered by the nack itself, carrying the reason it actually failed. A delivery that never reaches a nack, because it crashed or because its **ack failed** and the visibility timeout redelivered it, is dead-lettered by the poll loop once `retry_count` reaches `MaxAttempts` (3 by default), without the controller running on that final attempt and with only a generic reason recorded. So a missing reconciler exposes more than malformed messages: a fact can be lost to a storage failure that would have succeeded on a later retry, or to an ack that never landed even though the write did.

The reconciler retains a stable `record_abandoned` event. For a Request already in `failed`, it creates a `DegreeBroken` whole-repository fact if absent, retains `validation_fact_recorded` history, and publishes `validation.repository.recorded` after both writes succeed. An existing fact owned by the same Request is adopted unchanged, including its degree and timestamp; a fact owned by another Request is an invariant error. Replays repair missing history and repeat the notification with the same event ID.
The reconciler retains a stable `record_abandoned` event and durably projects completion of project-result production before continuing. For a Request already in `failed`, it creates a `DegreeBroken` whole-repository fact if absent, retains `validation_fact_recorded` history, and publishes `validation.repository.recorded` after both writes succeed. An existing fact owned by the same Request is adopted unchanged, including its degree and timestamp; a fact owned by another Request is an invariant error. Replays repair missing history and repeat the notification with the same event ID.

This fallback does not resolve project attribution or mark project recording complete. A failed build establishes a repository failure even when no individual project verdict is known. Build-issue analysis and detectors remain independent. The reconciler does not change the Request state, advance the bookmark, promote, or release build slots. Other Request states retain abandonment history only, so a succeeded build whose fact could not be recorded still appears unvalidated.
This fallback does not resolve project attribution. Completion means production has stopped; missing project results remain unknown, and abandonment history distinguishes this path from successful project recording. A failed build establishes a repository failure even when no individual project verdict is known. Build-issue analysis and detectors remain independent. The reconciler does not change the Request state, advance the bookmark, promote, or release build slots. Other Request states retain abandonment and project-result completion without creating a repository fact, so a succeeded build whose fact could not be recorded still appears unvalidated.

The DLQ consumer uses `errs.AlwaysRetryableProcessor`: Request reads, fact writes, history persistence, and notification publication retry until recovery succeeds. Malformed payloads, invalid or unresolvable queue identities, queue mismatches, and missing Requests are logged, counted, and acknowledged.

Expand Down
4 changes: 3 additions & 1 deletion stovepipe/core/requestlog/materializer.go
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,9 @@ func (m *materializer) PersistLog(ctx context.Context, stores storage.Storage, l
return err
}

if retained.State == entity.RequestStateUnknown && retained.Event != entity.RequestEventProjectFactsRecorded {
if retained.State == entity.RequestStateUnknown &&
retained.Event != entity.RequestEventProjectFactsRecorded &&
retained.Event != entity.RequestEventRecordAbandoned {
return nil
}
if err := materializeRequestProjections(ctx, stores, retained); err != nil {
Expand Down
1 change: 0 additions & 1 deletion stovepipe/core/requestlog/materializer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,6 @@ func TestMaterializerRetainsOtherEventsWithoutProjectingSummary(t *testing.T) {
entity.RequestEventBuildTriggered,
entity.RequestEventBuildFinished,
entity.RequestEventValidationFactRecorded,
entity.RequestEventRecordAbandoned,
entity.RequestEventBuildAnalysisAbandoned,
entity.RequestEventIssueTrackingAbandoned,
} {
Expand Down
15 changes: 10 additions & 5 deletions stovepipe/core/requestlog/request_summary.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,17 @@ func projectRequestSummaryFacts(summary entity.RequestSummary, log entity.Reques
log.RequestID, summary.RequestID, log.Queue, summary.Queue,
)
}
if log.Event == entity.RequestEventProjectFactsRecorded {
if log.TimestampMs <= 0 || (summary.ProjectResultsCompletedAtMs != 0 && summary.ProjectResultsCompletedAtMs != log.TimestampMs) {
return entity.RequestSummary{}, fmt.Errorf("request summary completion time conflicts with retained log request_id=%q", log.RequestID)
if log.Event == entity.RequestEventProjectFactsRecorded || log.Event == entity.RequestEventRecordAbandoned {
if log.TimestampMs <= 0 {
return entity.RequestSummary{}, fmt.Errorf("request summary completion time must be positive request_id=%q", log.RequestID)
}
// Recording and abandonment can both be retained; the first projected completion stays authoritative.
if summary.ProjectResultsCompletedAtMs == 0 {
summary.ProjectResultsCompletedAtMs = log.TimestampMs
}
if log.Event == entity.RequestEventProjectFactsRecorded {
return projectRequestSummaryProjectCounts(summary, log)
}
summary.ProjectResultsCompletedAtMs = log.TimestampMs
return projectRequestSummaryProjectCounts(summary, log)
}
if log.State == entity.RequestStateAccepted {
if summary.AcceptedAtMs != 0 && summary.AcceptedAtMs != log.TimestampMs {
Expand Down
Loading
Loading