Skip to content

Report audit retention metrics - #6059

Open
warwickschroeder wants to merge 9 commits into
masterfrom
warwick/audit-retention-telemetry
Open

warwickschroeder wants to merge 9 commits into
masterfrom
warwick/audit-retention-telemetry

Conversation

@warwickschroeder

Copy link
Copy Markdown
Contributor

What this adds

The audit instance's SQL Server and PostgreSQL retention sweep now emits retention instruments.

  • sc.retention.cycle_duration_seconds and sc.retention.consecutive_failures_total on the audit meter, with the same descriptions, buckets and result values as the error instance.
  • No rows_deleted_total on the audit instance, because PostgreSQL drops whole partitions and a dropped partition reports no row count.
  • One isolated pass per table, tagged retention.entity as audit_messages or saga_snapshots, so a failing table no longer stops the other.
  • A pass that leaves rows because its table is too busy records failed, as the Audit retention check already counts it.
  • A sweep that fails before its passes records failed for both, the way an unreachable database shows on the error instance.
  • Docs in the Retention section of docs/telemetry.md.

Tests

  • ServiceControl.Audit.Persistence.Tests/EFCore/AuditRetentionMetricsTests (new): the instrument names, kinds, unit, buckets and meter, and how a cycle records success, failure and shutdown.
  • ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionMetricsTests (new): on SQL Server and PostgreSQL, a sweep records one result per table, keeps the passes isolated, reports deferral, early failure and shutdown, and records nothing when skipped for the lock.
  • ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests: a failing pass in either table turns the check red after three sweeps.

@johnsimons johnsimons left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, the only thing that may be nice is to also update the dashboard of Grafana in https://github.com/Particular/ServiceControl/blob/master/tools/testing-tool/TestingTool.AppHost/obs/grafana/dashboards/testing-tool.json

@warwickschroeder

Copy link
Copy Markdown
Contributor Author

Looks good, the only thing that may be nice is to also update the dashboard of Grafana

Will add to this PR. Thanks

@rbev
rbev enabled auto-merge (squash) October 30, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants