Skip to content

Ring: don't publish instance state before tokens are generated - #7901

Open
danielblando wants to merge 1 commit into
cortexproject:masterfrom
danielblando:fix/ring-tokenless-instance-never-recovers
Open

danielblando wants to merge 1 commit into
cortexproject:masterfrom
danielblando:fix/ring-tokenless-instance-never-recovers

Conversation

@danielblando

Copy link
Copy Markdown
Contributor

What this PR does:

An ingester can end up registered in the ring owning no tokens, and never recover. It serves no
writes, serves no reads, and never passes its readiness check. Restarting does not help. Getting
into this state needs -ingester.tokens-generator-strategy=minimize-spread, which is not the
default, but once an instance is in it, it cannot get out regardless of the strategy.

How it happens:

  1. autoJoin sets the instance to its target state at the top of its CAS callback, before
    generating any tokens.
  2. Under minimize-spread, generation can legitimately come back empty. The generator admits only
    the earliest-registered tokenless instance in a zone and returns nothing to the rest when
    force is false, which is how autoJoin calls it. The default random generator always
    returns the full count, so this branch is unreachable there.
  3. The callback returns a retryable error, but the instance is already in the target state owning
    no tokens. The heartbeat writes i.GetState() to the ring on every tick, so that combination
    gets persisted. The callback re-runs on each CAS retry, so this is not a brief race.
  4. On the next startup, initRing finds the entry and adopts its state from the ring.
  5. autoJoin only runs while the instance is PENDING, on a timer that fires once per process, so
    it is never reached again. The restart is what re-adopts the bad state, which is why restarting
    does not clear it.

Under minimize-spread the damage is not confined to that instance. It holds the head of its
zone's queue, so no other tokenless instance in the zone can generate either. We hit this on a cell
where 13 ingesters in one zone went tokenless and none recovered. The two oldest sat that way for
about 19 hours with zero restarts, because they never got far enough to fail.

The fix:

  • autoJoin sets the state only after tokens are secured, on both paths that publish the instance.
    This closes the window rather than narrowing it. What gets written to the ring is unchanged:
    setState now runs immediately before the existing i.GetState() read on each success path, and
    nothing between the old and new position reads the instance state.
  • initRing resets an instance found in the ring without tokens back to PENDING, mirroring the
    JOINING reset directly above it. Instances already stuck recover on restart, instead of needing
    the ring edited by hand. Forgetting such an entry only sticks while the process is down, since a
    live one re-creates it on its next heartbeat.

Which issue(s) this PR fixes:
Fixes #7899

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

@danielblando
danielblando requested a review from a team as a code owner October 28, 2026 20:58
@danielblando
danielblando requested a review from yeya24 October 28, 2026 20:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingester stuck in LEAVING when its ring entry has 0 tokens

1 participant