Skip to content

fix: auto-start hangs in pg_ctl start; reset password for POSTGRES_USER - #57

Open
yashmehrotra wants to merge 3 commits into
mainfrom
fix/pg-ctl-start-logfile
Open

yashmehrotra wants to merge 3 commits into
mainfrom
fix/pg-ctl-start-logfile

Conversation

@yashmehrotra

Copy link
Copy Markdown
Member

Problem

postgres-cli auto-start hangs forever whenever it starts a temporary server. That happens with --auto-reset-password and on major-version upgrades. In Kubernetes the postgres-upgrade init container never finishes, so the pod stays at Init:1/2.

The last log line is:

Killing process: /usr/lib/postgresql/17/bin/pg_ctl [...] with timeout 5s

Start() runs pg_ctl start without -l. The background postgres server inherits pg_ctl's stdout and keeps that pipe open. cmd.Wait() therefore never returns. Since clicky v1.21.40 the timeout path also waits for cmd.Wait() after killing pg_ctl, so the CLI blocks forever. Images built against clicky v1.21.1 didn't wait, so the bug was hidden.

A second bug sat behind the hang. The entrypoint runs initdb -U $POSTGRES_USER, but the CLI's username defaults to PG_USER or postgres. With a non-default user (e.g. POSTGRES_USER=xero), the password reset fails with pq: role "postgres" does not exist.

Changes

  • fix(server): Start() passes -l $TMPDIR/postgres.log to pg_ctl start, so the server no longer holds the CLI's pipe. When start fails or times out, the server log is printed so startup errors stay visible. Also fixes a fmt.Println that printed a literal %w.
  • fix(cli): the default --username falls back to PG_USER, then POSTGRES_USER, then postgres, matching the user the entrypoint initialises the cluster with.
  • test(server): the env tests now restore the environment after os.Clearenv(). They were wiping PATH for the rest of the package, so tests that need pg_ctl skipped silently.

Behaviour change

On a successful start, the temporary server's log now goes to $TMPDIR/postgres.log instead of the CLI's output. The main server's logging is unchanged: the entrypoint still execs postgres directly.

Testing

  • New TestStartReturnsWhileServerKeepsRunning runs initdb and Start() against a real cluster on a private port and socket dir.
    • Before the fix it hangs the same way as in production, ending on the same Killing process line.
    • After the fix it passes in about 0.6s.
  • New TestDefaultUsername covers the precedence order.
  • Checked end to end with Docker, replaying the Kubernetes init-then-main container sequence twice with a changed password. Init exits 0, the new password logs in over the network, and the old one is rejected.

Without -l the daemonised postgres inherits pg_ctl's stdout and holds the
pipe open, so the timed pg_ctl run never returns and auto-start hangs on
--auto-reset-password and major-version upgrades. Print the server log
when start fails or times out so startup errors stay visible.
The env tests cleared PATH for the rest of the package, so tests that
need pg_ctl on PATH were silently skipped.
docker-entrypoint.sh runs initdb with POSTGRES_USER, so with a
non-default user the "postgres" role does not exist and the password
reset fails.
@yashmehrotra
yashmehrotra requested a review from moshloop October 27, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant