Repository navigation
Log kubo at info, in JSON, on the bootstrappers - #16
Merged
Merged
Conversation
kubo logs only errors by default. ProbeLab asked us to ship the bootstrappers' logs for them after the bootstrap.libp2p.io handover; info adds libp2p's and the DHT's warnings and info lines, and JSON matches needle so Loki can parse the fields. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ProbeLab asked us to ship the bootstrappers' logs for them. Their bootstrapper data stopped in early October, after the bootstrap.libp2p.io identities moved to these boxes (26c2ffb, 2026-09-29).
Why
The bootstrap daemon sets no
GOLOG_LOG_LEVEL, so kubo runs at go-log's default,error, and Loki gets almost nothing from it. Info adds libp2p's and the DHT's warnings and info lines (failed identify negotiations, resource-manager trouble, DHT record errors).What this does not give: per-connection peer IDs or agent versions. In go-libp2p v0.49.0 (kubo 0.43.1), identify logs successful connections only at debug, with peer ID and address but never the agent version. ProbeLab's weekly
count(distinct peer)/count(distinct agent)needs a collector readingswarm peers --identify, not log levels.Change
k8s/bootstrap/deployment.yaml.j2:GOLOG_LOG_LEVEL=infoandGOLOG_LOG_FMT=jsonon the kubo container, as needle already does. Applies to every instance (bootstrap, and bootstrap-b on chic-1).Checked
scripts/render-manifests.ymlrenders, and every overlay builds withkubectl kustomize.bootstrapdiff againstmain: only the two env vars.bootstrap-wssunchanged..j2template). CI runs them.Rolling out
bootstrap.ymlis applied by hand, not from CI, so merging changes nothing on the boxes. Applying it restarts each bootstrapper (one box at a time underproduction_rollout). After that, watch the bootstrappers' log volume in Loki: if info is noisy (identify failures on high peer churn), narrow it toerror,net/identify=infoorwarn.🤖 Generated with Claude Code