Skip to content

Reject incomplete sitemap generation and recover after source failures - #1267

Open
tannerlinsley wants to merge 86 commits into
taren/docs-freshness-uifrom
taren/sitemap-completeness
Open

tannerlinsley wants to merge 86 commits into
taren/docs-freshness-uifrom
taren/sitemap-completeness

Conversation

@tannerlinsley

@tannerlinsley tannerlinsley commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

A failed or empty documentation manifest could produce a 200 sitemap with an entire library missing. Sitemap generation now fails explicitly with 503, no-store headers, and Retry-After instead of publishing that reduced result. Logs identify the failed library or catalog and record successful generation after recovery.

GitHub directory fallback now visits all depths and rejects a listed child directory that disappears. Cached trees with unvisited directories are rejected, and the manifest cache key changes so old depth-limited manifests are rebuilt. Local directory reads use the same complete traversal. Local-only one-millisecond file/tree caching was removed after the HTTP recovery test demonstrated that it kept serving an empty tree after the fixture was restored. Production GitHub cache behavior remains in place.

Stacked on #1264, which supplies the freshness metadata already used by sitemap entries and still needs its required maintainer review. Current main is integrated. This addresses only part of #1153, which stays open.

Validation:

  • Full pnpm test passed: 698 unit tests, 2,278 chat tests and 20 desktop tests passed, with 6, 6 and 1 skipped respectively, TypeScript and lint clean. The commit hook repeated the final checks successfully.
  • New remote fallback regression proves a missing child rejects without caching a partial tree, then recovers through an old incomplete cache with a deeply nested document intact.
  • The opt-in HTTP test was run separately on 78b30e5 against a credential-free local application snapshot, and passed with fixtures for all sitemap-enabled documentation libraries. Removing the Start fixture and corrupting its frontmatter each produce an uncached 503; restoring each source immediately restores every expected library URL. Server logs name Start and show recovery.
  • The integration uses the existing published/cached Charts catalog. It does not claim to simulate a catalog-service outage.
  • sitemap-testing.md and scripts/prepare-sitemap-test.ts make the HTTP check reproducible.

Workers built the current head successfully. A read-only request to its sitemap preview timed out after 60 seconds without response bytes, so deployed sitemap behavior is not yet verified.

@coderabbitai

coderabbitai Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 921675a3-e530-407e-9696-aed35a5b3322

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
tanstack-com 78b30e5 Commit Preview URL

Branch Preview URL
Nov 01 2026, 04:13 AM

tannerlinsley and others added 6 commits October 13, 2026 15:35
* fix: read local docs directories through the worker bridge

* fix(seo): canonicalize Markdown docs to HTML

* fix: allow local repository root directory reads
Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
Co-authored-by: Tanner Linsley <[email protected]>
@changeset-bot

changeset-bot Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 78b30e5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@tannerlinsley

Copy link
Copy Markdown
Member Author

Updated this branch with the current docs-freshness parent and resolved the local-docs fallback conflict.

The integration test caught two cases worth keeping covered: an existing empty local docs directory must stay empty instead of falling back to remote docs, and malformed YAML must fail manifest generation instead of silently dropping metadata. Reference freshness handling also preserves that parse failure.

Verification on 2a8a2b2:

  • Full pnpm test and commit hooks passed: types, lint, 562 unit tests, 2,269 chat tests and 20 desktop tests. Five unit tests and one desktop test were skipped.
  • The isolated HTTP test covered all 18 sitemap-enabled docs libraries. Removing a source or corrupting its frontmatter returned 503 with no-store headers and Retry-After: 60. Restoring each source immediately restored the complete sitemap.
  • Focused regressions cover missing-directory fallback, empty-directory recovery, malformed metadata and referenced metadata.

This still depends on #1264 and the required maintainer review. It only addresses part of #1153, which should stay open for the remaining canonical URL and deployed-page checks.

sukvvon and others added 19 commits October 15, 2026 02:50
…viewport (#1169)

* refactor(components/MaintainerCard): switch name size from viewport to card width with container query

* refactor(maintainers): remove 'max-w' cap so cards fill grid tracks and adjust responsive columns

* refactor(maintainers): keep a two-column floor so narrow screens stop scrolling one card at a time
Restore library status badges in the library list
* feat(hotkeys): Refresh landing page with live demos

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* style(hotkeys): Format landing page

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix hotkeys demo selection beyond visible rows

* feat(hotkeys): Update landing page for Hotkeys 0.14 and mark beta

Sequences now use useHotkey, the copy covers app hotkeys, and the
library badge moves from alpha to beta.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: Tanner Linsley <[email protected]>
* Revert Worker split after production outage

* ci: apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Label application starter prompt fields for assistive technology

* Include visible prompt text in accessible names
…er (#1245)

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
…ate (#1394)

* Stage isolated Worker split and verify deployment name handling

* ci: apply automated fixes

* Verify staging Builds and authenticated flows against isolated Neon

* Record actual CI destinations and keep staging checks in ESM

* Preserve matching staging CI identity and refresh synthetic sessions

* Record successful isolated Cloudflare Builds and authentication verification

* Label application starter prompt fields for assistive technology (#1389)

* Label application starter prompt fields for assistive technology

* Include visible prompt text in accessible names

* Add keyboard skip link for site navigation (#1391)

* fix: expose homepage main landmark (#1393)

* Guard production Worker destinations and preserve recovery namespaces

* Validate CI account context and record verified production rollout

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fix(sentry): filter stale app chunk load errors

* fix: retain Sentry reports when stale app reload is throttled

---------

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
Co-authored-by: Tanner Linsley <[email protected]>
Flag suspect first-session traffic without excluding analytics
- Bump @tanstack/react-hotkeys to ^0.15.1 for scopes.
- Replace the platform labels section with a live hotkey scopes demo.
- Move the platform and layout toggles into the recorder demo, which now
  uses app hotkeys and has a ? cheat sheet.
- Rework the keydown listener cards and show the patched listener beside them.
- Describe what Hotkeys is in the hero, and update the library tagline.
…ion (#1401)

fix(hotkeys): allow dismissing the cheat sheet after clicking inside
* feat(application-starter): inject migration guide URLs into final prompt

When the user brief indicates a Next.js or Remix/React Router migration,
append an instruction telling the downstream agent to fetch the relevant
tanstack.com migration guide and follow it. Keeps the guide out of the
analyze stage and out of the UI; only affects the final generation prompt.

* fix: match migration guides to explicit frameworks

---------

Co-authored-by: Tanner Linsley <[email protected]>
tannerlinsley and others added 25 commits October 26, 2026 14:47
* fix(stats): redirect comparison links on the server

* fix(stats): reject overlong comparison package names

---------

Co-authored-by: Tanner Linsley <[email protected]>
* blog: draft TanStack DB 1.0 RC1 oracle essay

* docs: sync RC1 oracle essay with latest revisions

* docs: sync latest TanStack DB RC1 essay

* docs: sync latest TanStack DB RC1 essay revisions

* docs: update DB RC1 essay with revised oracle explanations

* docs: address RC1 post copy review

* docs: revise DB RC1 essay and add illustrations

* docs: remove draft flag from DB RC1 post

* docs: add hero image to DB RC1 post

* docs: link DB RC migration guide from release intro
docs: add Redact and Preact homepage comparison
* docs: schedule DB RC1 post for October 29

* docs: clarify RC1 post framing and replace crane illustration

* docs: refine DB RC1 essay after editorial review

* docs: improve oracle explanation flow and subscriber example
* Verify required Builder migration within shared history

* fix: select required Builder migration from journal

---------

Co-authored-by: Tanner Linsley <[email protected]>
* Add live visitor presence and Scarf company lookup portal

* Make live presence dashboard company first
* Protect live presence intake and isolate retention cleanup

* Apply live page eligibility before limiting results

* Add company prioritization tools and Scarf history to live dashboard

* Add active and background company status filters and column

* Add admin-initiated visitor chat with deferred interface assets

* Fix visitor chat Worker startup and background polling

* Include recently left companies in dashboard breakdowns

* Enable visitor chat for production rollout

* Dock admin visitor chats in independent bottom tabs

* Fix chat dock dark mode and show minimized inbox by default

* Reduce duplicate R2 cache refresh writes

* Add one-click visitor chat self-test for admins

* Isolate deferred chat styles from the site layout

* Share TanStack identity and add resource-bound Source OAuth

* ci: apply automated fixes

* Isolate self-test conversations and sync shared admin tabs

* Include existing OAuth tables in the migration test boundary

* fix: load existing conversations in open visitor chat tabs (#1450)

* Keep chat composers focused while sending

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* feat(highlight): theme editor page

* chore: pr comments

* fix(highlight): keep numeric theme names valid in exports

* fix(highlight): validate restored editor drafts

---------

Co-authored-by: Tanner Linsley <[email protected]>
fix: show baseline display names in npm chart tooltips
)

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
* fix(intent): load skill content and resolve package references

* fix(markdown): tolerate malformed package references

* ci: apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Update design system and shop experience

* Resolve current main compatibility

* Preserve date-based badges across paginated shop results

---------

Co-authored-by: Tanner Linsley <[email protected]>
* Add semantic breadcrumbs to partner detail pages

* Use shared breadcrumbs on bespoke partner pages
* Add semantic breadcrumbs to chart examples

* Escape chart breadcrumb JSON-LD script content
* Add semantic breadcrumbs to chart examples

* Escape chart breadcrumb JSON-LD script content

* Add semantic breadcrumbs to chart collections
* Add semantic breadcrumbs to chart examples

* Escape chart breadcrumb JSON-LD script content

* Add semantic breadcrumbs to chart collections

* Complete the Charts and Catalog breadcrumb hierarchy
@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updated@​tanstack/​react-hotkeys@​0.10.0 ⏵ 0.15.1100 +2310010098 +10100

View full report

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants