cli: add a dev command with in-memory builds and HMR #1188

Hey folks 👋

Summary

Now that doc-kit preview is on its way (#1082), the natural follow-up is a doc-kit dev command: a long-running process that keeps the whole pipeline in memory, serves pages through Vite's dev server, and on every change redoes only the work that change affects, pushing the result straight to the browser.

Today the loop for anyone editing docs (or working on doc-kit itself) is: change a file, run a full generate, serve the output, refresh. node --run watch just restarts the whole CLI. On the Node.js API docs, a full web build takes roughly 7-9s with 4 threads and around 25s with one, and almost all of that is spent on pages you didn't touch:

Editing one Markdown file should only re-parse that file, rebuild its page, and render it when the browser asks for it.

Proposal

1. Output that isn't necessarily a folder

Every generator writes its own files with join(config.output, ...), through @doc-kit/core/utils/file.mjs or node:fs directly (15 generators), and html also writes its page programs to os.tmpdir() and runs vite build with write: true. A dev command can't keep things in memory until that goes through one place.

The idea is an output abstraction the generators write through, with the destination picked by --output:

// instead of writeFile(join(config.output, 'sitemap.xml'), sitemap)
await output.write('sitemap.xml', sitemap);

• a folder, as today (-o out/)
• memory, for dev (and for tests, which today write to temp folders)
• a stream or callback, for programmatic use (e.g. piping pages somewhere, or an upload target)

Several generators already skip writing when output isn't set and just return their result, so this mostly generalizes what's there.

2. dev reuses the preview server

dev would use the same server as #1082 (createStaticServer), serving from the in-memory output instead of a folder, with Vite's middleware in front of it for client assets and HMR:

doc-kit dev -i 'doc/api/*.md' --port 5173

3. A development build mode

dev doesn't need production output, so it can skip the expensive parts:

• Vite's dev server instead of vite build for the client (server.middlewareMode, appType: 'custom'), so CSS and island components get real HMR.
• Server components loaded through Vite's Environment API (server.environments.ssr.runner.import(), the replacement for ssrLoadModule) instead of bundling the component library up front.
• No minification (JS, CSS through lightningcss, or --minify HTML), with source maps on.
• Pages rendered on request, the way VitePress and hugo server work, instead of rendering all 782 up front. This is probably the single biggest win.
• Aggregates (all.html, the search index, sitemap.xml, llms.txt) built lazily, or skipped by default.

4. HMR that knows what changed

Watch the inputs and map each change to the work it invalidates:

The browser gets a reload of the page it's on through Vite's HMR channel, which can get finer-grained later.

Open questions

• Per-file results. Generators stream whole collections today (metadata returns one array), and caching.mjs evicts each result after its last consumer, because a run ends. dev needs results kept and invalidated per input file, which is the same groundwork as incremental builds (feat(caching): support incremental builds #1010 and core: incremental generation against an existing output (--incremental) #1189).
• Workers. Keep a warm pool in dev instead of ending idle workers after 250ms?
• Which targets. Starting with web seems right; legacy-html could follow once the output abstraction exists.
• Determinism. Comparing a rebuilt page with what's in memory works best once outputs are byte-identical between builds (core: output order changes between builds #1183, orama-db: orama-db.json changes on every build #1184).

Acceptance criteria

• doc-kit dev serves the site without writing build output to disk.
• Editing one Markdown file updates the open page without a full rebuild.
• Editing a component or stylesheet updates the page through HMR.
• generate's output stays exactly the same.

References

• feat: add preview command #1082: doc-kit preview (this is its suggested follow-up)
• feat(caching): support incremental builds #1010: incremental builds and a watch command (prior art)
• Vite: Server-Side Rendering and Environment API for frameworks
• hugo server --renderToMemory
• VitePress dev server
• Eleventy's incremental builds (how it maps a change to what depends on it)