Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 42 additions & 4 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -5586,6 +5586,46 @@ store's `wordHits()`, which with no word now returns every entry. The decisions:
M1 of §20.1), at the same peak memory (≈ 0.5 GB). A store a model wrote is
owed, as above.

**As built — the glossary in the page (2026-10-28).** `/glossary.json`, the
Vocabulary tab's entries and the Navigate panel's, over `answerGlossary` in
`@codegraph/insights` (`glossary.ts`). The decisions:

- **The scope is the model's, injected.** `answerGlossary(store, query,
placement)` takes a `Placement` (an id → its containing module and type);
the CLI builds it from the analyzer's fold over the entities the view keeps,
so a word's carriers and its glossary are of the same entities, and the
package still opens no model. Under a module: what folds into it, its own
record included; under a type: the type and its members. A module's
reconciliation is under its module and the corpus, not under a type. A scope
is a kept module or type; anything else is `unknown-scope`, 404.
- **One store reader.** `insightLookup` and `glossaryLookup` share
`storeReader(path)`, opened at the first request that finds the file. No
store is `{status: "no-store"}`, 404, and the tab shows nothing. The
placement is built at the first request a store can answer, then the graph
is let go, as for the carriers.
- **The answer** is the projection's entries without the natural key (the
page has no use for it), an operation's `owner` (its type, else its module:
the node a page reveals), ordered module, types, operations, then by id; the
modules' glossary entries whole. Each list is cut at `limit` (200) with its
total. No `meaning` per entity: the projection does not hold it, and the
panel shows it from the block. `withoutWords` is the store's.
- **The tab** lists them under the clicked word's carriers, inside the
explanation panel's border and badge (`LLM glossary`), with the models that
wrote them; an entry reveals its entity in Navigate (an operation, its type);
the prompt-1 records are counted in a footnote.
- **The panel reads `words` and `glossary` by name** — the one exception to
its generic block rendering, since generically a word entry showed the
identifiers where a definition reads, and a glossary entry lost the types
that disagree: `order (domain) — sense. Meaning. In Order · 0.90`.
- **One remembered lookup** (`lookup.ts`): the insight, carriers and glossary
loaders ask once per URL and stop asking a server with no such route.
- **Measured** on the Broadleaf store of the previous note: the first request
takes 0.21 s with the placement built, later ones 0.11 s; `order` over the
corpus is 1 908 entries, 200 sent (53 KB). The server's RSS after a click
is the same within noise (605 MB, against 619 MB without the route).
Screenshots reviewed at 1600×1000 on the Java fixture (a hand-worded store
over its ids) and on Broadleaf.

### 20.4 Milestone split

**M19a — the lexicon, the split, the artifact** (closes the profile half of #19)
Expand Down Expand Up @@ -5660,11 +5700,9 @@ store's `wordHits()`, which with no word now returns every entry. The decisions:
Broadleaf measurement. *(In: `--terms`, the estimate's per-word
figures, the fake-client run, the prompt-1 answer. Owed: the
measurement, live runs.)*
- [ ] Readers: `insights --word`, `--glossary`, `--polysemy`, `--concerns`;
- [x] Readers: `insights --word`, `--glossary`, `--polysemy`, `--concerns`;
`vocabulary --drift-insights`; `/glossary.json`; the entries in the
Vocabulary tab and in the Navigate panel as inferences. *(In: the four
`insights` readers, `--drift-insights`. Owed: `/glossary.json`, the tab
and the panel.)*
Vocabulary tab and in the Navigate panel as inferences.
- [ ] `docs/vocabulary.md` (the idioms per profile, the classes, the
weights, the artifact, the three layers, the readers, the drift
reports); README names the feature; `docs/cli.md` the command and the
Expand Down
18 changes: 16 additions & 2 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,14 +181,28 @@ artifact, `/insight.json?id=…`, looked up in `<model>.insights.db` beside the
first model (or `--insights FILE`) when asked for. With no store the route
answers 404 and the page looks exactly as it did.

Over the same store, a clicked word on the Vocabulary tab also shows **what
the explanations under that scope said it means**: each module's
reconciliation (and the types that disagree on it), then each explained
entity's entry — its concern, sense and confidence, and the identifiers the
word is in — in the same inference colour and badge, each one click from its
node (an operation reveals its type). That is the server's third lookup,
`/glossary.json?term=WORD[&scope=ID][&limit=N]`: the entries of the entities
the model folds under the scope, as the carriers are, from the store's word
index; the first `limit` (200) of each list, and their totals. An unknown
scope is `404`, and so is having no store (the tab then shows nothing); no
term, or a limit that is no whole number, is `400`. Navigate's explanation
panel shows the same words for a selected type or module, with their
meaning, in its structured block.

**`--app`** is the same page as the daemon behind the desktop app (and the
development loop for it: run it from a checkout, open the URL in a browser).
No model on argv. It binds `127.0.0.1:0` unless `--host`/`--port` say
otherwise, prints exactly one JSON line on stdout — `{"port":N,"token":"…"}`
— and serves every route under `/<token>/`: the page, `app` (the registry
and the current project), `recent`, `navigator.json`, `city.json`,
`vocabulary.json` (no `carriers.json`: the daemon holds no model between
jobs, and the tab shows the counts without the names),
`vocabulary.json` (no `carriers.json` or `glossary.json`: the daemon holds no
model or store between jobs, and the tab shows the counts without the names),
`POST jobs` with `{"src": DIR-or-model.jsonl, "extractor"?: NAME}` and the
`jobs/current` event stream (`started`, `phase`, `progress` — the
extractor's own stderr lines — `done`, `failed`; `idle` when nothing runs).
Expand Down
17 changes: 15 additions & 2 deletions docs/insights.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,8 +264,8 @@ id: *explained* (the record), *failed* (the unit's failure — matched by member
so every member of a failed cycle is answered), or *unknown*. "No explanation"
is two different facts, and only one of them is silence.

Two readers answer about records (a third, `vocabulary --drift-insights`,
about words: below). `codegraph insights` (cli.md) opens the store and nothing
Two readers answer about records (two more, `vocabulary --drift-insights` and
`serve`'s `/glossary.json`, about words: below). `codegraph insights` (cli.md) opens the store and nothing
else — not the model, not `explain.ts`, whose module graph is the provider SDK.
And `codegraph serve` answers `/insight.json?id=…`, its one route that is not an
artifact: tens of megabytes of prose do not belong in `navigator.json`, so a
Expand Down Expand Up @@ -309,6 +309,19 @@ every entry through `wordHits()`, hands them to `@codegraph/vocabulary` in that
package's own shape (neither package imports the other), and sets the model's
concern beside the list's class, word by word, with the entities.

The page asks the same question of one word: `codegraph serve` answers
`/glossary.json?term=…&scope=…` with `answerGlossary(store, query, placement)`,
the entries of the entities under a module or a type, and the modules'
reconciliations. Which entities are under a scope is the model's (only the
model knows an operation's type), so the CLI hands in a `Placement` — each
kept entity's containing module and type, by the fold the vocabulary counts
with — and this package still opens no model. The route reads the store
`/insight.json` reads, through one reader. The kind (`codegraph.glossary/1`)
is restated by the page and pinned, and the page shows the entries in the
inference colour; the explanation panel reads a block's `words` and `glossary`
by name (`order (domain) — sense. Meaning. In Order · 0.90`), the one exception
to its generic rendering.

`@codegraph/insights` stays pure: `store.ts` is a port, and `store-sqlite.ts`
is handed an OPEN database, so the path, the file and `node:sqlite` itself
(one import site, in the analyzer's `store/sqlite.ts`) stay with the CLI.
Expand Down
4 changes: 2 additions & 2 deletions packages/cli/src/app/daemon.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ import { RECENT_KIND, readRecent } from "./store.js";
* with relative URLs, so `navigator.json`, `city.json` and `vocabulary.json`
* are fetched exactly as `codegraph serve` hands them out — no Tauri IPC, no
* CORS. The lookups are not: the daemon holds no store open and no model
* between jobs, so `insight.json` and `carriers.json` are 404s the page
* reads as "not here".
* between jobs, so `insight.json`, `carriers.json` and `glossary.json` are
* 404s the page reads as "not here".
*/
export const APP_KIND = "codegraph.app/1";

Expand Down
4 changes: 2 additions & 2 deletions packages/cli/src/args.ts
Original file line number Diff line number Diff line change
Expand Up @@ -527,7 +527,7 @@ export const SERVE_SPEC: CommandSpec = {
name: "insights",
type: "string",
describe:
"The insights store behind the page's explanation panel (what 'codegraph explain' wrote). " +
"The insights store behind the page's explanation panel and the Vocabulary tab's glossary (what 'codegraph explain' wrote). " +
"Default: <model>.insights.db beside the first model; with none, or one an older codegraph wrote that no 'codegraph explain' has migrated yet, the page shows no explanations.",
placeholder: "FILE",
},
Expand Down Expand Up @@ -1309,7 +1309,7 @@ export interface ServeOptions extends ModelInputOptions, CityBuildOptions, Cache
readonly host: string;
/** `--app`: the daemon form; `models` is then empty. */
readonly app: AppOptions | undefined;
/** `--insights FILE`: the insights store behind the explanation panel; undefined = beside the first model. */
/** `--insights FILE`: the insights store behind the explanation panel and the glossary; undefined = beside the first model. */
readonly insights: string | undefined;
}

Expand Down
21 changes: 13 additions & 8 deletions packages/cli/src/commands/serve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ import { parseRegistry, type Registry } from "../app/registry.js";
import { CARRIERS_ROUTE, carriersLookup } from "../carriers-route.js";
import { EXIT, UsageError, type ExitCode } from "../exit.js";
import { loadHistoryJoin } from "../history-metrics.js";
import { INSIGHT_ROUTE, insightLookup } from "../insight-route.js";
import { GLOSSARY_ROUTE, glossaryLookup, modelPlacement } from "../glossary-route.js";
import { INSIGHT_ROUTE, insightLookup, storeReader } from "../insight-route.js";
import { storePathForModel } from "../insights-paths.js";
import { errLine, errLines, type IoSink } from "../io.js";
import { openAnalysis } from "../source.js";
Expand Down Expand Up @@ -37,11 +38,13 @@ import { vocabularyWarnings } from "./vocabulary.js";
* `/vocabulary.json`, each exactly the file the respective command's `--out`
* would have written (the vocabulary under its default weights and list).
*
* TWO ROUTES ARE NOT ARTIFACTS: `/insight.json?id=…`, a selected node's
* THREE ROUTES ARE NOT ARTIFACTS: `/insight.json?id=…`, a selected node's
* explanation, looked up in `<model>.insights.db` when asked for (M16c) — no
* store means a 404 the page ignores, so a model nobody explained looks
* exactly as it did; and `/carriers.json?term=…`, the names that carry a word
* of the vocabulary, answered from the model this command holds (M19b).
* exactly as it did; `/carriers.json?term=…`, the names that carry a word
* of the vocabulary, answered from the model this command holds (M19b); and
* `/glossary.json?term=…`, what the explained entities under a scope said
* that word means — the same store, placed by the same model (M19c).
*
* Stdout stays empty: the server is the destination. Warnings and the reach
* announcement are stderr. The command returns at once; the live server is
Expand Down Expand Up @@ -109,17 +112,19 @@ export function serveCommand(options: ServeOptions, io: IoSink, deps: ServeDeps
]);

// Beside the FIRST model, like explain's default; --insights names it otherwise. Opened on first request.
const insights = insightLookup(options.insights ?? storePathForModel(options.models[0] ?? ""));
// Holds the graph until the first answerable request builds the index, then lets it go.
const store = storeReader(options.insights ?? storePathForModel(options.models[0] ?? ""));
const insights = insightLookup(store);
// Each holds the graph until its first answerable request builds what it needs, then lets it go.
const carriers = carriersLookup(() => vocabularyCarriers(graph, vocabulary, counting));
const glossary = glossaryLookup(store, () => modelPlacement(graph, view));
deps.startServer({
routes: {
[NAVIGATOR_ROUTE]: navigatorToJsonString(navigator),
[CITY_ROUTE]: cityToJsonString(city),
[VOCABULARY_ROUTE]: vocabularyToJsonString(vocabulary),
},
lookups: { [INSIGHT_ROUTE]: insights.handle, [CARRIERS_ROUTE]: carriers.handle },
onClose: () => insights.close(),
lookups: { [INSIGHT_ROUTE]: insights.handle, [CARRIERS_ROUTE]: carriers.handle, [GLOSSARY_ROUTE]: glossary.handle },
onClose: () => store.close(),
label: "codegraph",
assets,
port: options.port,
Expand Down
92 changes: 92 additions & 0 deletions packages/cli/src/glossary-route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import type { EntityId } from "@codegraph/core";
import { folderFor, type CodeGraph, type View } from "@codegraph/analyzer";
import { GLOSSARY_ANSWER_KIND, answerGlossary, type GlossaryAnswer, type Place, type Placement } from "@codegraph/insights";
import type { LookupAnswer, StoreReader } from "./insight-route.js";

/**
* `/glossary.json?term=…[&scope=…][&limit=…]` — the third lookup of `codegraph
* serve` (PLAN.md §20.3): what the explained entities under a scope said a
* word means, from the store's `insight_word` projection, through the same
* reader as `/insight.json`. The answer is `@codegraph/insights`'; which
* entities are under a scope is the model's, by the fold the vocabulary counts
* with, so a word's carriers and its glossary are of the same entities.
*
* QUIET LIKE `/insight.json`: no store, or one this build must not read, is a
* 404 the page ignores. An unknown scope is 404, as on `/carriers.json`; a
* request that cannot be asked — no term, a limit that is no whole number —
* is 400.
*
* THE PLACEMENT IS BUILT AT THE FIRST REQUEST A STORE CAN ANSWER, reading
* every kept entity once; the function that builds it (and the graph it
* holds) is then dropped.
*/

export const GLOSSARY_ROUTE = "/glossary.json";

const USAGE = "glossary.json needs ?term=<word>, and takes &scope=<module or type id>&limit=<n>";

export interface GlossaryLookup {
handle(query: URLSearchParams): LookupAnswer;
}

export function glossaryLookup(reader: StoreReader, build: () => Placement): GlossaryLookup {
let pending: (() => Placement) | undefined = build;
let placement: Placement | undefined;

return {
handle(query) {
const term = query.get("term");
const limit = query.get("limit");
if (term === null || term === "") return refused(USAGE);
if (limit !== null && !/^\d+$/u.test(limit)) return refused(`glossary.json: a limit is a whole number >= 0, not ${JSON.stringify(limit)}`);

const store = reader.get();
if (store === undefined) return answered({ kind: GLOSSARY_ANSWER_KIND, status: "no-store" });
if (placement === undefined && pending !== undefined) {
placement = pending();
pending = undefined;
}
const scope = query.get("scope");
return answered(
answerGlossary(
store,
{ term, ...(scope === null || scope === "" ? {} : { scope }), ...(limit === null ? {} : { limit: Number(limit) }) },
placement as Placement,
),
);
},
};
}

/**
* Where the model folds each kept entity — the vocabulary's rule: a non-stub
* entity the view keeps, under its containing module and type. Entities that
* share a place share its object.
*/
export function modelPlacement(graph: CodeGraph, view: View): Placement {
const folder = folderFor(graph);
const shared = new Map<string, Place>();
const places = new Map<EntityId, Place>();
for (const id of graph.ids()) {
const entity = graph.entity(id);
if (entity === undefined || graph.isStub(id) || !view.entity(entity, graph)) continue;
const module = folder.containingModule(id);
const type = folder.containingType(id);
const key = `${module ?? ""}\u0000${type ?? ""}`;
let place = shared.get(key);
if (place === undefined) {
place = { ...(module === undefined ? {} : { module }), ...(type === undefined ? {} : { type }) };
shared.set(key, place);
}
places.set(id, place);
}
return (id) => places.get(id);
}

function answered(answer: GlossaryAnswer): LookupAnswer {
return { status: answer.status === "found" ? 200 : 404, body: JSON.stringify(answer) };
}

function refused(error: string): LookupAnswer {
return { status: 400, body: JSON.stringify({ error }) };
}
57 changes: 33 additions & 24 deletions packages/cli/src/insight-route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,42 +25,51 @@ export interface LookupAnswer {
readonly body: string;
}

export interface InsightLookup {
handle(query: URLSearchParams): LookupAnswer;
/** The store the lookups read, opened at the first request that finds it; `/glossary.json` reads the same one. */
export interface StoreReader {
/** Undefined while there is no store this build may read. */
get(): InsightsStore | undefined;
close(): void;
}

export const INSIGHT_ROUTE = "/insight.json";

export function insightLookup(storePath: string): InsightLookup {
export function storeReader(storePath: string): StoreReader {
let store: InsightsStore | undefined;

const reader = (): InsightsStore | undefined => {
if (store !== undefined) return store;
// Checked first: opening a missing path would CREATE it, and a page asking must not leave a file behind.
if (!existsSync(storePath)) return undefined;
const db = loadSqlite().open(storePath);
try {
store = sqliteInsightsStore(db, { readOnly: true });
} catch {
// Not ours, from a newer build, or from an older one no explain has migrated yet: nothing this page can show.
// `codegraph insights` says why.
db.close();
}
return store;
return {
get() {
if (store !== undefined) return store;
// Checked first: opening a missing path would CREATE it, and a page asking must not leave a file behind.
if (!existsSync(storePath)) return undefined;
const db = loadSqlite().open(storePath);
try {
store = sqliteInsightsStore(db, { readOnly: true });
} catch {
// Not ours, from a newer build, or from an older one no explain has migrated yet: nothing this page can show.
// `codegraph insights` says why.
db.close();
}
return store;
},
close() {
store?.close();
store = undefined;
},
};
}

export interface InsightLookup {
handle(query: URLSearchParams): LookupAnswer;
}

export const INSIGHT_ROUTE = "/insight.json";

export function insightLookup(reader: StoreReader): InsightLookup {
return {
handle(query) {
const id = query.get("id");
if (id === null || id === "") return { status: 400, body: JSON.stringify({ error: "insight.json needs ?id=<entity id>" }) };
const open = reader();
const open = reader.get();
const answer = open === undefined ? { kind: INSIGHT_ANSWER_KIND, id, status: "unknown" as const } : answerInsight(open, id);
return { status: answer.status === "unknown" ? 404 : 200, body: JSON.stringify(answer) };
},
close() {
store?.close();
store = undefined;
},
};
}
Loading
Loading