Skip to content

Sync ako/mxcli: incremental source-catalog rebuilds (re-describe only what changed) - #1679

Merged
ako merged 3 commits into
mendixlabs:mainfrom
ako:main
Oct 30, 2026
Merged

ako merged 3 commits into
mendixlabs:mainfrom
ako:main

Conversation

@ako

@ako ako commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Syncs ako/mxcli:main into upstream: 1 commit from fork PR ako#1802.

Refs #1081 and #1329.

Incremental source-catalog rebuilds

Problem: once a project had a source index (refresh catalog full source), every later rebuild described every document again. That included the implicit rebuilds that search and show references start after every save. On a 4,753-unit app that took 38 s each time.

This follows from #1081's never-narrow guard: the cheapest rebuild that can still be saved is the most expensive one.

Fix: a source build now records what it described from:

  • source_units: every unit's content hash and container;
  • source_elements: which document came from which unit;
  • a fingerprint of the mxcli build that wrote the rows.

The next build carries a document's row over only when all of these hold:

  1. Its own unit is unchanged. An entity's unit is its domain model, and the container is compared too, because a move into another folder changes the described folder.
  2. No document its references name has changed. For example, a page prints its data-source flow's return entity.
  3. Nothing changed outside the described documents. A folder, module, settings or Java action change describes everything, as before.
  4. The same mxcli build wrote the rows.

FORCE still describes everything. Anything the plan can't vouch for is described again, so reuse can make a build faster but can't change its result.

Other details:

  • Reader.UnitContentHashes reads v2 hashes from the Unit table, and computes v1 hashes from the blob, because older v1 writes left them stale.
  • No schema bump is needed: a cache without the new tables simply reuses nothing.

Evidence

  • TestSourceCatalog_IncrementalBuildEqualsFullBuild compares the incremental build with a forced full build, row for row, after each of these:
    • no change;
    • a flow edit;
    • a callee's return type changed;
    • a data-source flow returning another entity;
    • a flow added, renamed or dropped;
    • an entity altered;
    • a flow moved to a new folder, and another moved into it.
  • Controls:
    • With dependency tracking stubbed out, the test fails on the page whose data-source flow changed.
    • Without the container in the comparison, it fails on the folder move.
    • It fails if no build reused rows.
  • On a 4,753-unit MPR v2 app, after alter entity … add attribute, a plain refresh catalog takes 11.2 s instead of 38.2 s, reusing 3,525 of 3,718 rows. The result is byte-identical to a forced full rebuild.
  • Docs: the Caching section of docs-site/src/tools/refresh-catalog.md explains the incremental rebuild and what FORCE does

ako and others added 3 commits October 30, 2026 01:10
…index (#1790)

Once a project had a source index, every rebuild described every
document again - including the implicit rebuilds `search` and `show
references` start after each save (38 s on a 4,753-unit app).

A source build now records its basis (source_units: every unit's content
hash + container; source_elements: which document came from which unit)
and the next build carries a row over when
- its unit is unchanged (an entity's unit is its domain model),
- no document its refs name changed (a page prints its data-source
  flow's return entity),
- nothing else changed (a folder, module, settings or Java action change
  describes everything), and
- the same mxcli build wrote it (fingerprint: executable path/size/mtime).
FORCE describes everything.

UnitContentHashes: v2 from the Unit table, v1 from the blob (v1 hashes
were left stale before #1159).

FactoryManagement, one attribute added: plain refresh 11.2 s vs 38.2 s,
3,525 of 3,718 rows reused, byte-identical to `force`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_011oK82ttaGzay5CwiC2V7cf
fix(catalog): re-describe only what changed when rebuilding a source index (#1790)
@ako
ako merged commit aed6c20 into mendixlabs:main Oct 30, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant